Multimodal integration for data-driven classification of mental fatigue during construction equipment operations::Incorporating electroencephalography, electrodermal activity, and video signals


Construction equipment operations that require high levels of attention can cause mental fatigue, which can lead to inefficiencies and accidents. Previous studies classified mental fatigue using single-modal data with acceptable accuracy. However, mental fatigue is a multimodal problem, and no single modality is superior. Moreover, none of the previous studies in construction industry have investigated multimodal data fusion for classifying mental fatigue and whether such an approach would improve mental fatigue detection. This study proposes a novel approach using three machine learning models and multimodal data fusion to classify mental fatigue states. Electroencephalography, electrodermal activity, and video signals were acquired during an excavation operation, and the decision tree model using multimodal sensor data fusion outperformed other models with 96.2% accuracy and 96.175%–98.231% F1 scores. Multimodal sensor data fusion can aid in the development of a real-time system to classify mental fatigue and improve safety management at construction sites.

Publication DOI:
Divisions: College of Engineering & Physical Sciences > School of Infrastructure and Sustainable Engineering > Civil Engineering
College of Engineering & Physical Sciences > Smart and Sustainable Manufacturing
Additional Information: Copyright © 2023 The Authors. Published by Elsevier Ltd. This is an open access article under the CCBY-NC-ND license (
Uncontrolled Keywords: Construction equipment operators,Construction safety,Machine learning,Mental fatigue,Multimodal data,Building and Construction,Computer Science Applications,Architecture ,Civil and Structural Engineering,Materials Science (miscellaneous),Computer Graphics and Computer-Aided Design
Publication ISSN: 2666-1659
Last Modified: 29 Feb 2024 08:22
Date Deposited: 17 Jul 2023 08:19
Full Text Link:
Related URLs: https://www.sci ... 0807?via%3Dihub (Publisher URL)
http://www.scop ... tnerID=8YFLogxK (Scopus URL)
PURE Output Type: Article
Published Date: 2023-10
Published Online Date: 2023-07-13
Accepted Date: 2023-07-11
Authors: Mehmood, Imran
Li, Heng
Umer, Waleed
Arsalan, Aamir
Anwer, Shahnawaz
Mirza, Mohammed Aquil
Ma, Jie
Antwi-Afari, Maxwell Fordjour (ORCID Profile 0000-0002-6812-7839)

Export / Share Citation


Additional statistics for this record