Fatigue Detection Method Based on Facial Features and Head Posture

doi:10.16182/j.issn1004731x.joss.21-0583

Abstract

Abstract:

Aiming at the of the single fatigue characteristics, low robustness and inability to customize fatigue thresholds for different drivers of fatigue detection methods, a method based on facial features and head posture is proposed. In face detection and face key point positioning HOG feature operator and regression tree algorithm are used. In head posture estimation, head posture Euler angle is estimated by combining the face key points with the coordinate system transformation. In fatigue feature extraction, a deep residual neural network model is established to extract the eye fatigue features, which the eye, mouth aspect ratio and head posture Euler angle. The fatigue characteristics of eyes, mouth and head are used to establish the support vector machine models for different drivers to provide the early fatigue driving warning. Experiments show that on YawDD and self-built fatigue simulation data sets, the method shows high accuracy and robustness, and can provide better fatigue warning when a certain fatigue feature detection is blocked.

Key words: fatigue driving, face detection, head posture, deep learning, support vector machine

CLC Number:

TP391

Rongxiu Lu, Bihao Zhang, Zhenlong Mo. Fatigue Detection Method Based on Facial Features and Head Posture[J]. Journal of System Simulation, 2022, 34(10): 2279-2292.

Figures/Tables 20

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Table 1

Fig. 14

Table 2

Table 3

Fig. 15

Fig. 16

Table 4

References 23

1	Tuncer T, Dogan S, Ertam F, et al. A Dynamic Center and Multi Threshold Point Based Stable Feature Extraction Network for Driver Fatigue Detection Utilizing EEG Signals[J]. Cognitive Neurodynamics(S1871-4080), 2021, 15(7): 223-237.
2	王博石, 吴修诚, 胡馨艺, 等. 基于单通道脑电信号的疲劳检测系统[J]. 计算机科学, 2020, 47(5): 225-229.
	Wang Boshi, Wu Xiucheng, Hu Xinyi, et al. Fatigue Detection System Based on Single Channel EEG Signal [J]. Computer Science, 2020, 47(5): 225-229.
3	Kyehoon, LEE, SUNG-AE, et al. Detecting Driver Fatigue by Steering Wheel Grip Force[J]. International Journal of Contents(S1738-6764), 2016, 12(1): 44-48.
4	Wang M, Guo L, Chen W. Blink Detection Using Adaboost and Contour Circle for Fatigue Recognition[J]. Computers&Electrical Engineering(S0045-7906), 2017, 58: 502-512.
5	Mandal B, Li L, Wang G S, et al. Towards Detection of Bus Driver Fatigue Based on Robust Visual Analysis of Eye State[J]. IEEE Transactions on Intelligent Transportation Systems(S1524-9050), 2017, 18(3): 1-13.
6	Zhao Xuepeng, Meng Chunning, Feng Mingkui, et al. Fatigue Detection Based on Cascade Convolutional Neural Network[J]. Journal of Optoelectronics·Laser (S1005-0086), 2017, 28(5): 497-502.
7	Omidyeganeh M, Shirmohammadi S, Abtahi S, et al. Yawning Detection Using Embedded Smart Cameras[J]. IEEE Transactions on Instrumentation and Measurement (S0018-9456), 2016, 65(3): 570-582.
8	Kazemi V, Sullivan J. One Millisecond Face Alignment with an Ensemble of Regression Trees[C]// IEEE Conference on Computer Vision and Pattern Recognition, Columbus. New York: IEEE Press, 2014: 1867-1874.
9	Zhu X, Ramanan D. Face Detection, Pose Estimation, and Landmark Localization in the wild[C]// IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2012: 2879-2886.
10	Le V, Brandt J, Lin Z, et al. Interactive Facial Feature Localization[C]// European Conference on Computer Vision. Heidelberg, Berlin: Springer, 2012: 679-692.
11	Belhumeur P N, Jacobs D W, Kriegman D J, et al. Localizing Parts of Faces Using a Consensus of Exemplars[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence(S0162-8828), 2013, 35(12): 2930-2940.
12	Sagonas C, Tzimiropoulos G, Zafeiriou S, et al. 300 Faces in-the-wild Challenge: The First Facial Landmark Localization Challenge [C]//IEEE International Conference on Computer Vision Workshops. Sydney: IEEE, 2013: 397-403.
13	Sagonas C, Antonakos E, Tzimiropoulos G, et al. 300Faces in-the-Wild Challenge: Database and Results [J]. Image and vision computing(S0262-8856), 2016, 47: 3-18.
14	Rafacl C, Richard E. Digital Image Processing[M]. Beijing: PHI, 2017.
15	Zhang Z. A Flexible New Technique for Camera Calibration [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence(S0162-8828), 2000, 22(11): 1330-1334.
16	Huber P, Hu G, Tena R, et al. A Multiresolution 3D Morphable Face Model and Fitting Framework[C]//11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. 2016: 79-86.
17	He K, Zhang X, Ren S, et al. Deep Residual Learning for Image Recognition[C]// IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Las Vegas, NV: IEEE, 2016: 770-778.
18	Yosinski J, Clune J, Bengio Y, et al. How Transferable are Features in Deep Neural Networks?[C]// International Conference on Neural Information Processing Systems. Quebec, Canada: MIT Press, 2014: 3320-3328.
19	Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-scale Image Recognition [C]//International Conference on Learning Representations. 2015.
20	Ferrario V F, Sforza C, Serrao G, et al. Active Range of Motion of the Head and Cervical Spine: A Three-Dimensional Investigation in Healthy Young Adults[J]. Journal of Orthopaedic Research (S1554-527X), 2002, 20(1): 122-129.
21	Song F, Tan X, Chen S, et al. A Literature Survey on Robust and Efficient Eye Localization in Real-Life Scenarios [J]. Pattern Recognition(S0031-3203), 2013, 46(12): 3157-3173.
22	Abtahi S, Omidyeganeh M, Shirmohammadi S, et al. YawDD: A Yawning Detection Dataset[C]// ACM Multimedia Systems (MMSys). Singapore: ACM, 2014: 24-28.
23	汪磊, 孙瑞山. 基于面部特征识别的管制员疲劳监测方法研究[J]. 中国安全科学学报, 2012, 22(7): 66-71.
	Wang Lei, Sun Ruishan. Study on Feature Recognition-based Fatigue Monitoring Method for Air Traffic Contorller[J]. China Safety Science Journal, 2012, 22(7): 66-71.

测试员编号	EAR		MAR
测试员编号	睁眼	闭眼	嘴闭合	说话	哈欠
1	0.274 1	0.085 9	0.372 8	0.640 0	1.521 6
2	0.262 0	0.107 8	0.380 2	0.750 4	1.428 1
3	0.250 3	0.139 4	0.155 2	0.500 3	1.063 1
4	0.291 9	0.159 3	0.295 5	0.867 1	1.160 4
5	0.236 6	0.145 8	0.427 7	0.794 6	1.238 9

视频编号	实际闭眼数	检测闭眼数/准确率%
视频编号	实际闭眼数	单一阈值	平均阈值	SVM模型
1	90	78/86.7	83/92.2	89/98.9
2	157	148/94.3	142/90.4	154/98.1
3	131	97/74.0	120/91.6	129/98.5
4	87	60/69.1	77/88.5	85/97.7
5	106	74/69.8	98/92.5	103/97.2

视频编号	实际哈欠数	检测哈欠数/准确率%
视频编号	实际哈欠数	单一阈值	平均阈值	SVM模型
1	3	2/66.7	3/100	3/100
2	0	0	0	0
3	2	2/100	1/50	2/100
4	5	6/83.3	4/80	6/83.3
5	4	2/50	4/100	4/100

视频编号	眼部			嘴部				头部		预警次数
视频编号	实际闭眼数	检测闭眼数	准确率/%	实际哈欠数	检测哈欠数	实际说话时间/s	检测说话时间/s	实际低头数	检测低头数	预警次数
1	90	89	98.90	3	3	12	9	5	5	8
2	157	155	98.70	0	0	20	18	1	0	0
3	131	130	99.20	2	2	17	13	4	5	7
4	87	85	97.70	5	6	6	8	8	6	13
5	106	104	98.10	4	4	18	17	13	15	21

[1]	Li Mingyu, Lin Jiaquan. Lightweight Driver Face Object Detection Algorithm Based on YOLOv8-DF [J]. Journal of System Simulation, 2025, 37(8): 2103-2114.
[2]	Wang Bingheng, Liu Tingrui, Yang Fan, Zhang Huan, Li Wei, Ma Ping, Yang Ming. Research on Requirements and Methods for Intelligent Assessment of Simulation Credibility [J]. Journal of System Simulation, 2025, 37(7): 1710-1722.
[3]	Chen Kun, Chen Liang, Xie Jiming, Liu Fengbo, Chen Taixiong, Wei Lukuan. Simulation Study on Adaptive Signal Control of Deformed Intersection Based on LSTM-GNN [J]. Journal of System Simulation, 2025, 37(6): 1343-1351.
[4]	Jiang Dawei, Dong Yangyang, Zhang Lidong, Lu Xiao, Dong Chunxi. Research on Air Target Threat Assessment Technology Based on Deep Learning [J]. Journal of System Simulation, 2025, 37(3): 791-802.
[5]	Wang Xiao, Li Xiangyang, Liang Feng, Zhang Zhili. Research on Infrared and Visible Light Fusion Method Based on ResNet-50 and Laplacian Filtering [J]. Journal of System Simulation, 2025, 37(12): 3202-3211.
[6]	Hu yang, Li Zihao, Fu Deyi, Song Ziqiu, Fang Fang, Liu Jizhen. Deep Learning Modeling of Multi-scale Characteristics of Large-scale Wind Turbine Gearbox [J]. Journal of System Simulation, 2025, 37(10): 2454-2468.
[7]	Chen Guiliang, Liu Guowei, Li Yongchao, Cai Chao, Li Zihao, Yang Dong. Multisource Information Fusion Method for Human Gait Perception [J]. Journal of System Simulation, 2025, 37(10): 2522-2532.
[8]	Gu Hao, Wang Jiayu, Xiong Weili. Soft Sensor Modeling Based on Improved Transformer in Dual-stream Framework [J]. Journal of System Simulation, 2025, 37(10): 2594-2604.
[9]	Guo Yecai, Tong Shuang. A Multimodal Residual Spatial-temporal Fusion Model Based on Automatic Sleep Classification [J]. Journal of System Simulation, 2024, 36(9): 2065-2074.
[10]	Liu Zesen, Bi Sheng, Guo Chuanhong, Wang Yankui, Dong Min. Deep Learning Based Local Path Planning Method for Moving Robots [J]. Journal of System Simulation, 2024, 36(5): 1199-1210.
[11]	Wei Jinyang, Wang Keping, Yang Yi, Fei Shumin. Incremental Image Dehazing Algorithm Based on Multiple Transfer Attention [J]. Journal of System Simulation, 2024, 36(4): 969-980.
[12]	Yang Zhe, Cui Yinghan, Guo Lingxi, Li Jiaxin, Wu Xusheng. Search Technology for Aircraft Debris Integrating Data Augmentation and Deep Learning Algorithm [J]. Journal of System Simulation, 2024, 36(10): 2238-2245.
[13]	Li Chen, He Ming, Dong Chen, Li Wei. Action Recognition Model of Directed Attention Based on Cosine Similarity [J]. Journal of System Simulation, 2024, 36(1): 67-82.
[14]	Zhang Fengquan, Cao Duo, Ma Xiaohan, Chen Baijun, Zhang Jiangxiao. Style Transfer Network for Generating Opera Makeup Details [J]. Journal of System Simulation, 2023, 35(9): 2064-2076.
[15]	Yu Du, Xinquan Yang, Jianhua Zhang, Suchun Yuan, Huachao Xiao, Jingjing Yuan. Modulation Recognition Method of Mixed Signal Based on Intelligent Analysis of Cyclic Spectrum Section [J]. Journal of System Simulation, 2023, 35(1): 146-157.