Fatigue Detection Method Based on Facial Features and Head Posture

doi:10.16182/j.issn1004731x.joss.21-0583

Abstract

Abstract:

Aiming at the of the single fatigue characteristics, low robustness and inability to customize fatigue thresholds for different drivers of fatigue detection methods, a method based on facial features and head posture is proposed. In face detection and face key point positioning HOG feature operator and regression tree algorithm are used. In head posture estimation, head posture Euler angle is estimated by combining the face key points with the coordinate system transformation. In fatigue feature extraction, a deep residual neural network model is established to extract the eye fatigue features, which the eye, mouth aspect ratio and head posture Euler angle. The fatigue characteristics of eyes, mouth and head are used to establish the support vector machine models for different drivers to provide the early fatigue driving warning. Experiments show that on YawDD and self-built fatigue simulation data sets, the method shows high accuracy and robustness, and can provide better fatigue warning when a certain fatigue feature detection is blocked.

Key words: fatigue driving, face detection, head posture, deep learning, support vector machine

CLC Number:

TP391

Rongxiu Lu, Bihao Zhang, Zhenlong Mo. Fatigue Detection Method Based on Facial Features and Head Posture[J]. Journal of System Simulation, 2022, 34(10): 2279-2292.

Figures/Tables 20

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Table 1

Fig. 14

Table 2

Table 3

Fig. 15

Fig. 16

Table 4

References 23

1	Tuncer T, Dogan S, Ertam F, et al. A Dynamic Center and Multi Threshold Point Based Stable Feature Extraction Network for Driver Fatigue Detection Utilizing EEG Signals[J]. Cognitive Neurodynamics(S1871-4080), 2021, 15(7): 223-237.
2	王博石, 吴修诚, 胡馨艺, 等. 基于单通道脑电信号的疲劳检测系统[J]. 计算机科学, 2020, 47(5): 225-229.
	Wang Boshi, Wu Xiucheng, Hu Xinyi, et al. Fatigue Detection System Based on Single Channel EEG Signal [J]. Computer Science, 2020, 47(5): 225-229.
3	Kyehoon, LEE, SUNG-AE, et al. Detecting Driver Fatigue by Steering Wheel Grip Force[J]. International Journal of Contents(S1738-6764), 2016, 12(1): 44-48.
4	Wang M, Guo L, Chen W. Blink Detection Using Adaboost and Contour Circle for Fatigue Recognition[J]. Computers&Electrical Engineering(S0045-7906), 2017, 58: 502-512.
5	Mandal B, Li L, Wang G S, et al. Towards Detection of Bus Driver Fatigue Based on Robust Visual Analysis of Eye State[J]. IEEE Transactions on Intelligent Transportation Systems(S1524-9050), 2017, 18(3): 1-13.
6	Zhao Xuepeng, Meng Chunning, Feng Mingkui, et al. Fatigue Detection Based on Cascade Convolutional Neural Network[J]. Journal of Optoelectronics·Laser (S1005-0086), 2017, 28(5): 497-502.
7	Omidyeganeh M, Shirmohammadi S, Abtahi S, et al. Yawning Detection Using Embedded Smart Cameras[J]. IEEE Transactions on Instrumentation and Measurement (S0018-9456), 2016, 65(3): 570-582.
8	Kazemi V, Sullivan J. One Millisecond Face Alignment with an Ensemble of Regression Trees[C]// IEEE Conference on Computer Vision and Pattern Recognition, Columbus. New York: IEEE Press, 2014: 1867-1874.
9	Zhu X, Ramanan D. Face Detection, Pose Estimation, and Landmark Localization in the wild[C]// IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2012: 2879-2886.
10	Le V, Brandt J, Lin Z, et al. Interactive Facial Feature Localization[C]// European Conference on Computer Vision. Heidelberg, Berlin: Springer, 2012: 679-692.
11	Belhumeur P N, Jacobs D W, Kriegman D J, et al. Localizing Parts of Faces Using a Consensus of Exemplars[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence(S0162-8828), 2013, 35(12): 2930-2940.
12	Sagonas C, Tzimiropoulos G, Zafeiriou S, et al. 300 Faces in-the-wild Challenge: The First Facial Landmark Localization Challenge [C]//IEEE International Conference on Computer Vision Workshops. Sydney: IEEE, 2013: 397-403.
13	Sagonas C, Antonakos E, Tzimiropoulos G, et al. 300Faces in-the-Wild Challenge: Database and Results [J]. Image and vision computing(S0262-8856), 2016, 47: 3-18.
14	Rafacl C, Richard E. Digital Image Processing[M]. Beijing: PHI, 2017.
15	Zhang Z. A Flexible New Technique for Camera Calibration [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence(S0162-8828), 2000, 22(11): 1330-1334.
16	Huber P, Hu G, Tena R, et al. A Multiresolution 3D Morphable Face Model and Fitting Framework[C]//11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. 2016: 79-86.
17	He K, Zhang X, Ren S, et al. Deep Residual Learning for Image Recognition[C]// IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Las Vegas, NV: IEEE, 2016: 770-778.
18	Yosinski J, Clune J, Bengio Y, et al. How Transferable are Features in Deep Neural Networks?[C]// International Conference on Neural Information Processing Systems. Quebec, Canada: MIT Press, 2014: 3320-3328.
19	Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-scale Image Recognition [C]//International Conference on Learning Representations. 2015.
20	Ferrario V F, Sforza C, Serrao G, et al. Active Range of Motion of the Head and Cervical Spine: A Three-Dimensional Investigation in Healthy Young Adults[J]. Journal of Orthopaedic Research (S1554-527X), 2002, 20(1): 122-129.
21	Song F, Tan X, Chen S, et al. A Literature Survey on Robust and Efficient Eye Localization in Real-Life Scenarios [J]. Pattern Recognition(S0031-3203), 2013, 46(12): 3157-3173.
22	Abtahi S, Omidyeganeh M, Shirmohammadi S, et al. YawDD: A Yawning Detection Dataset[C]// ACM Multimedia Systems (MMSys). Singapore: ACM, 2014: 24-28.
23	汪磊, 孙瑞山. 基于面部特征识别的管制员疲劳监测方法研究[J]. 中国安全科学学报, 2012, 22(7): 66-71.
	Wang Lei, Sun Ruishan. Study on Feature Recognition-based Fatigue Monitoring Method for Air Traffic Contorller[J]. China Safety Science Journal, 2012, 22(7): 66-71.

测试员编号	EAR		MAR
测试员编号	睁眼	闭眼	嘴闭合	说话	哈欠
1	0.274 1	0.085 9	0.372 8	0.640 0	1.521 6
2	0.262 0	0.107 8	0.380 2	0.750 4	1.428 1
3	0.250 3	0.139 4	0.155 2	0.500 3	1.063 1
4	0.291 9	0.159 3	0.295 5	0.867 1	1.160 4
5	0.236 6	0.145 8	0.427 7	0.794 6	1.238 9

视频编号	实际闭眼数	检测闭眼数/准确率%
视频编号	实际闭眼数	单一阈值	平均阈值	SVM模型
1	90	78/86.7	83/92.2	89/98.9
2	157	148/94.3	142/90.4	154/98.1
3	131	97/74.0	120/91.6	129/98.5
4	87	60/69.1	77/88.5	85/97.7
5	106	74/69.8	98/92.5	103/97.2

视频编号	实际哈欠数	检测哈欠数/准确率%
视频编号	实际哈欠数	单一阈值	平均阈值	SVM模型
1	3	2/66.7	3/100	3/100
2	0	0	0	0
3	2	2/100	1/50	2/100
4	5	6/83.3	4/80	6/83.3
5	4	2/50	4/100	4/100

视频编号	眼部			嘴部				头部		预警次数
视频编号	实际闭眼数	检测闭眼数	准确率/%	实际哈欠数	检测哈欠数	实际说话时间/s	检测说话时间/s	实际低头数	检测低头数	预警次数
1	90	89	98.90	3	3	12	9	5	5	8
2	157	155	98.70	0	0	20	18	1	0	0
3	131	130	99.20	2	2	17	13	4	5	7
4	87	85	97.70	5	6	6	8	8	6	13
5	106	104	98.10	4	4	18	17	13	15	21

[1]	Huimin Chai, Yong Zhang, Xinyue Li, Yanan Song. Aerial Target Threat Assessment Method based on Deep Learning [J]. Journal of System Simulation, 2022, 34(7): 1459-1467.
[2]	Xu Kang, Xiaofeng Zhang. Radar Remote Sensing Data Augmentation Method Based on Generative Adversarial Network [J]. Journal of System Simulation, 2022, 34(4): 920-927.
[3]	Bangqian Ao, Sha Yang, Jinqing Linghu, zhenhuan Ye. Design of Fatigue Driving Detection System Based on Cascaded Neural Network [J]. Journal of System Simulation, 2022, 34(2): 323-333.
[4]	Weiguo Tong, Shichao Zeng, Lifeng Zhang, Zhe Hou, Jiayue Guo. Electrical Resistance Tomography and Flow Pattern Identification Method Based on Deep Residual Neural Network [J]. Journal of System Simulation, 2022, 34(09): 2028-2036.
[5]	Lin Shuo, An Lei, Gao Zhijun, Shan Dan, Shang Wenli. Research on Intrusion Detection Based on Stacked Autoencoder and Long-short Memory [J]. Journal of System Simulation, 2021, 33(6): 1288-1296.
[6]	Cheng Wencong, Shi Xiaokang, Wang Zhigang. Creating Synthetic Satellite Cloud Data Based on GAN Method [J]. Journal of System Simulation, 2021, 33(6): 1297-1306.
[7]	Wei Jingjing, Liu Qinming, Ye Chunming, Li Guanlin. Fault Diagnosis of Mechanical Equipment Based on GA-SVR with Missing Data in Small Samples [J]. Journal of System Simulation, 2021, 33(6): 1342-1349.
[8]	Shao Cheng, Sun Hongyuan, Zhang Lin. Simulation of Anti-ship Missile Automatic Control Section Trajectory Based on PSO-LSSVM [J]. Journal of System Simulation, 2021, 33(4): 918-926.
[9]	Dong Shuqin, Zhang Bin. Network Traffic Anomaly Detection Method for Imbalanced Data [J]. Journal of System Simulation, 2021, 33(3): 679-689.
[10]	Wang Xiaohan, Zhang Lin, Ren Lei, Xie Kunyu, Wang Kunyu, Ye Fei, Chen Zhen. Brief Review on Applying Reinforcement Learning to Job Shop Scheduling Problems [J]. Journal of System Simulation, 2021, 33(12): 2782-2791.
[11]	Wang Buwei, Wang Min, Fan Qian, Wang Ya'nan, zhang hanwen, Yue Yunliang. Study on Prediction of Crystal Properties Based on Deep Learning [J]. Journal of System Simulation, 2021, 33(12): 2854-2863.
[12]	Feng Xiao, Zhang Hui, Zhou Rui, Qiao Lu, Wei Dong, Li Dandan, Zhang Yuyao, Zheng Guoqing. Variety Recognition Based on Deep Learning and Double-Sided Characteristics of Maize Kernel [J]. Journal of System Simulation, 2021, 33(12): 2983-2991.
[13]	Liu Xueqin, Liu Ning, Su Zhong, Wang Jingxiao, Yuan Chaojie. Research on Intelligent Gait Recognition Method Based on Plantar Pressure Perception [J]. Journal of System Simulation, 2021, 33(11): 2572-2578.
[14]	Du Jinlian, Li Shufei, Jin Xueyun. Research on the Network of 3D Smoke Flow Super-Resolution Data Generation [J]. Journal of System Simulation, 2021, 33(10): 2381-2389.
[15]	An Yu, Jiao Pengpeng, Bai Zixiu. Safety Evaluation and Risk Level Prediction of Driving Behavior Considering Multi-factors Influence [J]. Journal of System Simulation, 2021, 33(1): 118-126.