A Multimodal Residual Spatial-temporal Fusion Model Based on Automatic Sleep Classification

doi:10.16182/j.issn1004731x.joss.23-0588

Abstract

Abstract:

Highly accurate sleep staging plays a crucial role in correctly assessing sleep conditions. Aiming at the problem that the existing convolutional network cannot obtain the topological characteristics of physiological signals, a sleep staging algorithm based on multi-modal residual spatio-temporal fusion is proposed. Time-frequency images and spatio-temporal images are obtained using short-time Fourier transform and adaptive map convolution, which are converted into high-dimensional feature vectors; lightweight interaction of feature information flow is realized through time-frequency feature and spatio-temporal feature extraction modules; the feature enhancement fusion module fuses feature information to outputs sleep staging results. The results show that the model has a high accuracy. On the ISRUC-S3 data set, the overall accuracy is 85.3%, the F1 score is 83.8%, Cohen’s kappa is 81%, and the N1 stage accuracy reaches 69.81%. Experiments on the ISRUC-S1 dataset demonstrate the generality of the model.

Key words: sleep staging, multi-view fusion, graph convolutional network, deep learning, electroencephalogram

CLC Number:

TP391

Guo Yecai, Tong Shuang. A Multimodal Residual Spatial-temporal Fusion Model Based on Automatic Sleep Classification[J]. Journal of System Simulation, 2024, 36(9): 2065-2074.

Figures/Tables 13

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Table 1

Table 2

Fig. 5

Table 3

Table 4

Fig. 6

Fig. 7

Fig. 8

Fig. 9

References 26

1	Kohn T P, Kohn J R, Haney N M, et al. The Effect of Sleep on Men's Health[J]. Translational Andrology and Urology, 2020, 9(S2): S178-S185.
2	Guillot Antoine, Thorey Valentin. RobustSleepNet: Transfer Learning for Automated Sleep Staging at Scale[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021, 29: 1441-1451.
3	Wolpert E A. A Manual of Standardized Terminology, Techniques and Scoring System for Sleep Stages of Human Subjects[J]. Archives of General Psychiatry, 1969, 20(2): 246-247.
4	Xiao Meng, Yan Hong, Song Jinzhong, et al. Sleep Stages Classification Based on Heart Rate Variability and Random Forest[J]. Biomedical Signal Processing and Control, 2013, 8(6): 624-633.
5	Bansal Aayushi, Sharma Rewa, Kathuria Mamta. A Systematic Review on Data Scarcity Problem in Deep Learning: Solution and Applications[J]. ACM Computing Surveys, 2022, 54(10S): 208.
6	Md Mosheyur Rahman, Mohammed Imamul Hassan Bhuiyan, Ahnaf Rashik Hassan. Sleep Stage Classification Using Single-channel EOG[J]. Computers in Biology and Medicine, 2018, 102: 211-220.
7	Jameer Basha A, Saravana Balaji B, Poornima S, et al. Support Vector Machine and Simple Recurrent Network Based Automatic Sleep Stage Classification of Fuzzy Kernel[J]. Journal of Ambient Intelligence and Humanized Computing, 2021, 12(6): 6189-6197.
8	Pei Wei, Li Yan, Siuly S, et al. A Hybrid Deep Learning Scheme for Multi-channel Sleep Stage Classification[J]. Computers, Materials & Continua, 2022, 71(1): 889-905.
9	Phan H, Andreotti F, Cooray N, et al. SeqSleepNet: End-to-end Hierarchical Recurrent Neural Network for Sequence-to-sequence Automatic Sleep Staging[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 27(3): 400-410.
10	Wenpeng Neng, Lu Jun, Xu Lei. CCRRSleepNet: A Hybrid Relational Inductive Biases Network for Automatic Sleep Stage Classification on Raw Single-channel EEG[J]. Brain Sciences, 2021, 11(4): 456.
11	Supratak A, Dong Hao, Wu Chao, et al. DeepSleepNet: A Model for Automatic Sleep Stage Scoring Based on Raw Single-channel EEG[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017, 25(11): 1998-2008.
12	Perslev Mathias, Darkner Sune, Kempfner Lykke, et al. U-sleep: Resilient High-frequency Sleep Staging[J]. npj Digital Medicine, 2021, 4(1): 72.
13	Jia Z, Lin Y, Wang J, et al. Salientsleepnet: Multimodal Salient Wave Detection Network for Sleep Staging[C]//2021 30th International Joint Conference on Artificial Intelligence (IJCAI). Montreal: Morgan Kaufmann, 2021: 1-10.
14	Kong G, Li C, Peng H, et al. EEG-based Sleep Stage Classification via Neural Architecture Search[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2023, 31: 1075-1085.
15	Zhou Jie, Cui Ganqu, Hu Shengding, et al. Graph Neural Networks: A Review of Methods and Applications[J]. AI Open, 2020, 1: 57-81.
16	Zhang Si, Tong Hanghang, Xu Jiejun, et al. Graph Convolutional Networks: A Comprehensive Review[J]. Computational Social Networks, 2019, 6(1): 11.
17	Jia Ziyu, Lin Youfang, Wang Jing, et al. GraphSleepNet: Adaptive Spatial-temporal Graph Convolutional Networks for Sleep Stage Classification[C]//Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. New York, NY, USA: ACM, 2021: 1324-1330.
18	Jia Ziyu, Lin Youfang, Wang Jing, et al. Multi-view Spatial-temporal Graph Convolutional Networks with Domain Generalization for Sleep Stage Classification[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021, 29: 1977-1986.
19	Ji Xiaopeng, Li Yan, Wen Peng. Jumping Knowledge Based Spatial-temporal Graph Convolutional Networks for Automatic Sleep Stage Classification[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 1464-1472.
20	Li Miaofen, Liu Youmin, Zhi Shaodan, et al. Short-time Fourier Transform Using Odd Symmetric Window Function[J]. Journal of Dynamics, Monitoring and Diagnostics, 2022, 1(1): 37-45.
21	Zhou Haoran, Feng Yidan, Fang Mingsheng, et al. Adaptive Graph Convolution for Point Cloud Analysis[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway, NJ, USA: IEEE, 2021: 4945-4954.
22	Supratak Akara, Guo Yike. TinySleepNet: An Efficient Deep Learning Model for Sleep Stage Scoring Based on Raw Single-channel EEG[C]//2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Piscataway, NJ, USA: IEEE, 2020: 641-644.
23	Cai Xiyang, Jia Ziyu, Jiao Zehui. Two-stream Squeeze-and-excitation Network for Multi-modal Sleep Staging[C]//2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Piscataway, NJ, USA: IEEE, 2021: 1262-1265.
24	Xu Ziliang, Yang Xuejuan, Sun Jinbo, et al. Sleep Stage Classification Using Time-frequency Spectra from Consecutive Multi-time Points[J]. Frontiers in Neuroscience, 2020, 14: 14.
25	Cai Qing, Gao Zhongke, An Jianpeng, et al. A Graph-temporal Fused Dual-input Convolutional Neural Network for Detecting Sleep Stages from EEG Signals[J]. IEEE Transactions on Circuits and Systems II: Express Briefs, 2021, 68(2): 777-781.
26	Khalighi Sirvan, Sousa Teresa, José Moutinho Santos, et al. ISRUC-sleep: A Comprehensive Public Dataset for Sleep Researchers[J]. Computer Methods and Programs in Biomedicine, 2016, 124: 180-192.

数据集	阶段	阶段数	比例/%	总数
S1	W	20 098	23.05	87 187
	N1	11 062	12.69
	N2	27 511	31.55
	N3	17 251	19.79
	REM	11 265	12.92
S3	W	1 651	19.31	8 549
	N1	1 215	14.21
	N2	2 609	30.52
	N3	2 014	23.56
	REM	1 060	12.40

超参数	值
训练次数	60
训练批次	32
优化器	Adam
学习率	0.000 02
下采样率	0.5
自适应图卷积正则化参数	0.000 5
切比雪夫多项式阶数	9
卷积核大小	3×3

睡眠阶段	预测值					性能指标/%
睡眠阶段	W	N1	N2	N3	REM	Pre	Re
W	1 518	99	24	4	6	89.08	91.94
N1	140	733	215	2	125	69.81	60.33
N2	31	159	2 260	118	38	82.97	86.62
N3	1	1	189	1 822	1	93.58	90.47
REM	8	61	36	1	954	84.88	90.00

数据集	模型	总体性能指标			各阶段F1分数
数据集	模型	A	F₁	K	W	N1	N2	N3	REM
ISRUC-S3	DeepSleep^[11]	78.8	77.9	73.0	88.7	60.2	74.6	85.8	80.8
	U-Sleep^[12]	77.0	76.4	—	90.0	55.0	78.0	74.0	85.0
	SalientSleepNet^[13]	79.9	78.6	74.2	86.0	58.9	79.3	88.6	80.2
	GraphSleepNet^[17]	79.9	78.7	74.1	87.8	57.4	77.6	86.4	84.1
	MSTGCN^[18]	82.1	80.8	76.9	89.4	59.6	80.6	89.0	85.6
	JK-STGCN^[19]	83.1	81.4	78.2	90.0	59.8	82.6	90.1	84.7
	MJ-Sleep	85.3	83.8	81.0	90.5	64.7	84.8	92.0	87.4
ISRUC-S1	DeepSleep^[11]	71.7	69.1	63.8	82.3	46.6	73.8	80.9	62.1
	U-Sleep^[12]	77.0	77.0	—	89.0	52.0	79.0	77.0	88.0
	SalientSleepNet^[13]	81.5	80.1	76.2	89.9	57.0	80.0	87.8	85.7
	GraphSleepNet^[17]	78.6	75.4	72.3	88.4	43.7	77.5	83.8	83.5
	MSTGCN^[18]	80.4	78.5	74.8	88.7	54.5	79.1	87.2	83.2
	JK-STGCN^[19]	82.0	79.8	76.7	89.5	55.0	81.1	88.3	85.0
	MJ-Sleep	83.2	81.9	78.4	90.1	61.5	82.8	90.3	84.8

[1]	Liu Zesen, Bi Sheng, Guo Chuanhong, Wang Yankui, Dong Min. Deep Learning Based Local Path Planning Method for Moving Robots [J]. Journal of System Simulation, 2024, 36(5): 1199-1210.
[2]	Wei Jinyang, Wang Keping, Yang Yi, Fei Shumin. Incremental Image Dehazing Algorithm Based on Multiple Transfer Attention [J]. Journal of System Simulation, 2024, 36(4): 969-980.
[3]	Li Chen, He Ming, Dong Chen, Li Wei. Action Recognition Model of Directed Attention Based on Cosine Similarity [J]. Journal of System Simulation, 2024, 36(1): 67-82.
[4]	Zhang Fengquan, Cao Duo, Ma Xiaohan, Chen Baijun, Zhang Jiangxiao. Style Transfer Network for Generating Opera Makeup Details [J]. Journal of System Simulation, 2023, 35(9): 2064-2076.
[5]	Benyue Su, Manzhen Sun, Qing Ma, Min Sheng. Action Recognition Method Based on Projection Subspace Views under Single Viewing Angle [J]. Journal of System Simulation, 2023, 35(5): 1098-1108.
[6]	Yu Du, Xinquan Yang, Jianhua Zhang, Suchun Yuan, Huachao Xiao, Jingjing Yuan. Modulation Recognition Method of Mixed Signal Based on Intelligent Analysis of Cyclic Spectrum Section [J]. Journal of System Simulation, 2023, 35(1): 146-157.
[7]	Dayong Zhang, Jingyu Yang, Xi Wu. Research on Intelligent Prediction Method of Wargaming Air Mission [J]. Journal of System Simulation, 2023, 35(1): 212-220.
[8]	Huimin Chai, Yong Zhang, Xinyue Li, Yanan Song. Aerial Target Threat Assessment Method based on Deep Learning [J]. Journal of System Simulation, 2022, 34(7): 1459-1467.
[9]	Xu Kang, Xiaofeng Zhang. Radar Remote Sensing Data Augmentation Method Based on Generative Adversarial Network [J]. Journal of System Simulation, 2022, 34(4): 920-927.
[10]	Jiarui Liu, Guotian Yang, Xiaowei Wang. A Wind Turbine Fault Diagnosis Method Based on Siamese Deep Neural Network [J]. Journal of System Simulation, 2022, 34(11): 2348-2358.
[11]	Rongxiu Lu, Bihao Zhang, Zhenlong Mo. Fatigue Detection Method Based on Facial Features and Head Posture [J]. Journal of System Simulation, 2022, 34(10): 2279-2292.
[12]	Weiguo Tong, Shichao Zeng, Lifeng Zhang, Zhe Hou, Jiayue Guo. Electrical Resistance Tomography and Flow Pattern Identification Method Based on Deep Residual Neural Network [J]. Journal of System Simulation, 2022, 34(09): 2028-2036.
[13]	Lin Shuo, An Lei, Gao Zhijun, Shan Dan, Shang Wenli. Research on Intrusion Detection Based on Stacked Autoencoder and Long-short Memory [J]. Journal of System Simulation, 2021, 33(6): 1288-1296.
[14]	Cheng Wencong, Shi Xiaokang, Wang Zhigang. Creating Synthetic Satellite Cloud Data Based on GAN Method [J]. Journal of System Simulation, 2021, 33(6): 1297-1306.
[15]	Dong Shuqin, Zhang Bin. Network Traffic Anomaly Detection Method for Imbalanced Data [J]. Journal of System Simulation, 2021, 33(3): 679-689.