Research on Real-time Gesture Classification Algorithm Based on IMU and sEMG Mixed Signals

doi:10.16182/j.issn1004731x.joss.21-1117

Abstract

Abstract:

In order to improve the gesture classification accuracy of surface electromyography (sEMG), the mixed signal of attitude and sEMG is collected by inertial measurement unit (IMU) and EMG sensor, and a GRU-BiLSTM double-layer network real-time gesture classification algorithm is proposed. The first layer of gated recurrent unit (GRU) detects the mutation point of the initial mixed signal though energy combination operator feature and locates the starting point of the dynamic data. The second layer Bi-directional long short term memory (BiLSTM) classifies the motion state mixed signal into 10 gestures in two directions though energy kernel phase map feature. Through offline model optimization, the recognition time of the online GRU-BiLSTM double-layer classifier algorithm is less than 40 ms, the detection accuracy of mutation points is over 88.7%, the accuracy of gesture classification is 85%, and the information transmission rate (ITR) reaches 89.9 bits/min. Compared with machine learning-based classification algorithms, the algorithm has advantages in accuracy and computational efficiency.

Key words: surface electromyography(sEMG), inertial measurement unit(IMU), gated recurrent unit(GRU), Bi-directional long short term memory(BiLSTM), gesture classification

CLC Number:

TP249

Tao Wang, Yingnian Wu, Rui Yang, Yueying Sun. Research on Real-time Gesture Classification Algorithm Based on IMU and sEMG Mixed Signals[J]. Journal of System Simulation, 2023, 35(2): 359-371.

Figures/Tables 12

Fig. 1

Fig. 2

Fig. 3

Table 1

Energy core training characteristics

能量核相图特征	维度	参数范围
椭圆曲线参数	$α = [a, b, c, d, e] T$	[10, 100]
水平倾角/(°)	$θ$	±2~8
能量核幅值	[smallest_eigenval; smallest_eigenvec]	[10⁵，10⁸]
相位散点率/%	Phase scatter rate	1.8~7.6

Table 1

Fig. 4

Fig. 5

Table 2

Fig. 6

Fig. 7

Table 3

Fig. 8

Table 4

References 28

1	张晓俊, 李长勇. 基于深度学习多特征融合的手势分割识别算法[J]. 济南大学学报(自然科学版), 2022, 36(3): 286-291.
	Zhang Xiaojun, Li Changyong. Gesture Segmentation and Recognition Algorithm Based on Deep Learning Multi-feature Fusion[J]. Journal of University of Jinan(Science and Technology), 2022, 36(3): 286-291.
2	张维, 林泽一, 程坚, 等. 动态手势理解与交互综述[J]. 软件学报, 2021, 32(10): 3051-3067.
	Zhang Wei, Lin Zeyi, Cheng Jian, et al. Survey of Dynamic Hand Gesture Understanding and Interaction [J]. Journal of Software, 2021, 32(10): 3051- 3067.
3	刘光达, 董梦坤, 许蓝予, 等. 手臂疲劳时表面肌电信号特征[J]. 科学技术与工程, 2021, 21(25): 10690-10696.
	Liu Guangda, Dong Mengkun, Xu Lanyu, et al. Characteristic of Surface Electromyography Signal in Arm Fatigue[J]. Science Technology and Engineering, 2021, 21(25): 10690-10696.
4	Duan Feng, Yang Yikang. Recognizing Missing Electromyography Signal by Data Split Reorganization Strategy and Weight-Based Multiple Neural Network Voting Method[J]. IEEE Transactions on Neural Networks and Learning Systems(S2162-2388), 2021,33(5): 2070- 2079.
5	Lee S J, Hong K R, Lee T W, et al. A Study on the Reliability Comparison of Median Frequency and Spike Parameter and the Improved Spike Detection Algorithm for the Muscle Fatigue Measurement[J]. Transactions-Korean Institute of Electrical Engineers D(S1975-8359), 2004, 53(5): 380-388.
6	Hossen A, Muthuraman M, Raethjen J, et al. Discrimination of Parkinsonian Tremor from Essential Tremor by Implementation of a Wavelet-Based Soft-Decision Technique on EMG and Accelerometer Signals[J]. Biomedical Signal Processing and Control(S1746-8094), 2010, 5(3): 181-188.
7	Loconsole C, Cascarano G D, Brunetti A, et al. A Model-Free Technique Based on Computer Vision and sEMG for Classification in Parkinson's Disease by Using Computer-Assisted Handwriting Analysis[J]. Pattern Recognition Letters(S0167-8655), 2019, 121: 28-36.
8	Zhou Y, Bi Z, Ji M, et al. A Data-Driven Volitional EMG Extraction Algorithm During Functional Electrical Stimulation with Time Variant Parameters[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering(S1534-4320), 2020, 28(5): 1069-1080.
9	Zhang Guangyi, Ali Etemad. Capsule Attention for Multimodal EEG-EOG Representation Learning with Application to Driver Vigilance Estimation[J] IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021, 29: 1138-1149.
10	Qi J, Jiang G, Li G, et al. Surface EMG Hand Gesture Recognition System Based on PCA and GRNN[J]. Neural Computing and Applications(S1433-3058), 2020, 32(10): 6343-6351.
11	Rabin N, Kahlon M, Malayev S, et al. Classification of Human Hand Movements Based on EMG signals Using Nonlinear Dimensionality Reduction and Data Fusion Techniques[J]. Expert Systems with Applications(S0957-4174), 2020, 149: 113281.
12	李虹飞, 胡满红. 采摘机器人SEMG手势识别研究—基于RNN循环神经网络[J]. 农机化研究, 2022, 44(5): 212-216.
	Li Hongfei, Hu Manhong. SEMG Gesture Recognition of Picking Robot Based on RNN Cyclic Neural Network[J]. Journal of Agricultural Mechanization Research, 2022, 44(5): 212-216.
13	李自由, 赵新刚, 张弼, 等. 基于表面肌电的意图识别方法在非理想条件下的研究进展[J]. 自动化学报, 2021, 47(5): 955-969.
	Li Ziyou, Zhao Xingang, Zhang Bi, et al. Review of sEMG-Based Motion Intent Recognition Methods in Non-ideal Conditions[J]. Acta Automatica Sinica, 2021, 47(5): 955-969.
14	李建亮, 王从庆, 展文豪, 等. 基于表面肌电信号的人机交互软件设计[J]. 计算机测量与控制, 2021, 29(7): 171-174, 179.
	Li Jianliang, Wang Congqing, Zhan Wenhao, et al. Human-Computer Interaction Software Design Based on Surface EMG Signal[J]. Computer Measurement &Control, 2021, 29(7): 171-174, 179.
15	尤波, 陶守通, 黄玲, 等. 基于MATLAB和ADAMS的肌电假手联合仿真[J]. 系统仿真学报, 2017, 29(5): 957-964.
	You Bo, Tao Shoutong, Huang Ling, et al. Coordinated Simulation of EMG Prosthetic Hand Based on MATLAB and ADAMS[J]. Journal of System Simulation, 2017, 29(5): 957-964.
16	王远明, 张珺, 秦远辉, 等. 基于多特征融合的指挥手势识别方法研究[J]. 系统仿真学报, 2019, 31(2): 346-352.
	Wang Yuanming, Zhang Jun, Qin Yuanhui, et al. Gesture Recognition Method Based on Multi-feature Fusion[J]. Journal of System Simulation, 2019, 31(2): 346-352.
17	宋佳强, 裴晓敏, 赵强, 等. 基于组合能量特征的表面肌电信号手势识别算法[J]. 传感器与微系统, 2020, 39(6): 139-142.
	Song Jiaqiang, Pei Xiaomin, Zhao Qiang, et al. Surface EMG Signal Hand Motion Recognition Algorithm Based on Combinated Energy Characteristics[J]. Transducer and Microsystem Technologies, 2020, 39(6): 139-142.
18	蒋明峰, 王洪. 基于小波分析与神经网络相结合的表面肌电信号识别的研究[J]. 生物医学工程研究, 2005(1): 50-52.
	Jiang Mingfeng, Wang Hong. The Classification of Surface EMG Signal Based on Wavelet Transform and Neural Networks[J]. Journal of Biomedical Engineering Research, 2005(1): 50-52.
19	许留凯, 张克勤, 徐兆红, 等. 基于表面肌电信号能量核相图的卷积神经网络人体手势识别算法[J]. 生物医学工程学杂志, 2021, 38(4): 621-629.
	Xu Liukai, Zhang Keqin, Xu Zhaohong, et al. Convolutional Neural Network Human Gesture Recognition Algorithm Based on Phase Portrait of Surface Electromyography Energy Kernel[J]. Journal of Biomedical Engineering, 2021, 38(4): 621-629.
20	Guo Y, Gravina R, Gu X, et al. EMG-Based Abnormal Gait Detection and Recognition[C]//2020 IEEE International Conference on Human-Machine Systems (ICHMS). Rome: IEEE, 2020: 1-6.
21	冯媛硕, 宋吉江. 基于小波变换的信号特征与突变点检测算法研究[J]. 曲阜师范大学学报(自然科学版), 2015, 41(1): 76-80.
	Feng Yuanshuo, Song Jijiang. Research on the Wavelet-Based Algorithms for Signal Feature and Singularity Detection[J]. Journal of Qufu Normal University(Natural Science Edition), 2015, 41(1): 76-80.
22	张丹, 余义德, 王红萍. 基于时频滤波技术的时间相位信号检测方法[J]. 无线电工程, 2016, 46(12): 31-34, 42.
	Zhang Dan, Xu Yide, Wang Hongping. Time Phase Detection Method of Time-Frequency Filter Technique[J]. Radio Engineering, 2016, 46 (12): 31-34, 42.
23	吴娜. 基于表面肌电信号的双通道实时手势识别系统研究[D].杭州: 浙江大学, 2021.
	Wu Na. Research on Real-Time Gesture Recognition System Based on Dual-Channel sEMGs Signal[D]. Hangzhou: Zhejiang University, 2021.
24	石欣, 范智瑞, 张杰毅, 等. 基于LMS-随机森林的肌电信号下肢动作快速分类[J]. 仪器仪表学报, 2020, 41(6): 218-224.
	Shi Xin, Fan Zhirui, Zhang Jieyi, et al. Rapid Classification of Lower Limb Movements of EMG Signals Based on LMS-Random Forest[J]. Chinese Journal of Scientific Instrument, 2020, 41(6): 218-224.
25	叶科学. 基于表面肌电信号的脊柱侧弯症辅助诊断技术研究[D]. 南京: 南京航空航天大学, 2020.
	Ye Kexue. Research on Scoliosis Assisted Diagnosis Technology Based on sEMG[D]. Nanjing: Nanjing University of Aeronautics and Astronautics, 2020.
26	温倩. 基于表面肌电信号的人体行走模式识别[D]. 天津: 河北工业大学, 2015.
	Wen Qian. Walking Pattern Recognition Based on Surface Electromyography[D]. Tianjin: Hebei University of Technology, 2015.
27	Ketykó I, Kovács F, Varga K Z. Domain Adaptation for Semg-Based Gesture Recognition with Recurrent Neural Networks[C]//International Joint Conference on Neural Networks. Budapest: IEEE, 2019: 1-7.
28	李沿宏, 江茜, 邹可, 等. 融合注意力机制的多流卷积肌电手势识别网络[J]. 计算机应用研究, 2021, 38(11): 3258-3263.
	Li Yanhong, Jiang Qian, Zou Ke, et al. Multi-stream Convolutional Myoelectric Gesture Recognition Networks Fusing Attentional Mechanisms[J]. Application Research of Computers, 2021, 38(11): 3258-3263.

	不同时间窗	GRU平均识别精度/%	平均识别时间/ms	突变点检测误差率/%
水平方向	100 ms时间窗	90.7	2.81	1.44
	200 ms时间窗	92.3	2.96	1.08
	0.1~0.5 s	93.8	3.16	1.26
	0.1~1 s	89.2	3.22	2.71
垂直方向	100 ms时间窗	91.4	2.89	1.53
	200 ms时间窗	91.8	3.06	1.43
	0.1~0.5 s	90.7	3.77	1.76
	0.1~1 s	87.6	3.48	2.01

滑动窗采样输入段/s	BiLSTM网络
滑动窗采样输入段/s	单次动作准确率/%	随机动作准确率/%	平均识别时间/ms
0~0.4	80.4	78.7	28.6
0.1~0.5	86.7	80.4	30.2
0.2~0.6	83.6	77.3	34.8
0.3~0.7	80.7	71.9	28.9
0.4~0.8	78.2	69.3	24.3
0.5~0.9	67.1	57.1	21.1
0.6~1.0	58.6	54.9	20.7

分类方法	识别目标个数N/个	平均识别准确率P/%	识别时间t/s	信息传输率ITR/ (bits/min)
GRU-BiLSTM	20	85.4	2.13	89.90
BiLSTM	20	79.1	2.56	63.15
LMS-随机森林	4	97.3	2.14	49.85
SVM	6	97.5	2.67	52.98
Fisher判别	3	95.0	1.75	42.81
KNN	5	94.0	1.82	61.80
GengNet	12	87.3	3.01	51.76
MSCAnet	52	86.9	2.87	91.92

[1]	Yanqin Long, Guifang Qiao, Guangming Song, Ying Zhang, Linlin Cheng. Motion Simulation and Performance Analysis of 2D Variable Stiffness Snake-like Robot [J]. Journal of System Simulation, 2022, 34(4): 759-767.
[2]	Zhaoqiang Li, Shiyu Zhang. Research on 3D Path Planning Algorithm Based on Fast RRT Algorithm [J]. Journal of System Simulation, 2022, 34(3): 503-511.