Research on Motion Recognition and Tracking for Space Survey and Launch Tasks

doi:10.16182/j.issn1004731x.joss.21-0236

Abstract

Abstract:

Space survey and launch task has high precision and long cycle, and needs to be exposed to direct sunlight for a long time, so that the non-contact action calibration and comparison in a virtual working environment is an efficient way to improve the mission completion success rate. Aiming at the real-time motion tracking of aerospace personnel, a keyframe optimization algorithm for the action recognition is proposed. According to the bone data in the depth image, the bone features are extracted, and the keyframes are extracted by the feature threshold. The characteristic data of the keyframe is input into bi-directional long short-term memory to optimize the accuracy of the overall action recognition. Data-driven bone recognition and motion tracking can effectively identify the movements of the aerospace personnel to complete the related tasks more efficiently and safely.

Key words: space survey missions, bone identification, keyframe, bi-directional long short-term memory, motion tracking

CLC Number:

TP212

Bin Ren, Xiaoyu Wang. Research on Motion Recognition and Tracking for Space Survey and Launch Tasks[J]. Journal of System Simulation, 2022, 34(8): 1674-1681.

Figures/Tables 16

Fig. 1

Table 1

Fig. 2

Fig. 3

Table 2

Fig. 4

Fig. 5

Fig. 6

Table 3

Table 4

Table 5

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

References 25

1	Newcombe R A, Izadi S, Hilliges O, et al. KinectFusion: Real-Time Dense Surface Mapping and Tracking[C]// IEEE International Symposium on Mixed & Augmented Reality. Switzerland: IEEE, 2011: 127-136.
2	徐连瑞, 张锦明. 基于Kinect的室内环境建模[J]. 系统仿真学报, 2019, 31(12): 2643-2651.
	Xu Lianrui, Zhang Jinming. Modeling Indoor Environment with Kinect[J]. Journal of System Simulation, 2019, 31(12): 2643-2651.
3	Sun Y, Li C, Li G, et al. Gesture Recognition Based on Kinect and sEMG Signal Fusion[J]. Mobile Networks and Applications(S1383-469X), 2018, 23(4): 797-805.
4	Chang X, Ma Z, Lin M, et al. Feature Interaction Augmented Sparse Learning for Fast Kinect Motion Detection[J]. IEEE Transactions on Image Processing(S1941-0042), 2017, 26(8): 3911-3920.
5	Matthew Kyan, Sun Guoyu, Li Haiyan, et al. An Approach to Ballet Dance Training Through MS Kinect and Visualization in a CAVE Virtual Reality Environment[J]. ACM Transactions on Intelligent Systems and Technology(S2157-6904), 2015, 6(2): 1-37.
6	鲁明, 王真水, 田元, 等. 一种基于Kinect的虚拟现实姿态交互工具[J]. 系统仿真学报, 2013, 25(9): 2124-2130.
	Lu Ming, Wang Zhenshui, Tian Yuan, et al. Posture Interaction Tool for Virtual Reality System Based on Kinect[J]. Journal of System Simulation, 2013, 25(9): 2124-2130.
7	Zhang Mingshao, Zhang Zhou, Chang Yizhe, et al. Recent Developments in Game-Based Virtual Reality Educational Laboratories Using the Microsoft Kinect[J]. International Journal of Emerging Technologies in Learning(S1863-0383), 2018, 13(1): 138.
8	Min R, Kose N, Dugelay J. KinectFaceDB: A Kinect Database for Face Recognition[J]. IEEE Transactions on Systems Man & Cybernetics Systems(S2168-2216), 2017, 44(11): 1534-1548.
9	Wang C, Liu Z, Chan S C, et al. Superpixel-Based Hand Gesture Recognition with Kinect Depth Camera[J]. IEEE Transactions on Multimedia(S1520-9210), 2014, 17(1): 29-39.
10	Wang J, Zhang J, Honda K, et al. Audio-Visual Speech Recognition Integrating 3D Lip Information Obtained from the Kinect[J]. Multimedia Systems(S0942-4962), 2016, 22(3): 315-323.
11	Seddik B, Gazzah S, Najoua E B A. Human-Action Recognition using a Multi-Layered Fusion Scheme of Kinect Modalities[J]. Iet Computer Vision(S1751-9632), 2017, 11(7): 530-540.
12	Gao J, Chen Y, Li F. Kinect-Based Motion Recognition Tracking Robotic Arm Platform[J]. Intelligent Control and Automation(S2153-0653), 2019, 10(3): 79-89.
13	Ding I J, Shi J Y. Kinect Microphone Array-Based Speech and Speaker Recognition for the Exhibition Control of Humanoid Robots[J]. Computers & Electrical Engineering(S0045-7906), 2017, 62: 719-729.
14	Alexiadis Dimitrios S, Philip Kelly, Daras Petros, et al. Evaluating a Dancer's Performance Using Kinect-Based Skeleton Tracking[C]//19th ACM International Conference on Multimedia. USA: Association for Computing Machinery, 2011: 659-662.
15	尚华强. 基于Kinect的虚拟人物动作仿真研究[D]. 杭州: 杭州电子科技大学, 2013: 21-43.
	Shang Huaqiang. Simulation Study of Virtual Movement Based on Kinect[D]. Hangzhou: Hangzhou Dianzi University, 2013: 21-43.
16	敖琳. 基于Kinect骨骼信息的人体动作识别与行为分析[D]. 哈尔滨: 哈尔滨工程大学, 2018: 21-55.
	Ao Lin. Research on Human Action Recognition and Behavior Analysis Based on Kinect Human Skeleton[D]. Harbin: Harbin Engineering University, 2018: 21-55.
17	Zhu K, Wang R, Zhao Q, et al. A Cuboid CNN Model with an Attention Mechanism for Skeleton-Based Action Recognition[J]. IEEE Transactions on Multimedia (S1520-9210), 2020, 22(11): 2977-2989.
18	Donahue J, Hendricks L A, Guadarrama S, et al. Long-Term Recurrent Convolutional Networks for Visual Recognition and Description[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA: IEEE, 2015: 2625-2634.
19	Wu X, Ji Q. TBRNet: Two-Stream Bi-LSTM Residual Network for Video Action Recognition[J]. Algorithms(S1999-4893), 2020, 13(7): 169.
20	Ullah A, Ahmad J, Muhammad K, et al. Action Recognition in Video Sequences Using Deep Bi-Directional LSTM With CNN Features[J]. IEEE Access(S2169-3536), 2018, 6(99): 1155-1166.
21	Liu J, Wang G, Duan L Y, et al. Skeleton Based Human Action Recognition with Global Context-Aware Attention LSTM Networks[J]. IEEE Transactions on Image Processing(S1057-7149), 2018, 7(4): 1586-1599.
22	裴启程. 基于Kinect的人体行为识别研究[D]. 南京: 南京邮电大学, 2018: 21-30.
	Pei Qicheng. The Research on the Kinect-Based Action Recognition[D]. Nanjing: Nanjing University, 2018: 21-30.
23	Wang L, Zhao X, Liu Y. Skeleton Feature Fusion Based on Multi-Stream LSTM for Action Recognition[J]. IEEE Access(S2169-3536), 2018, 6: 50788-50800.
24	Bui A V, Nguyen T O. Multi-View Human Action Recognition Based on TSN Architecture Integrated with GRU[J]. Procedia Computer Science(S1877-0509), 2020, 176: 948-955.
25	张洁庆, 郭敏, 肖冰. 基于GoogLeNet和双层GRU的图像描述[J].陕西师范大学学报(自然科学版), 2021, 49(1): 68-73.
	Zhang Jieqing, Guo Min, Xiao Bing. Image Description Based on GoogLeNet and Double-Layer GRU[J]. Journal of Shaanxi Normal University(Natural Science Edition), 2021, 49(1): 68-73.

编号	部位	编号	部位	编号	部位	编号	部位
1	脊柱底部	8	左手	15	左脚踝	22	左指尖
2	脊柱中部	9	右肩	16	左脚	23	左拇指
3	脖子	10	右肘	17	右盆骨	24	右指尖
4	头	11	右腕	18	右膝	25	右拇指
5	左肩	12	右手	19	右脚踝
6	左肘	13	左盆骨	20	右脚
7	左腕	14	左膝	21	上脊柱

符号	夹角	符号	夹角
α_LB	左肩左臂	α_LD	左髋左大腿
α_LH	左臂左前臂	α_LX	左大腿左小腿
α_RB	右肩右臂	α_RD	右髋右大腿
α_RH	右臂右前臂	α_RX	右大腿右小腿

sum_s	动作1	动作2	动作3	动作4	动作5	动作6	动作7
0	0	0	0	0	0	0	0
0.10	0.96	0.84	0.90	0.53	0.78	0.76	0.92
0.11	0.93	0.75	0.91	0.43	0.80	0.75	0.95
0.12	0.88	0.73	0.87	0.55	0.73	0.74	0.81
0.13	0.93	0.81	0.92	0.74	0.76	0.83	0.82
0.14	0.93	0.85	0.93	0.76	0.79	0.70	0.91
0.15	0.96	0.79	0.95	0.72	0.78	0.74	0.94
0.16	0.92	0.86	0.85	0.73	0.68	0.79	0.92
0.17	0.87	0.90	0.94	0.31	0.70	0.78	0.83
0.18	0.89	0.86	0.89	0.41	0.66	0.79	0.85
0.19	0.97	0.86	0.94	0.33	0.64	0.82	0.80
0.20	0.90	0.86	0.96	0.49	0.81	0.73	0.95
0.21	0.96	0.83	0.96	0.57	0.74	0.74	0.74
0.22	0.94	0.84	0.90	0.29	0.84	0.74	0.89
0.23	0.95	0.84	0.95	0.53	0.89	0.60	0.77
0.24	0.90	0.77	0.93	0.42	0.58	0.74	0.91
0.25	0.97	0.89	0.92	0.52	0.85	0.64	0.93

训练模型	动作1	动作2	动作3	动作4	动作5	动作6	动作7
LSTM	0.85	0.78	0.97	0.44	0.71	0.70	0.85
双层LSTM	0.93	0.87	0.94	0.55	0.81	0.70	0.73
GRU	0.64	0.84	0.94	0.12	0.70	0.81	0.74
双层GRU	0.34	0.86	0.96	0.32	0.69	0.65	0.87
LRCN	null	null	null	null	null	null	0
Bi-LSTM	0.93	0.85	0.96	0.76	0.79	0.70	0.91

训练模型	识别率	训练模型	识别率
LSTM	0.76	双层GRU	0.67
双层LSTM	0.79	LRCN	0
GRU	0.68	Bi-LSTM	0.84