融合球空间下旋转角度编码的人体动作识别

doi:10.16182/j.issn1004731x.joss.23-1263

摘要/Abstract

摘要：

针对现有的人体动作识别方法较多考虑骨架结构的坐标和位移等平移信息，较少关注代表骨架结构的运动趋势以及代表关节、骨骼运动方向的旋转信息，提出一种融合球空间下旋转角度编码的时空卷积神经网络方法。通过人体动作在三维球空间中的映射，获取具有尺度不变性的角度信息，提取其动态角速度信息作为角度编码，表征动作轨迹中关节点和骨骼边的旋转信息；构建了时空特征提取与共现模块来更好地捕获数据的时空特征；用合适的融合策略对平移特征和旋转特征进行运动特征融合。实验结果证明了旋转角度编码有利于提升运动表征的准确性，以及时空特征提取与共现模块的有效性。

关键词: 人体动作识别, 骨架数据, 旋转角度编码, 3D球空间, 时空特征

Abstract:

The existing human action recognition methods focus more on the translation information such as the coordinates and displacements of skeleton structure, and pay less attention to the motion trend of skeleton structure and the rotation information representing the motion direction of joints and bones. A spatio-temporal convolutional neural network method combining the rotation angle coding in spherical space is introduced. The angle information with scale invariance is obtained by mapping the human action in three-dimensional spherical space, and the dynamic angular velocity information is extracted as the angle code to represent the rotation information of joints and bones in the action trajectory. A spatio-temporal feature extraction and co-occurrence module(STCN) is constructed to better capture the spatio-temporal features of data. A suitable fusion strategy is utilized to fuse the translation features and rotation features. The experimental results show that the rotation angle coding benefits the accuracy improvement of motion representation and the effectiveness of the spatio-temporal feature extraction and co-occurrence module.

Key words: human action recognition, skeleton data, rotation angle encoding, 3D spherical space, spatial-temporal feature

中图分类号:

TP391.9

苏本跃,朱邦国,郭梦娟等 . 融合球空间下旋转角度编码的人体动作识别[J]. 系统仿真学报, 2024, 36(6): 1433-1441.

Su Benyue,Zhu Bangguo,Guo Mengjuan,et al . Fusing Rotation Angle Coding in Spherical Space for Human Action Recognition[J]. Journal of System Simulation, 2024, 36(6): 1433-1441.

图/表 9

图1

图2

图3

表1

旋转角度编码有效性验证

特征	NTU RGB+D60/%				NTU RGB+D120/%				单次训练时间/s
特征	CV	Add	CS	Add	Cset	Add	CS	Add	NTU-RGB+D 60 (CS)
P	91.9		86.8		76.9		76.5		21
+B	92.3	0.4	87.3	0.5	78.0	1.1	80.0	3.5	34
+joints_A	92.3	0.4	87.1	0.3	77.0	0.1	80.0	3.5	34
+B+ $A ̂$	92.5	0.6	87.8	1.0	78.9	2.0	80.4	3.9	38

表1

表2

表3

表4

图4

图5

参考文献 34

1	Trelinski Jacek, Kwolek Bogdan. CNN-based and DTW Features for Human Activity Recognition on Depth Maps[J]. Neural Computing and Applications, 2021, 33(21): 14551-14563.
2	Shi Lei, Zhang Yifan, Cheng Jian, et al. Skeleton-based Action Recognition with Directed Graph Neural Networks[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2019: 7904-7913.
3	苏本跃, 孙满贞, 马庆, 等. 单视角下基于投影子空间视图的动作识别方法[J]. 系统仿真学报, 2023, 35(5): 1098-1108.
	Su Benyue, Sun Manzhen, Ma Qing, et al. Action Recognition Method Based on Projection Subspace Views Under Single Viewing Angle[J]. Journal of System Simulation, 2023, 35(5): 1098-1108.
4	Li Chao, Zhong Qiaoyong, Xie Di, et al. Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2018: 786-792.
5	Talha S A W, Hammouche M, Ghorbel E, et al. Features and Classification Schemes for View-invariant and Real-time Human Action Recognition[J]. IEEE Transactions on Cognitive and Developmental Systems, 2018, 10(4): 894-902.
6	Yan Sijie, Xiong Yuanjun, Lin Dahua. Spatial Temporal Graph Convolutional Networks for Skeleton-based Action Recognition[C]//Proceedings of the Thirty-second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2018: 912.
7	Guan Shannan, Lu Haiyan, Zhu Linchao, et al. AFE-CNN: 3D Skeleton-based Action Recognition with Action Feature Enhancement[J]. Neurocomputing, 2022, 514: 256-267.
8	Kim T S, Reiter A. Interpretable 3D Human Action Analysis with Temporal Convolutional Networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway, NJ, USA: IEEE, 2017: 1623-1631.
9	Si Chenyang, Jing Ya, Wang Wei, et al. Skeleton-based Action Recognition with Spatial Reasoning and Temporal Stack Learning[C]//Computer Vision – ECCV 2018. Cham: Springer International Publishing, 2018: 106-121.
10	Du Y, Fu Y, Wang L. Skeleton Based Action Recognition with Convolutional Neural Network[C]//2015 3rd IAPR Asian Conference on Pattern Recognition. 2015: 579-583.
11	Liu Shuhua, Bai Xiaoying, Fang Ming, et al. Mixed Graph Convolution and Residual Transformation Network for Skeleton-based Action Recognition[J]. Applied Intelligence, 2022, 52(2): 1544-1555.
12	赵瑛, 陆耀, 张健, 等. 基于深度神经网络的多视角人体动作识别[J]. 系统仿真学报, 2021, 33(5): 1019-1030.
	Zhao Ying, Lu Yao, Zhang Jian, et al. Multi-view Human Action Recognition Based on Deep Neural Network[J]. Journal of System Simulation, 2021, 33(5): 1019-1030.
13	Ke Qiuhong, Bennamoun M, An Senjian, et al. A New Representation of Skeleton Sequences for 3D Action Recognition[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2017: 4570-4579.
14	Xu Kailin, Ye Fanfan, Zhong Qiaoyong, et al. Topology-aware Convolutional Neural Network for Efficient Skeleton-based Action Recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2022: 2866-2874.
15	Huang Hong'en, Su Hang, Chang Zhigang, et al. Convolutional Neural Network with Adaptive Inferential Framework for Skeleton-based Action Recognition[J]. Journal of Visual Communication and Image Representation, 2020, 73: 102925.
16	Ofli F, Chaudhry R, Kurillo G, et al. Sequence of the Most Informative Joints (SMIJ): A New Representation for Human Skeletal Action Recognition[J]. Journal of Visual Communication and Image Representation, 2014, 25(1): 24-38.
17	Ömer Faruk İnce, Ibrahim Furkan Ince, Mustafa Eren Yıldırım, et al. Human Activity Recognition with Analysis of Angles Between Skeletal Joints Using a RGB-depth Sensor[J]. ETRI Journal, 2020, 42(1): 78-89.
18	Ibrahim Furkan Ince. Performance Boosting of Scale and Rotation Invariant Human Activity Recognition (HAR) with LSTM Networks Using Low Dimensional 3D Posture Data in Egocentric Coordinates[J]. Applied Sciences, 2020, 10(23): 8474.
19	Zatsiorsky V M. Kinetics of Human Motion[M]. Champaign, IL, USA: Human Kinetics, 2002.
20	Shahroudy A, Liu Jun, Ng T T, et al. NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2016: 1010-1019.
21	Liu Jun, Shahroudy Amir, Perez M, et al. NTU RGB+D 120: A Large-scale Benchmark for 3D Human Activity Understanding[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(10): 2684-2701.
22	Song S, Lan C, Xing J, et al. An end-to-end spatio-temporal Attention Model for Human Action Recognition from Skeleton Data[C]//Proceedings of the AAAI conference on artificial intelligence. 2017, 31(1).
23	Zhang Pengfei, Lan Cuiling, Xing Junliang, et al. View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data[C]//2017 IEEE International Conference on Computer Vision (ICCV). Piscataway, NJ, USA: IEEE, 2017: 2136-2145.
24	Zhang Pengfei, Xue Jianru, Lan Cuiling, et al. Adding Attentiveness to the Neurons in Recurrent Neural Networks[C]//Computer Vision-ECCV 2018. Cham: Springer International Publishing, 2018: 136-152.
25	Qin Yang, Mo Lingfei, Li Chenyang, et al. Skeleton-based Action Recognition by Part-aware Graph Convolutional Networks[J]. The Visual Computer, 2020, 36(3): 621-631.
26	Si Chenyang, Jing Ya, Wang Wei, et al. Skeleton-based Action Recognition with Hierarchical Spatial Reasoning and Temporal Stack Learning Network[J]. Pattern Recognition, 2020, 107: 107511.
27	Chen Han, Jiang Yifan, Ko Hanseok. Action Recognition with Domain Invariant Features of Skeleton Image[C]//2021 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). Piscataway, NJ, USA: IEEE, 2021: 1-7.
28	Su Benyue, Zhang Peng, Sun Manzhen, et al. Direction-guided Two-stream Convolutional Neural Networks for Skeleton-based Action Recognition[J]. Soft Computing, 2023, 27(16): 11833-11842.
29	Liu Jun, Shahroudy A, Xu D, et al. Skeleton-based Action Recognition Using Spatio-temporal LSTM Network with Trust Gates[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(12): 3007-3021.
30	Liu Jun, Wang Gang, Duan Lingyu, et al. Skeleton-based Human Action Recognition with Global Context-aware Attention LSTM Networks[J]. IEEE Transactions on Image Processing, 2018, 27(4): 1586-1599.
31	Li Maosen, Chen Siheng, Chen Xu, et al. Actional-structural Graph Convolutional Networks for Skeleton-based Action Recognition[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2019: 3590-3598.
32	Liu Xing, Li Yanshan, Xia Rongjie. Adaptive Multi-view Graph Convolutional Networks for Skeleton-based Action Recognition[J]. Neurocomputing, 2021, 444: 288-300.
33	Liu Mengyuan, Liu Hong, Chen Chen. Enhanced Skeleton Visualization for View Invariant Human Action Recognition[J]. Pattern Recognition, 2017, 68: 346-362.
34	Zhang Pengfei, Lan Cuiling, Zeng Wenjun, et al. Semantics-guided Neural Networks for Efficient Skeleton-based Human Action Recognition[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2020: 1109-1118.

方法	CV	Add	CS	Add
STA-LSTM^[22]	81.2	11.3	73.4	14.4
TCN^[8]	83.1	9.4	74.3	13.5
VA-LSTM^[23]	87.7	4.8	79.2	8.6
ElAtt-GRU^[24]	88.4	4.1	80.7	7.1
PA-GCN^[25]	82.7	9.8	80.4	7.4
ST-GCN^[6]	88.3	4.2	81.5	6.3
LSTM+GCN^[26]	90.2	2.3	84.8	3.0
DIF-CNN^[27]	85.8	6.7	81.0	6.8
DG-2sCNN^[28]	91.2	1.3	87.1	0.7
AFE-CNN^[7]	92.2	0.3	86.2	1.6
HCN^[4](base)	89.1	3.4	84.5	3.3
R-STCNN	92.5		87.8

方法	CS	Add	Cset	Add
ST-LSTM+Trust Gate^[29]	56.5	33.9	54.1	24.8
GCA-LSTM^[30]	58.3	22.1	59.2	19.7
Two-stream network^[21]	62.2	18.2	61.8	17.1
ST-GCN^[6]	70.7	9.7	73.2	5.7
AS-GCN^[31]	77.7	2.7	78.9	0
STF-GCN^[32]	76.7	3.7	79.0	-0.1
Synthesized CNN^[33]	60.3	20.1	63.2	15.7
SGN^[34]	79.2	79.2	81.5	-2.6
DG-2sCNN^[28]	78.0	2.4	81.0	-2.1
HCN(base)	76.5	3.9	76.6	2.3
R-STCNN	80.4		78.9

[1]	赵瑛, 陆耀, 张健, 梁启弟, 龙炜. 基于深度神经网络的多视角人体动作识别[J]. 系统仿真学报, 2021, 33(5): 1019-1030.
[2]	胡冬梅, 吴建平, 田琨, 廖律超, 杜怡曼. 北京重霾期空气污染物、气象和交通的时空特征[J]. 系统仿真学报, 2019, 31(3): 403-412.
[3]	刘望, 孙金玉, 马世伟. 基于时空特征金字塔网络的动作时序检测方法[J]. 系统仿真学报, 2019, 31(11): 2382-2387.
[4]	廖鹰, 易卓, 胡晓峰, 田园, 陶九阳. 基于三维卷积神经网络的战场聚集行为预测[J]. 系统仿真学报, 2018, 30(3): 801-808.
[5]	詹永杰, 龙飞, 卜轶坤. 基于独立子空间分析特征学习的表情识别[J]. 系统仿真学报, 2015, 27(10): 2316-2319.
[6]	石祥滨, 刘拴朋, 张德园. 基于关键帧的人体动作识别方法[J]. 系统仿真学报, 2015, 27(10): 2401-2408.