Fusing Rotation Angle Coding in Spherical Space for Human Action Recognition

doi:10.16182/j.issn1004731x.joss.23-1263

Abstract

Abstract:

The existing human action recognition methods focus more on the translation information such as the coordinates and displacements of skeleton structure, and pay less attention to the motion trend of skeleton structure and the rotation information representing the motion direction of joints and bones. A spatio-temporal convolutional neural network method combining the rotation angle coding in spherical space is introduced. The angle information with scale invariance is obtained by mapping the human action in three-dimensional spherical space, and the dynamic angular velocity information is extracted as the angle code to represent the rotation information of joints and bones in the action trajectory. A spatio-temporal feature extraction and co-occurrence module(STCN) is constructed to better capture the spatio-temporal features of data. A suitable fusion strategy is utilized to fuse the translation features and rotation features. The experimental results show that the rotation angle coding benefits the accuracy improvement of motion representation and the effectiveness of the spatio-temporal feature extraction and co-occurrence module.

Key words: human action recognition, skeleton data, rotation angle encoding, 3D spherical space, spatial-temporal feature

CLC Number:

TP391.9

Su Benyue, Zhu Bangguo, Guo Mengjuan, Sheng Min. Fusing Rotation Angle Coding in Spherical Space for Human Action Recognition[J]. Journal of System Simulation, 2024, 36(6): 1433-1441.

Figures/Tables 9

Fig. 1

Fig. 2

Fig. 3

Table 1

Validation of rotation angle coding

特征	NTU RGB+D60/%				NTU RGB+D120/%				单次训练时间/s
特征	CV	Add	CS	Add	Cset	Add	CS	Add	NTU-RGB+D 60 (CS)
P	91.9		86.8		76.9		76.5		21
+B	92.3	0.4	87.3	0.5	78.0	1.1	80.0	3.5	34
+joints_A	92.3	0.4	87.1	0.3	77.0	0.1	80.0	3.5	34
+B+ $A ̂$	92.5	0.6	87.8	1.0	78.9	2.0	80.4	3.9	38

Table 1

Table 2

Table 3

Table 4

Fig. 4

Fig. 5

References 34

1	Trelinski Jacek, Kwolek Bogdan. CNN-based and DTW Features for Human Activity Recognition on Depth Maps[J]. Neural Computing and Applications, 2021, 33(21): 14551-14563.
2	Shi Lei, Zhang Yifan, Cheng Jian, et al. Skeleton-based Action Recognition with Directed Graph Neural Networks[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2019: 7904-7913.
3	苏本跃, 孙满贞, 马庆, 等. 单视角下基于投影子空间视图的动作识别方法[J]. 系统仿真学报, 2023, 35(5): 1098-1108.
	Su Benyue, Sun Manzhen, Ma Qing, et al. Action Recognition Method Based on Projection Subspace Views Under Single Viewing Angle[J]. Journal of System Simulation, 2023, 35(5): 1098-1108.
4	Li Chao, Zhong Qiaoyong, Xie Di, et al. Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2018: 786-792.
5	Talha S A W, Hammouche M, Ghorbel E, et al. Features and Classification Schemes for View-invariant and Real-time Human Action Recognition[J]. IEEE Transactions on Cognitive and Developmental Systems, 2018, 10(4): 894-902.
6	Yan Sijie, Xiong Yuanjun, Lin Dahua. Spatial Temporal Graph Convolutional Networks for Skeleton-based Action Recognition[C]//Proceedings of the Thirty-second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2018: 912.
7	Guan Shannan, Lu Haiyan, Zhu Linchao, et al. AFE-CNN: 3D Skeleton-based Action Recognition with Action Feature Enhancement[J]. Neurocomputing, 2022, 514: 256-267.
8	Kim T S, Reiter A. Interpretable 3D Human Action Analysis with Temporal Convolutional Networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway, NJ, USA: IEEE, 2017: 1623-1631.
9	Si Chenyang, Jing Ya, Wang Wei, et al. Skeleton-based Action Recognition with Spatial Reasoning and Temporal Stack Learning[C]//Computer Vision – ECCV 2018. Cham: Springer International Publishing, 2018: 106-121.
10	Du Y, Fu Y, Wang L. Skeleton Based Action Recognition with Convolutional Neural Network[C]//2015 3rd IAPR Asian Conference on Pattern Recognition. 2015: 579-583.
11	Liu Shuhua, Bai Xiaoying, Fang Ming, et al. Mixed Graph Convolution and Residual Transformation Network for Skeleton-based Action Recognition[J]. Applied Intelligence, 2022, 52(2): 1544-1555.
12	赵瑛, 陆耀, 张健, 等. 基于深度神经网络的多视角人体动作识别[J]. 系统仿真学报, 2021, 33(5): 1019-1030.
	Zhao Ying, Lu Yao, Zhang Jian, et al. Multi-view Human Action Recognition Based on Deep Neural Network[J]. Journal of System Simulation, 2021, 33(5): 1019-1030.
13	Ke Qiuhong, Bennamoun M, An Senjian, et al. A New Representation of Skeleton Sequences for 3D Action Recognition[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2017: 4570-4579.
14	Xu Kailin, Ye Fanfan, Zhong Qiaoyong, et al. Topology-aware Convolutional Neural Network for Efficient Skeleton-based Action Recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2022: 2866-2874.
15	Huang Hong'en, Su Hang, Chang Zhigang, et al. Convolutional Neural Network with Adaptive Inferential Framework for Skeleton-based Action Recognition[J]. Journal of Visual Communication and Image Representation, 2020, 73: 102925.
16	Ofli F, Chaudhry R, Kurillo G, et al. Sequence of the Most Informative Joints (SMIJ): A New Representation for Human Skeletal Action Recognition[J]. Journal of Visual Communication and Image Representation, 2014, 25(1): 24-38.
17	Ömer Faruk İnce, Ibrahim Furkan Ince, Mustafa Eren Yıldırım, et al. Human Activity Recognition with Analysis of Angles Between Skeletal Joints Using a RGB-depth Sensor[J]. ETRI Journal, 2020, 42(1): 78-89.
18	Ibrahim Furkan Ince. Performance Boosting of Scale and Rotation Invariant Human Activity Recognition (HAR) with LSTM Networks Using Low Dimensional 3D Posture Data in Egocentric Coordinates[J]. Applied Sciences, 2020, 10(23): 8474.
19	Zatsiorsky V M. Kinetics of Human Motion[M]. Champaign, IL, USA: Human Kinetics, 2002.
20	Shahroudy A, Liu Jun, Ng T T, et al. NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2016: 1010-1019.
21	Liu Jun, Shahroudy Amir, Perez M, et al. NTU RGB+D 120: A Large-scale Benchmark for 3D Human Activity Understanding[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(10): 2684-2701.
22	Song S, Lan C, Xing J, et al. An end-to-end spatio-temporal Attention Model for Human Action Recognition from Skeleton Data[C]//Proceedings of the AAAI conference on artificial intelligence. 2017, 31(1).
23	Zhang Pengfei, Lan Cuiling, Xing Junliang, et al. View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data[C]//2017 IEEE International Conference on Computer Vision (ICCV). Piscataway, NJ, USA: IEEE, 2017: 2136-2145.
24	Zhang Pengfei, Xue Jianru, Lan Cuiling, et al. Adding Attentiveness to the Neurons in Recurrent Neural Networks[C]//Computer Vision-ECCV 2018. Cham: Springer International Publishing, 2018: 136-152.
25	Qin Yang, Mo Lingfei, Li Chenyang, et al. Skeleton-based Action Recognition by Part-aware Graph Convolutional Networks[J]. The Visual Computer, 2020, 36(3): 621-631.
26	Si Chenyang, Jing Ya, Wang Wei, et al. Skeleton-based Action Recognition with Hierarchical Spatial Reasoning and Temporal Stack Learning Network[J]. Pattern Recognition, 2020, 107: 107511.
27	Chen Han, Jiang Yifan, Ko Hanseok. Action Recognition with Domain Invariant Features of Skeleton Image[C]//2021 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). Piscataway, NJ, USA: IEEE, 2021: 1-7.
28	Su Benyue, Zhang Peng, Sun Manzhen, et al. Direction-guided Two-stream Convolutional Neural Networks for Skeleton-based Action Recognition[J]. Soft Computing, 2023, 27(16): 11833-11842.
29	Liu Jun, Shahroudy A, Xu D, et al. Skeleton-based Action Recognition Using Spatio-temporal LSTM Network with Trust Gates[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(12): 3007-3021.
30	Liu Jun, Wang Gang, Duan Lingyu, et al. Skeleton-based Human Action Recognition with Global Context-aware Attention LSTM Networks[J]. IEEE Transactions on Image Processing, 2018, 27(4): 1586-1599.
31	Li Maosen, Chen Siheng, Chen Xu, et al. Actional-structural Graph Convolutional Networks for Skeleton-based Action Recognition[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2019: 3590-3598.
32	Liu Xing, Li Yanshan, Xia Rongjie. Adaptive Multi-view Graph Convolutional Networks for Skeleton-based Action Recognition[J]. Neurocomputing, 2021, 444: 288-300.
33	Liu Mengyuan, Liu Hong, Chen Chen. Enhanced Skeleton Visualization for View Invariant Human Action Recognition[J]. Pattern Recognition, 2017, 68: 346-362.
34	Zhang Pengfei, Lan Cuiling, Zeng Wenjun, et al. Semantics-guided Neural Networks for Efficient Skeleton-based Human Action Recognition[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2020: 1109-1118.

方法	CV	Add	CS	Add
STA-LSTM^[22]	81.2	11.3	73.4	14.4
TCN^[8]	83.1	9.4	74.3	13.5
VA-LSTM^[23]	87.7	4.8	79.2	8.6
ElAtt-GRU^[24]	88.4	4.1	80.7	7.1
PA-GCN^[25]	82.7	9.8	80.4	7.4
ST-GCN^[6]	88.3	4.2	81.5	6.3
LSTM+GCN^[26]	90.2	2.3	84.8	3.0
DIF-CNN^[27]	85.8	6.7	81.0	6.8
DG-2sCNN^[28]	91.2	1.3	87.1	0.7
AFE-CNN^[7]	92.2	0.3	86.2	1.6
HCN^[4](base)	89.1	3.4	84.5	3.3
R-STCNN	92.5		87.8

方法	CS	Add	Cset	Add
ST-LSTM+Trust Gate^[29]	56.5	33.9	54.1	24.8
GCA-LSTM^[30]	58.3	22.1	59.2	19.7
Two-stream network^[21]	62.2	18.2	61.8	17.1
ST-GCN^[6]	70.7	9.7	73.2	5.7
AS-GCN^[31]	77.7	2.7	78.9	0
STF-GCN^[32]	76.7	3.7	79.0	-0.1
Synthesized CNN^[33]	60.3	20.1	63.2	15.7
SGN^[34]	79.2	79.2	81.5	-2.6
DG-2sCNN^[28]	78.0	2.4	81.0	-2.1
HCN(base)	76.5	3.9	76.6	2.3
R-STCNN	80.4		78.9

[1]	Su Benyue, Zhang Peng, Zhu Bangguo, Guo Mengjuan, Sheng Min. Human Action Recognition Based on Skeleton Edge Information Under Projection Subspace [J]. Journal of System Simulation, 2024, 36(3): 555-563.
[2]	Zhao Ying, Lu Yao, Zhang Jian, Liang Qidi, Long Wei. Multi-view Human Action Recognition Based on Deep Neural Network [J]. Journal of System Simulation, 2021, 33(5): 1019-1030.
[3]	Tang Chao, Zhang Miaohui, Li Wei, Cao Feng, Wang Xiaofeng, Tong Xiaohong. Fusing Local and Global Features for Human Action Recognition [J]. Journal of System Simulation, 2018, 30(7): 2497-2506.
[4]	Tang Chao, Zhang Miaohui, Li Wei, Cao Feng, Wang Xiaofeng, Tong Xiaohong. Human Action Recognition Based on Depth Image [J]. Journal of System Simulation, 2018, 30(5): 1641-1649.
[5]	Zhan Yongjie, Long Fei, Bu Yikun. Facial Expression Recognition with Independent Subspace Analysis Based Feature Learning [J]. Journal of System Simulation, 2015, 27(10): 2316-2319.
[6]	Shi Xiangbin, Liu Shuanpeng, Zhang Deyuan. Human Action Recognition Method Based on Key Frames [J]. Journal of System Simulation, 2015, 27(10): 2401-2408.