Gesture Recognition for Dynamic Vision Sensor Based on Multi-dimensional Projection Spatiotemporal Event Frame

doi:10.16182/j.issn1004731x.joss.23-0223

Abstract

Abstract:

Vision-based gesture recognition is a commonly used means of human-computer interaction in the fields of virtual reality and game simulation. In practical applications, rapid changes in gesture movements will lead to blurred imaging with traditional RGB cameras or depth cameras, which brings great challenges to gesture recognition. To solve the above problems, a dynamic visual data gesture recognition method based on a multi-dimensional projection spatiotemporal event frame (STEF) is proposed by a using dynamic vision sensor to capture high-speed gesture movement information. The spatiotemporal information is embedded in the data projection surface and fused to form a multi-dimensional projection STEF, which overcomes the limitation of the time-domain information loss of the existing event frame expression method of dynamic visual information and improves the feature expression ability of dynamic visual sensing data. On this basis, advanced spiking neural networks are used to classify STEFs to realize gesture recognition. The recognition accuracy of the above method on the public dataset reaches 96.67%, which is better than similar methods, indicating that the proposed method can significantly improve the accuracy of gesture recognition in dynamic visual sensing data.

Key words: dynamic vision sensor, gesture recognition, multi-dimensional projection, spatiotemporal event frame, spiking neural network

CLC Number:

TP391

Kang Lai, Zhang Yakun. Gesture Recognition for Dynamic Vision Sensor Based on Multi-dimensional Projection Spatiotemporal Event Frame[J]. Journal of System Simulation, 2024, 36(3): 649-658.

Figures/Tables 9

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Table 1

Fig. 5

Fig. 6

Fig. 7

Fig. 8

References 22

1	Amir A, Taba B, Berg D, et al. A Low Power, Fully Event-based Gesture Recognition System[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2017: 7388-7397.
2	Glover Arren, Bartolozzi Chiara. Event-driven Ball Detection and Gaze Fixation in Clutter[C]//2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway, NJ, USA: IEEE, 2016: 2203-2208.
3	Orchard G, Meyer Cedric, Etienne-Cummings R, et al. HFirst: A Temporal Approach to Object Recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(10): 2028-2040.
4	Matsuda N, Cossairt O, Gupta M. MC3D: Motion Contrast 3D Scanning[C]//2015 IEEE International Conference on Computational Photography (ICCP). Piscataway, NJ, USA: IEEE, 2015: 1-10.
5	Zhu Zihao, Yuan Liangzhe, Chaney K, et al. EV-FlowNet: Self-supervised Optical Flow Estimation for Event-based Cameras[EB/OL]. (2018-08-13) [2023-01-07]. .
6	Rebecq Henri, Ranftl René, Koltun V, et al. High Speed and High Dynamic Range Video with an Event Camera[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(6): 1964-1980.
7	Rebecq Henri, Horstschaefer Timo, Gallego Guillermo, et al. EVO: A Geometric Approach to Event-based 6-DOF Parallel Tracking and Mapping in Real Time[J]. IEEE Robotics and Automation Letters, 2017, 2(2): 593-600.
8	Antoni Rosinol Vidal, Rebecq Henri, Horstschaefer Timo, et al. Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High-speed Scenarios[J]. IEEE Robotics and Automation Letters, 2018, 3(2): 994-1001.
9	Gallego Guillermo, Delbrück Tobi, Orchard G, et al. Event-based Vision: A Survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 154-180.
10	Maqueda Ana I, Loquercio Antonio, Gallego Guillermo, et al. Event-based Vision Meets Deep Learning on Steering Prediction for Self-driving Cars[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2018: 5419-5427.
11	Gehrig Daniel, Rebecq Henri, Gallego Guillermo, et al. EKLT: Asynchronous Photometric Feature Tracking Using Events and Frames[J]. International Journal of Computer Vision, 2020, 128(3): 601-618.
12	Liu Min, Delbruck T. Adaptive Time-slice Block-Matching Optical Flow Algorithm for Dynamic Vision Sensors[C]//British Machine Vision Conference (BMVC) 2018. UK: BMVC, 2018: 168589.
13	Aimar Alessandro, Mostafa Hesham, Calabrese Enrico, et al. NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(3): 644-656.
14	Lagorce Xavier, Orchard G, Galluppi Francesco, et al. HOTS: A Hierarchy of Event-based Time-surfaces for Pattern Recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(7): 1346-1359.
15	Manderscheid Jacques, Sironi Amos, Bourdis Nicolas, et al. Speed Invariant Time Surface for Learning to Detect Corner Points with Event-based Cameras[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2019: 10237-10246.
16	Sironi Amos, Brambilla Manuele, Bourdis Nicolas, et al. HATS: Histograms of Averaged Time Surfaces for Robust Event-based Object Classification[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2018: 1731-1740.
17	Wang Qinyi, Zhang Yexin, Yuan Junsong, et al. Space-time Event Clouds for Gesture Recognition: from RGB Cameras to Event Cameras[C]//2019 IEEE Winter Conference on Applications of Computer Vision (WACV). Piscataway, NJ, USA: IEEE, 2019: 1826-1835.
18	Gallego Guillermo, Jon E A Lund, Mueggler Elias, et al. Event-based, 6-DOF Camera Tracking from Photometric Depth Maps[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(10): 2402-2412.
19	Roy K, Jaiswal A, Panda P. Towards Spike-based Machine Intelligence with Neuromorphic Computing[J]. Nature, 2019, 575(7784): 607-617.
20	Fang Wei, Yu Zhaofei, Chen Yanqi, et al. Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway, NJ, USA: IEEE, 2021: 2641-2651.
21	黄铁军, 余肇飞, 李源, 等. 脉冲视觉研究进展[J]. 中国图象图形学报, 2022, 27(6): 1823-1839.
	Huang Tiejun, Yu Zhaofei, Li Yuan, et al. Advances in Spike Vision[J]. Journal of Image and Graphics, 2022, 27(6): 1823-1839.
22	Loshchilo Ilya, Hutter Frank. SGDR: Stochastic Gradient Descent with Warm Restarts[C]//ICLR 2017 (5th International Conference on Learning Representations). New York, USA: ICLR, 2017: 1-16.

标签	动作类型
1	hand clap
2	right hand wave
3	left hand wave
4	right arm clockwise
5	right arm counter clockwise
6	left arm clockwise
7	left arm counter clockwise
8	arm roll
9	air drums
10	air guitar
11	other gestures

[1]	Xu Sheng, Feng Wenyu, Liu Zhicheng, Tu Xintao, Fei Minrui, Zhang Kun. Research on Accurate Gesture Recognition Algorithm in Complex Environment Based on Machine Vision [J]. Journal of System Simulation, 2021, 33(10): 2460-2469.
[2]	Xue Yanping, Wang Xuesong, Wu Zhongke, Wang Xingce, Zhou Mingquan. Application of Myo Gesture Recognition Method in Ancient Building Roaming System [J]. Journal of System Simulation, 2019, 31(9): 1907-1915.
[3]	Wang Yuanming, Zhang Jun, Qin Yuanhui, Chai Xiujuan. Gesture Recognition Method Based on Multi-feature Fusion [J]. Journal of System Simulation, 2019, 31(2): 346-352.
[4]	Cai Zhiwei, Wu Shuyan, Song Junfeng. Study on Hand Gesture Recognition and Portfolio Optimization Model Based on SVM [J]. Journal of System Simulation, 2016, 28(8): 1812-1817.
[5]	Huo Yuping, Zhang Xiu’e, Li Bing, Li Weiqing. Research on Motion Capture in Substation Virtual Environment [J]. Journal of System Simulation, 2016, 28(10): 2632-2637.
[6]	Mao Yanming, Zhang Liliang. Gesture Segmentation and Recognition Based on Kinect Depth Data [J]. Journal of System Simulation, 2015, 27(4): 830-835.
[7]	Wang Renda, Yin Yong, Xing Shengwei. Research on Skin Color Image Post-processing Methods for Gesture Recognition System [J]. Journal of System Simulation, 2015, 27(10): 2483-2488.