Construction Technology of Hand Posture Dataset Based on Virtual Simulation Method

doi:10.16182/j.issn1004731x.joss.21-1265

Abstract

Abstract:

Hand posture is an important carrier of human-computer interaction, and the acquisition and recognition of posture information largely depends on the hand posture dataset. Existing datasets can be divided into two categories, real datasets and synthetic datasets. As real data is limited by equipment, environment, and other factors, the classification of hand posture is insufficient and the annotation is mixed with a lot of manual errors. The existing synthetic data can solve the data scale problem of real data, but the synthetic hand posture volume is limited and with some unreasonable kinematic postures of which the data form are only RGB images. By studying the hand anatomical structure, a virtual 3D kinematic model of hand is created based on virtual simulation, and a hand posture simulation generator is built. The hand posture dataset constructed by the generator can not only effectively solve the problems of the insufficient data volume of real data, low accuracy of annotation, and large workload of manual annotation, but also solve the problems of insufficient hand posture category and single data mode of synthetic data.

Key words: virtual simulation, hand posture dataset, kinematics modeling, hand posture estimation, human-computer interaction

CLC Number:

TP391.9

Jiaxin Chen, Guohui Zhou, Jianbai Yang. Construction Technology of Hand Posture Dataset Based on Virtual Simulation Method[J]. Journal of System Simulation, 2023, 35(4): 862-870.

Figures/Tables 11

Fig. 1

Fig. 2

Fig. 3

Table 1

Table 2

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

References 22

1	Gao Q, Chen Y, Ju Z, et al. Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human-Robot Interaction[J]. IEEE Sensors Journal(S1530-437X), 2021, 22: 17421-17430.
2	Wu M Y, Ting P W, Tang Y H, et al. Hand Pose Estimation in Object-Interaction Based on Deep Learning for Virtual Reality Applications[J]. Journal of Visual Communication and Image Representation(S1047-3203), 2020, 70: 102802.
3	Brereton M, Khan S, Rajapakse H, et al. GesCAD: An Intuitive Interface for Conceptual Architectural[C]// 29th Australian Conference on Computer-Human Interaction. 2017: 402-406.
4	Lee1 L H, Braud T, Zhou P. All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda[EB/OL]. (2021-11-03) [2022-03-30]. .
5	Supančič J S, Rogez G, Yang Y, et al. Depth-Based Hand Pose Estimation: Data, Methods, and Challenges[C]// 2015 IEEE International Conference on Computer Vision(ICCV). Santiago, Chile: IEEE, 2015: 1868-1876.
6	Yuan S, Garcia-Hernando G, Stenger B. Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals[C]// IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Salt Lake City, UT, USA: IEEE, 2018: 2636-2645.
7	Yuan S, Ye Q, Stenger B, et al. Bighand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis[C]//IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE, 2017: 4866-4874.
8	Zimmermann C, Ceylan D, Yang J, et al. FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images[C]// 2019 IEEE/CVF International Conference on Computer Vision(ICCV). Seoul, Korea (South): IEEE, 2019: 813-822.
9	Kulon D, Güler R A, Kokkinos I, et al. Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Seattle, WA, USA: IEEE, 2020: 4989-4999.
10	Chen L, Lin S Y, Xie Y, et al. MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate[C]//IEEE/CVF Winter Conference on Applications of Computer Vision(WACV). Waikoloa, HI, USA: IEEE, 2021: 836-845.
11	Tang D, Chang H J, Tejani A, et al. Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture[C]//IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Columbus, OH, USA: IEEE, 2014: 3786-3793.
12	Tompson J, Stein M, Lecun Y, et al. Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks[J]. ACM Transactions on Graphics(TOG)(S0730-0301), 2014, 33(5): 1-10.
13	Zimmermann C, Brox T. Learning to Estimate 3D Hand Pose from Single RGB Images[C]// 2017 IEEE International Conference on Computer Vision(ICCV). Venice, Italy: IEEE, 2017: 4913-4921.
14	Malik J, Elhayek A, Nunnari F, et al. DeepHPS: End-to-End Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth[C]// 2018 International Conference on 3D Vision(3DV). Verona, Italy: IEEE, 2018: 110-119.
15	Tubiana R, Thomine J M, Mackin E. Examination of the Hand and Wrist[M]. London, UK: Collaborative Research Centres Press, 1998.
16	Peña-Pitarch E, Falguera N T, Yang J. Virtual Human Hand: Model and Kinematics[J]. Computer Methods in Biomechanics and Biomedical Engineering(S1025-5842), 2014, 17(5): 568-579.
17	Llop-Harillo I, Pérez-González A, Gracia-Ibáñez V. Anthropomorphism Index of Mobility for Artificial Hands[J]. Applied Bionics and Biomechanics(S1176-2322), 2019: 1-11.
18	Lynch K M, Park F C. Modern Robotics[M]. Cambridge UK: Cambridge University Press, 2017.
19	Shen Y, Jia Q, Chen G, et al. Study of Rapid Collision Detection Algorithm for Manipulator[C]//2015 IEEE 10th Conference on Industrial Electronics and Applications(ICIEA). Auckland, New Zealand: IEEE, 2015: 934-938.
20	Hou Z, Ma S, Zeng Q, et al. Kinematics Analysis and Self-collision Detection of Truss Type Multi-robot Cooperative Welding Platform[J]. Procedia CIRP(S2212-8271), 2019, 81: 488-493.
21	吴长征, 岳义, 韦宝琛, 等. 双臂机器人自碰撞检测及其运动规划[J]. 上海交通大学学报, 2018, 52(1): 45-53.
	Wu Changzheng, Yue Yi, Wei Baochen, et al. Self-Collision Detection and Motion Planning for Dual-Arm Robot[J]. Journal of Shanghai Jiaotong University, 2018, 52(1): 45-53.
22	Wang T, Tang M, Wang Z, et al. Accurate Self-collision Detection Using Enhanced Dual-cone Method[J]. Computers & Graphics(S0097-8493), 2018, 73: 70-79.

大拇指关节	旋转类型	角度限制/(°)
CMC	Ad/Ab	0/60
CMC	E/F	25/35
MCP	E/F	-10/55
MCP	Ad/Ab	0/60
IP	E/F	-15/80

手指	MCP(E/F)	MCP(Ab/Ad)	PIP(E/F)	DIP(E/F)
II	0/80	13/42	0/100	-10/90
III	0/80	8/35	0/100	-10/90
IV	0/80	14/20	0/100	-20/90
V	0/80	19/33	0/100	-30/90

[1]	Zhijie Li, Shuangyu Ma, Changhua Li, Xiao Liang, Jie Zhang. Floor Evacuation Simulation Based on BIM and MR [J]. Journal of System Simulation, 2023, 35(3): 658-669.
[2]	Jun Liu, Yang Gao, Tao Xu, Qing Zhao, Guihua Shan, Xuebin Chi. Interactive Construction of Scientific Workflow Based on Process Mining [J]. Journal of System Simulation, 2022, 34(7): 1547-1558.
[3]	Feng Wu, Xiuluo Liu, Jia Wang, Yang Liu, Sujiang Li, Yan Zhong. Research on Space Launch Visualization Simulation Analysis Technology and Application [J]. Journal of System Simulation, 2022, 34(4): 856-869.
[4]	Liu Xiuluo, Wang Jia, Wu Feng, Liu Yang, Zhang Ailiang, Wang Min. Research and Application on Space-launching Mission Simulated Rehearsal Technology [J]. Journal of System Simulation, 2021, 33(9): 2227-2233.
[5]	Wei Shisong, Zhou Zhengdong, Zhang Xuling, Mao Ling, Jia Junshan, Liu Chuanle. Virtual Training System for Spacecraft Maintenance Based on Desktop Virtual Reality [J]. Journal of System Simulation, 2021, 33(6): 1358-1363.
[6]	Liu Xianmei, Jia Di, Liu Fang. Research on Virtual Simulation Training System for Multi-position Collaborative Operation in Oilfields [J]. Journal of System Simulation, 2021, 33(1): 74-83.
[7]	Hu Lingyan, Shi Kangbai, Zhang Bichun, Xu Shaoping. Tele-robot Control System with Online Virtual Simulation Prediction [J]. Journal of System Simulation, 2020, 32(2): 247-254.
[8]	Sun Xiaotian, Xiao Boxiang, Liu Zhengdong. Design and Implementation of Cloth Virtual Simulation System for Group Performance [J]. Journal of System Simulation, 2020, 32(11): 2282-2287.
[9]	Wu Jiajia, Liu Zhen, Xu Huihuang, Liu Tingting, Wang Jin, Chai Yanjie. Serious Game Human-Computer Interaction Design for Rehabilitation Training [J]. Journal of System Simulation, 2019, 31(5): 909-918.
[10]	Zhu Xiaoqiang, Pan Hongyi, Xu Hao, Song Lei, Zhu Mengyao, Wang Xiangyang. Interactive Modelling on Smooth Weighted Distance Surface in HMD VR Environment [J]. Journal of System Simulation, 2018, 30(7): 2459-2464.
[11]	Wang Zhiwen, Zhang Zihao, Jia Yuxiang, Wei Yi, Shen Yanfei. Human Motion Simulation Based on Speech Recognition Control [J]. Journal of System Simulation, 2018, 30(11): 4203-4209.
[12]	Han Liang, Gong Xuan, Wang Jiangyun, Du Yanghua. Study on visual Function of Human Computer Interaction in Parachuting Simulator [J]. Journal of System Simulation, 2017, 29(8): 1747-1752.
[13]	Du Baojiang, Zha Liang, Lin Ling. Key Technology of Fully Immersive Virtual Experience Cabin [J]. Journal of System Simulation, 2017, 29(2): 453-461.
[14]	He Shiwei, Zhou Leishan. Software Technology of Transportation Virtual Simulation Experiment Teaching Center [J]. Journal of System Simulation, 2016, 28(2): 322-328.
[15]	Liu Tingting, Liu Zhen, Chai Yanjie, Liu Cuijuan, Chen Jiaxin. Method for Crowd Trampling and Crushing Simulation Based on Psychological Model [J]. Journal of System Simulation, 2016, 28(10): 2448-2454.