基于虚拟仿真方法的手部姿态数据集构建技术

doi:10.16182/j.issn1004731x.joss.21-1265

摘要/Abstract

摘要：

手部姿态是人机交互的重要载体，姿态信息的获取和识别很大程度上依赖于手部姿态数据集。现有的数据集分成2类：真实数据集和合成数据集。真实数据受限于设备与环境等因素，手部姿态类别不足且标注掺入大量人工误差。而目前存在的合成数据虽然可以解决真实数据的数据量问题，但生成的手部姿态有限且存在一些不合理的运动学姿态，获取的数据往往也只有RGB图像。通过研究手部的解剖学结构，基于虚拟仿真方法，创建了手部的虚拟三维运动学模型和一个手部姿态仿真生成器。利用该生成器构建的手部姿态数据集既可以有效解决真实数据数据量不足、标注准确度低、人工标注工作量大等问题，也可以解决当前合成数据手部姿态类别不足且数据模态单一的问题。

关键词: 虚拟仿真, 手部姿态数据集, 运动学建模, 手部姿态估计, 人机交互

Abstract:

Hand posture is an important carrier of human-computer interaction, and the acquisition and recognition of posture information largely depends on the hand posture dataset. Existing datasets can be divided into two categories, real datasets and synthetic datasets. As real data is limited by equipment, environment, and other factors, the classification of hand posture is insufficient and the annotation is mixed with a lot of manual errors. The existing synthetic data can solve the data scale problem of real data, but the synthetic hand posture volume is limited and with some unreasonable kinematic postures of which the data form are only RGB images. By studying the hand anatomical structure, a virtual 3D kinematic model of hand is created based on virtual simulation, and a hand posture simulation generator is built. The hand posture dataset constructed by the generator can not only effectively solve the problems of the insufficient data volume of real data, low accuracy of annotation, and large workload of manual annotation, but also solve the problems of insufficient hand posture category and single data mode of synthetic data.

Key words: virtual simulation, hand posture dataset, kinematics modeling, hand posture estimation, human-computer interaction

中图分类号:

TP391.9

陈佳昕, 周国辉, 杨建柏. 基于虚拟仿真方法的手部姿态数据集构建技术[J]. 系统仿真学报, 2023, 35(4): 862-870.

Jiaxin Chen, Guohui Zhou, Jianbai Yang. Construction Technology of Hand Posture Dataset Based on Virtual Simulation Method[J]. Journal of System Simulation, 2023, 35(4): 862-870.

图/表 11

图1

图2

图3

表1

表2

图4

图5

图6

图7

图8

图9

参考文献 22

1	Gao Q, Chen Y, Ju Z, et al. Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human-Robot Interaction[J]. IEEE Sensors Journal(S1530-437X), 2021, 22: 17421-17430.
2	Wu M Y, Ting P W, Tang Y H, et al. Hand Pose Estimation in Object-Interaction Based on Deep Learning for Virtual Reality Applications[J]. Journal of Visual Communication and Image Representation(S1047-3203), 2020, 70: 102802.
3	Brereton M, Khan S, Rajapakse H, et al. GesCAD: An Intuitive Interface for Conceptual Architectural[C]// 29th Australian Conference on Computer-Human Interaction. 2017: 402-406.
4	Lee1 L H, Braud T, Zhou P. All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda[EB/OL]. (2021-11-03) [2022-03-30]. .
5	Supančič J S, Rogez G, Yang Y, et al. Depth-Based Hand Pose Estimation: Data, Methods, and Challenges[C]// 2015 IEEE International Conference on Computer Vision(ICCV). Santiago, Chile: IEEE, 2015: 1868-1876.
6	Yuan S, Garcia-Hernando G, Stenger B. Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals[C]// IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Salt Lake City, UT, USA: IEEE, 2018: 2636-2645.
7	Yuan S, Ye Q, Stenger B, et al. Bighand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis[C]//IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE, 2017: 4866-4874.
8	Zimmermann C, Ceylan D, Yang J, et al. FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images[C]// 2019 IEEE/CVF International Conference on Computer Vision(ICCV). Seoul, Korea (South): IEEE, 2019: 813-822.
9	Kulon D, Güler R A, Kokkinos I, et al. Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Seattle, WA, USA: IEEE, 2020: 4989-4999.
10	Chen L, Lin S Y, Xie Y, et al. MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate[C]//IEEE/CVF Winter Conference on Applications of Computer Vision(WACV). Waikoloa, HI, USA: IEEE, 2021: 836-845.
11	Tang D, Chang H J, Tejani A, et al. Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture[C]//IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Columbus, OH, USA: IEEE, 2014: 3786-3793.
12	Tompson J, Stein M, Lecun Y, et al. Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks[J]. ACM Transactions on Graphics(TOG)(S0730-0301), 2014, 33(5): 1-10.
13	Zimmermann C, Brox T. Learning to Estimate 3D Hand Pose from Single RGB Images[C]// 2017 IEEE International Conference on Computer Vision(ICCV). Venice, Italy: IEEE, 2017: 4913-4921.
14	Malik J, Elhayek A, Nunnari F, et al. DeepHPS: End-to-End Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth[C]// 2018 International Conference on 3D Vision(3DV). Verona, Italy: IEEE, 2018: 110-119.
15	Tubiana R, Thomine J M, Mackin E. Examination of the Hand and Wrist[M]. London, UK: Collaborative Research Centres Press, 1998.
16	Peña-Pitarch E, Falguera N T, Yang J. Virtual Human Hand: Model and Kinematics[J]. Computer Methods in Biomechanics and Biomedical Engineering(S1025-5842), 2014, 17(5): 568-579.
17	Llop-Harillo I, Pérez-González A, Gracia-Ibáñez V. Anthropomorphism Index of Mobility for Artificial Hands[J]. Applied Bionics and Biomechanics(S1176-2322), 2019: 1-11.
18	Lynch K M, Park F C. Modern Robotics[M]. Cambridge UK: Cambridge University Press, 2017.
19	Shen Y, Jia Q, Chen G, et al. Study of Rapid Collision Detection Algorithm for Manipulator[C]//2015 IEEE 10th Conference on Industrial Electronics and Applications(ICIEA). Auckland, New Zealand: IEEE, 2015: 934-938.
20	Hou Z, Ma S, Zeng Q, et al. Kinematics Analysis and Self-collision Detection of Truss Type Multi-robot Cooperative Welding Platform[J]. Procedia CIRP(S2212-8271), 2019, 81: 488-493.
21	吴长征, 岳义, 韦宝琛, 等. 双臂机器人自碰撞检测及其运动规划[J]. 上海交通大学学报, 2018, 52(1): 45-53.
	Wu Changzheng, Yue Yi, Wei Baochen, et al. Self-Collision Detection and Motion Planning for Dual-Arm Robot[J]. Journal of Shanghai Jiaotong University, 2018, 52(1): 45-53.
22	Wang T, Tang M, Wang Z, et al. Accurate Self-collision Detection Using Enhanced Dual-cone Method[J]. Computers & Graphics(S0097-8493), 2018, 73: 70-79.

大拇指关节	旋转类型	角度限制/(°)
CMC	Ad/Ab	0/60
CMC	E/F	25/35
MCP	E/F	-10/55
MCP	Ad/Ab	0/60
IP	E/F	-15/80

手指	MCP(E/F)	MCP(Ab/Ad)	PIP(E/F)	DIP(E/F)
II	0/80	13/42	0/100	-10/90
III	0/80	8/35	0/100	-10/90
IV	0/80	14/20	0/100	-20/90
V	0/80	19/33	0/100	-30/90

[1]	李智杰, 马爽予, 李昌华, 梁霄, 张颉. 基于BIM与MR的楼层疏散仿真[J]. 系统仿真学报, 2023, 35(3): 658-669.
[2]	刘俊, 高阳, 徐涛, 赵青, 单桂华, 迟学斌. 基于过程挖掘的科学工作流交互式构建[J]. 系统仿真学报, 2022, 34(7): 1547-1558.
[3]	吴枫, 刘秀罗, 王佳, 刘阳, 李苏疆, 仲妍. 航天发射可视化仿真分析技术与应用研究[J]. 系统仿真学报, 2022, 34(4): 856-869.
[4]	刘秀罗, 王佳, 吴枫, 刘阳, 张爱良, 王敏. 航天发射数字化合练技术研究与应用[J]. 系统仿真学报, 2021, 33(9): 2227-2233.
[5]	卢柏宏, 赵建军, 刘戈三. 电影虚拟摄影半实物仿真的研究与实现[J]. 系统仿真学报, 2021, 33(8): 1938-1946.
[6]	魏士松, 周正东, 章栩苓, 毛玲, 贾峻山, 刘传乐. 基于桌面虚拟现实技术的航天器虚拟维修训练系统[J]. 系统仿真学报, 2021, 33(6): 1358-1363.
[7]	刘贤梅, 贾迪, 刘芳. 油田多岗位协同操作虚拟仿真培训系统研究[J]. 系统仿真学报, 2021, 33(1): 74-83.
[8]	郭园, 童倩倩, 郑宇铠, 王子琦, 张玉茹, 王党校. 融合多元触觉和沉浸式视觉的可移植VR软件框架[J]. 系统仿真学报, 2020, 32(7): 1385-1392.
[9]	胡凌燕, 史康柏, 张碧纯, 徐少平. 在线修正虚拟仿真预测的遥操作机器人系统[J]. 系统仿真学报, 2020, 32(2): 247-254.
[10]	孙小添, 肖伯祥, 刘正东. 群体演出活动服装虚拟仿真系统设计与实现[J]. 系统仿真学报, 2020, 32(11): 2282-2287.
[11]	薛艳萍, 王学松, 武仲科, 王醒策, 周明全. Myo手势识别方法在古建筑漫游系统中的应用[J]. 系统仿真学报, 2019, 31(9): 1907-1915.
[12]	吴佳佳, 刘箴, 许辉煌, 刘婷婷, 王瑾, 柴艳杰. 面向康复训练的严肃游戏人机交互设计[J]. 系统仿真学报, 2019, 31(5): 909-918.
[13]	朱晓强, 潘虹艺, 徐浩, 宋磊, 朱梦尧, 王向阳. 头戴设备VR环境下光滑加权等距面交互建模[J]. 系统仿真学报, 2018, 30(7): 2459-2464.
[14]	王志文, 张子豪, 贾玉祥, 魏毅, 沈燕飞. 基于语音识别控制的人体运动仿真[J]. 系统仿真学报, 2018, 30(11): 4203-4209.
[15]	韩亮, 公璇, 王江云, 杜阳华. 跳伞模拟器人机交互可视化功能研究[J]. 系统仿真学报, 2017, 29(8): 1747-1752.