DEHPR：基于扩散模型的端到端手部姿态重建网络

doi:10.16182/j.issn1004731x.joss.25-0863

摘要/Abstract

摘要：

针对传统方法如卷积神经网络(CNN)和Transformer在处理复杂场景的手部姿态重建任务时存在对大规模标注数据依赖性强、泛化能力不足等问题，提出了基于扩散模型的端到端手部姿态重建网络(diffusion-based end-to-end hand pose reconstruction network，DEHPR)。DEHPR通过引入扩散模型直接生成3D姿态假设并进行细化的策略，降低2D-to-3D建模方式导致的空间不确定性，引入端到端模型对多个3D姿态假设进行重投影选取最优关节点，最终生成预测的手部姿态。分别在HO3D V2数据集、DexYCB数据集以及FreiHand数据集上对所提出网络进行性能评估实验，结果表明，DEHPR性能效果优于现有方法，有效降低了对大规模标注数据的依赖性和单RGB图像2D-to-3D间接模型的不确定性，提升了手部姿态重建的准确性和鲁棒性。

关键词: 扩散模型, 端到端, 手部姿态, 手部遮挡, 姿态重建

Abstract:

Traditional methods such as convolutional neural networks (CNNs) and Transformers suffer from strong dependence on large-scale annotated data and limited generalization capability when dealing with hand pose reconstruction in complex scenarios. To address these issues, a diffusion-based end-to-end hand pose reconstruction network (DEHPR) is proposed. This method employs a diffusion model to directly generate and refine 3D predictions, thereby reducing spatial uncertainties inherent in 2D-to-3D modeling paradigms. By incorporating an end-to-end framework that reprojects multiple 3D candidate predictions to select optimal joint positions, the approach ultimately produces accurate hand pose estimations. Comprehensive evaluations conducted on HO3D V2, DexYCB, and FreiHand datasets demonstrate that DEHPR achieves superior performance compared to existing methods. The proposed solution effectively diminishes dependency on large-scale annotated data, mitigates uncertainties in indirect 2D-to-3D modeling from single RGB images, and consequently enhances both accuracy and robustness in hand pose reconstruction.

Key words: diffusion model, end-to-end, hand posture, hand occlusion, pose reconstruction

中图分类号:

TP391.41

廖国琼,黄龙杰,李清新等 . DEHPR：基于扩散模型的端到端手部姿态重建网络[J]. 系统仿真学报, 2026, 38(1): 174-188.

Liao Guoqiong,Huang Longjie,Li Qingxin,et al . DEHPR: A Diffusion-based End-to-end Hand Pose Reconstruction Network[J]. Journal of System Simulation, 2026, 38(1): 174-188.

图/表 19

图1

图2

图3

表1

表2

表3

表4

表5

表6

表7

表8

表9

图4

表10

表11

表12

图7

图8

参考文献 33

[1]	Gao Chengying, Yang Yujia, Li Wensheng. 3D Interacting Hand Pose and Shape Estimation from a Single RGB Image[J]. Neurocomputing, 2022, 474: 25-36.
[2]	Taylor J, Tankovich V, Tang Danhang, et al. Articulated Distance Fields for Ultra-fast Tracking of Hands Interacting[J]. ACM Transactions on Graphics, 2017, 36(6): 244.
[3]	Wang Jiayi, Mueller F, Bernard Florian, et al. RGB2Hands: Real-time Tracking of 3D Hand Interactions from Monocular RGB Video[J]. ACM Transactions on Graphics, 2020, 39(6): 218.
[4]	Zhang Yuxiang, Li Zhe, An Liang, et al. Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2021: 5540-5549.
[5]	Romero J, Tzionas D, Black M J. Embodied Hands: Modeling and Capturing Hands and Bodies Together[J]. ACM Transactions on Graphics, 2017, 36(6): 245.
[6]	Chen Yujin, Tu Zhigang, Kang Di, et al. Model-based 3D Hand Reconstruction via Self-supervised Learning[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 10446-10455.
[7]	Zhang Hongwen, Tian Yating, Zhang Yuxiang, et al. PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(10): 12287-12303.
[8]	Zhang Xiong, Li Qiang, Mo Hong, et al. End-to-end Hand Mesh Recovery from a Monocular RGB Image[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2019: 2354-2364.
[9]	Zhang Yuxiang, Zhang Hongwen, Hu Liangxiao, et al. ProxyCap: Real-time Monocular Full-body Capture in World Space via Human-centric Proxy-to-motion Learning[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 1954-1964.
[10]	Zhou Yuxiao, Habermann M, Xu Weipeng, et al. Monocular Real-time Hand Shape and Motion Capture Using Multi-modal Data[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2020: 5345-5354.
[11]	Chen Xingyu, Liu Yufeng, Ma Chongyang, et al. Camera-space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 13269-13278.
[12]	Chen Xingyu, Liu Yufeng, Dong Yajiao, et al. MobRecon: Mobile-friendly Hand Mesh Reconstruction from Monocular Image[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 20512-20522.
[13]	Ge Liuhao, Ren Zhou, Li Yuncheng, et al. 3D Hand Shape and Pose Estimation from a Single RGB Image[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2019: 10825-10834.
[14]	Kulon D, Riza Alp Güler, Kokkinos I, et al. Weakly-supervised Mesh-convolutional Hand Reconstruction in the Wild[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2020: 4989-4999.
[15]	Li Mengcheng, An Liang, Zhang Hongwen, et al. Interacting Attention Graph for Single Image Two-hand Reconstruction[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 2751-2760.
[16]	Tang Xiao, Wang Tianyu, Fu C W. Towards Accurate Alignment in Real-time 3D Hand-mesh Reconstruction[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2021: 11678-11687.
[17]	Moon Gyeongsik, Mu Lee Kyoung. I2L-MeshNet: Image-to-lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image[C]//Computer Vision – ECCV 2020. Cham: Springer International Publishing, 2020: 752-768.
[18]	Chen Ping, Chen Yujin, Yang Dong, et al. I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modeling[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2021: 12909-12918.
[19]	Lin K, Wang Lijuan, Liu Zicheng. Mesh Graphormer[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2021: 12919-12928.
[20]	Lin K, Wang Lijuan, Liu Zicheng. End-to-end Human Pose and Mesh Reconstruction with Transformers[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 1954-1963.
[21]	Feng Runyang, Gao Yixing, Elden Tse T H, et al. DiffPose: SpatioTemporal Diffusion Model for Video-based Human Pose Estimation[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2023: 14815-14826.
[22]	Gong Jia, Geng Lin, Fan Zhipeng, et al. DiffPose: Toward More Reliable 3D Pose Estimation[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2023: 13041-13051.
[23]	Holmquist Karl, Wandt Bastian. DiffPose: Multi-hypothesis Human Pose Estimation Using Diffusion Models[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2023: 15931-15941.
[24]	Shan Wenkang, Liu Zhenhua, Zhang Xinfeng, et al. Diffusion-based 3D Human Pose Estimation with Multi-hypothesis Aggregation[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2023: 14715-14725.
[25]	Nichol A, Jun H, Dhariwal P, et al. Point-E: A System for Generating 3D Point Clouds from Complex Prompts[EB/OL]. (2022-12-16) [2025-07-16]. .
[26]	Jun H, Nichol A. Shap-E: Generating Conditional 3D Implicit Functions[EB/OL]. (2023-05-03) [2025-07-16]. .
[27]	Ying Chengxuan, Cai Tianle, Luo Shengjie, et al. Do Transformers Really Perform Bad for Graph Representation?[C]//Proceedings of the 35th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2021: 28877-28888.
[28]	Kong Deying, Zhang Linguang, Chen Liangjian, et al. Identity-aware Hand Mesh Estimation and Personalization from RGB Images[C]//Computer Vision-ECCV 2022. Cham: Springer Nature Switzerland, 2022: 536-553.
[29]	Park JoonKyu, Oh Yeonguk, Moon Gyeongsik, et al. HandOccNet: Occlusion-robust 3D Hand Mesh Estimation Network[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 1486-1495.
[30]	Hasson Yana, Varol Gül, Schmid Cordelia, et al. Towards Unconstrained Joint Hand-object Reconstruction from RGB Videos[C]//2021 International Conference on 3D Vision (3DV). Piscataway: IEEE, 2021: 659-668.
[31]	Luan Tianyu, Zhai Yuanhao, Meng Jingjing, et al. High Fidelity 3D Hand Shape Reconstruction via Scalable Graph Frequency Decomposition[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2023: 16795-16804.
[32]	Zhao Lizhi, Lu Xuequan, Fan Runze, et al. GaussianHand: Real-time 3D Gaussian Rendering for Hand Avatar Animation[J]. IEEE Transactions on Visualization and Computer Graphics, 2025, 31(9): 6484-6496.
[33]	Hampali Shreyas, Sayan Deb Sarkar, Rad Mahdi, et al. Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 11080-11090.

编码器类型	FreiHand		DexYCB
编码器类型	E_PV	E_PJ	E_PV	E_PJ
HRNet-W64	6.48	6.72	5.02	5.90
Deit-Large	6.38	6.65	5.08	6.02

编码器类型	输入分辨率	参数量(M)	推理显存(G)
HRNet-W48	224×224	77.50	2.50
HRNet-W48	256×256	77.50	3.16
HRNet-W64	224×224	128.10	3.75
HRNet-W64	256×256	128.10	4.58
Deit-Large	224×224	304.15	5.31
Deit-Large	256×256	304.15	6.28

实验配置	假设数	推理速度(FPS)
RTX3060	2	22
	4	14
	8	5
RTX4080	2	42
	4	28
	8	12

跨模态解码器	扩散模型	关节优化网络	E_PV	E_PJ
×	×	×	7.87	7.08
×	√	√	6.99	6.74
×	×	√	7.88	8.15
×	√	×	9.88	7.82
×	√	√	7.09	6.59
×	√	√	7.85	8.10
√	√	×	7.65	7.85
√	√	√	6.48	6.72

建模网络	E_PV/mm	E_PJ/mm	参数量(M)
GCN	7.90	8.08	12.20
标准Transformer	6.89	7.21	47.30
Graphormer^[27]	6.52	6.78	52.10
跨模态解码器	6.48	6.72	37.19