DEHPR: A Diffusion-based End-to-end Hand Pose Reconstruction Network

doi:10.16182/j.issn1004731x.joss.25-0863

Abstract

Abstract:

Traditional methods such as convolutional neural networks (CNNs) and Transformers suffer from strong dependence on large-scale annotated data and limited generalization capability when dealing with hand pose reconstruction in complex scenarios. To address these issues, a diffusion-based end-to-end hand pose reconstruction network (DEHPR) is proposed. This method employs a diffusion model to directly generate and refine 3D predictions, thereby reducing spatial uncertainties inherent in 2D-to-3D modeling paradigms. By incorporating an end-to-end framework that reprojects multiple 3D candidate predictions to select optimal joint positions, the approach ultimately produces accurate hand pose estimations. Comprehensive evaluations conducted on HO3D V2, DexYCB, and FreiHand datasets demonstrate that DEHPR achieves superior performance compared to existing methods. The proposed solution effectively diminishes dependency on large-scale annotated data, mitigates uncertainties in indirect 2D-to-3D modeling from single RGB images, and consequently enhances both accuracy and robustness in hand pose reconstruction.

Key words: diffusion model, end-to-end, hand posture, hand occlusion, pose reconstruction

CLC Number:

TP391.41

Liao Guoqiong, Huang Longjie, Li Qingxin, Zhang Jiajun, Chen Kefan. DEHPR: A Diffusion-based End-to-end Hand Pose Reconstruction Network[J]. Journal of System Simulation, 2026, 38(1): 174-188.

Figures/Tables 19

Fig. 1

Fig. 2

Fig. 3

Table 1

Table 2

Table 3

Table 4

Table 5

Table 6

Table 7

Table 8

Table 9

Fig. 4

Table 10

Table 11

Table 12

Fig. 7

Fig. 8

References 33

[1]	Gao Chengying, Yang Yujia, Li Wensheng. 3D Interacting Hand Pose and Shape Estimation from a Single RGB Image[J]. Neurocomputing, 2022, 474: 25-36.
[2]	Taylor J, Tankovich V, Tang Danhang, et al. Articulated Distance Fields for Ultra-fast Tracking of Hands Interacting[J]. ACM Transactions on Graphics, 2017, 36(6): 244.
[3]	Wang Jiayi, Mueller F, Bernard Florian, et al. RGB2Hands: Real-time Tracking of 3D Hand Interactions from Monocular RGB Video[J]. ACM Transactions on Graphics, 2020, 39(6): 218.
[4]	Zhang Yuxiang, Li Zhe, An Liang, et al. Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2021: 5540-5549.
[5]	Romero J, Tzionas D, Black M J. Embodied Hands: Modeling and Capturing Hands and Bodies Together[J]. ACM Transactions on Graphics, 2017, 36(6): 245.
[6]	Chen Yujin, Tu Zhigang, Kang Di, et al. Model-based 3D Hand Reconstruction via Self-supervised Learning[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 10446-10455.
[7]	Zhang Hongwen, Tian Yating, Zhang Yuxiang, et al. PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(10): 12287-12303.
[8]	Zhang Xiong, Li Qiang, Mo Hong, et al. End-to-end Hand Mesh Recovery from a Monocular RGB Image[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2019: 2354-2364.
[9]	Zhang Yuxiang, Zhang Hongwen, Hu Liangxiao, et al. ProxyCap: Real-time Monocular Full-body Capture in World Space via Human-centric Proxy-to-motion Learning[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 1954-1964.
[10]	Zhou Yuxiao, Habermann M, Xu Weipeng, et al. Monocular Real-time Hand Shape and Motion Capture Using Multi-modal Data[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2020: 5345-5354.
[11]	Chen Xingyu, Liu Yufeng, Ma Chongyang, et al. Camera-space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 13269-13278.
[12]	Chen Xingyu, Liu Yufeng, Dong Yajiao, et al. MobRecon: Mobile-friendly Hand Mesh Reconstruction from Monocular Image[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 20512-20522.
[13]	Ge Liuhao, Ren Zhou, Li Yuncheng, et al. 3D Hand Shape and Pose Estimation from a Single RGB Image[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2019: 10825-10834.
[14]	Kulon D, Riza Alp Güler, Kokkinos I, et al. Weakly-supervised Mesh-convolutional Hand Reconstruction in the Wild[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2020: 4989-4999.
[15]	Li Mengcheng, An Liang, Zhang Hongwen, et al. Interacting Attention Graph for Single Image Two-hand Reconstruction[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 2751-2760.
[16]	Tang Xiao, Wang Tianyu, Fu C W. Towards Accurate Alignment in Real-time 3D Hand-mesh Reconstruction[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2021: 11678-11687.
[17]	Moon Gyeongsik, Mu Lee Kyoung. I2L-MeshNet: Image-to-lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image[C]//Computer Vision – ECCV 2020. Cham: Springer International Publishing, 2020: 752-768.
[18]	Chen Ping, Chen Yujin, Yang Dong, et al. I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modeling[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2021: 12909-12918.
[19]	Lin K, Wang Lijuan, Liu Zicheng. Mesh Graphormer[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2021: 12919-12928.
[20]	Lin K, Wang Lijuan, Liu Zicheng. End-to-end Human Pose and Mesh Reconstruction with Transformers[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 1954-1963.
[21]	Feng Runyang, Gao Yixing, Elden Tse T H, et al. DiffPose: SpatioTemporal Diffusion Model for Video-based Human Pose Estimation[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2023: 14815-14826.
[22]	Gong Jia, Geng Lin, Fan Zhipeng, et al. DiffPose: Toward More Reliable 3D Pose Estimation[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2023: 13041-13051.
[23]	Holmquist Karl, Wandt Bastian. DiffPose: Multi-hypothesis Human Pose Estimation Using Diffusion Models[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2023: 15931-15941.
[24]	Shan Wenkang, Liu Zhenhua, Zhang Xinfeng, et al. Diffusion-based 3D Human Pose Estimation with Multi-hypothesis Aggregation[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2023: 14715-14725.
[25]	Nichol A, Jun H, Dhariwal P, et al. Point-E: A System for Generating 3D Point Clouds from Complex Prompts[EB/OL]. (2022-12-16) [2025-07-16]. .
[26]	Jun H, Nichol A. Shap-E: Generating Conditional 3D Implicit Functions[EB/OL]. (2023-05-03) [2025-07-16]. .
[27]	Ying Chengxuan, Cai Tianle, Luo Shengjie, et al. Do Transformers Really Perform Bad for Graph Representation?[C]//Proceedings of the 35th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2021: 28877-28888.
[28]	Kong Deying, Zhang Linguang, Chen Liangjian, et al. Identity-aware Hand Mesh Estimation and Personalization from RGB Images[C]//Computer Vision-ECCV 2022. Cham: Springer Nature Switzerland, 2022: 536-553.
[29]	Park JoonKyu, Oh Yeonguk, Moon Gyeongsik, et al. HandOccNet: Occlusion-robust 3D Hand Mesh Estimation Network[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 1486-1495.
[30]	Hasson Yana, Varol Gül, Schmid Cordelia, et al. Towards Unconstrained Joint Hand-object Reconstruction from RGB Videos[C]//2021 International Conference on 3D Vision (3DV). Piscataway: IEEE, 2021: 659-668.
[31]	Luan Tianyu, Zhai Yuanhao, Meng Jingjing, et al. High Fidelity 3D Hand Shape Reconstruction via Scalable Graph Frequency Decomposition[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2023: 16795-16804.
[32]	Zhao Lizhi, Lu Xuequan, Fan Runze, et al. GaussianHand: Real-time 3D Gaussian Rendering for Hand Avatar Animation[J]. IEEE Transactions on Visualization and Computer Graphics, 2025, 31(9): 6484-6496.
[33]	Hampali Shreyas, Sayan Deb Sarkar, Rad Mahdi, et al. Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 11080-11090.

编码器类型	FreiHand		DexYCB
编码器类型	E_PV	E_PJ	E_PV	E_PJ
HRNet-W64	6.48	6.72	5.02	5.90
Deit-Large	6.38	6.65	5.08	6.02

编码器类型	输入分辨率	参数量(M)	推理显存(G)
HRNet-W48	224×224	77.50	2.50
HRNet-W48	256×256	77.50	3.16
HRNet-W64	224×224	128.10	3.75
HRNet-W64	256×256	128.10	4.58
Deit-Large	224×224	304.15	5.31
Deit-Large	256×256	304.15	6.28

实验配置	假设数	推理速度(FPS)
RTX3060	2	22
	4	14
	8	5
RTX4080	2	42
	4	28
	8	12

跨模态解码器	扩散模型	关节优化网络	E_PV	E_PJ
×	×	×	7.87	7.08
×	√	√	6.99	6.74
×	×	√	7.88	8.15
×	√	×	9.88	7.82
×	√	√	7.09	6.59
×	√	√	7.85	8.10
√	√	×	7.65	7.85
√	√	√	6.48	6.72

建模网络	E_PV/mm	E_PJ/mm	参数量(M)
GCN	7.90	8.08	12.20
标准Transformer	6.89	7.21	47.30
Graphormer^[27]	6.52	6.78	52.10
跨模态解码器	6.48	6.72	37.19