基于3DGS的可实时驱动人体化身生成研究

doi:10.16182/j.issn1004731x.joss.25-0829

摘要/Abstract

摘要：

3D人体化身生成及实时驱动技术在虚拟现实、远程协作等领域具有重要应用价值。针对现有方法在细节建模、实时性与新姿态驱动鲁棒性方面的不足，提出一种基于高斯泼溅(3D Gaussian splatting, 3DGS)的高效人体化身生成及驱动方法，结合优化参数化人体重建、三平面特征编码与动态偏移预测实现单目视频输入的高效建模。通过引入骨骼绑定与可见性分析策略，同时设计多尺度正则损失以解决过拟合问题。仿真实验结果表明：所提方法在各项指标上均取得了非常优异的表现，尤其在新姿态驱动与遮挡场景下表现出更强的鲁棒性，验证了方法的有效性与优越性。

关键词: 高斯泼溅, 可驱动人体化身, 单目视频, 实时渲染, 参数化模型

Abstract:

Real-time animatable 3D human avatar generation technology hold significant application value in fields such as virtual reality and remote collaboration. To address the limitations of existing methods in detail modeling, real-time performance, and robustness under novel pose driving, an efficient human avatar generation and driving method based on 3D Gaussian splatting (3DGS) is proposed. This method integrates optimized parametric human reconstruction, tri-plane feature encoding, and dynamic offset prediction to achieve efficient modeling from monocular video input. By introducing a skeleton binding and visibility analysis strategy, while designing a multi-scale regularization loss to address the overfitting problem. Simulation experiments demonstrate that the proposed method achieves outstanding performance across all evaluation metrics, particularly in novel pose driving and occluded scenarios, validating its effectiveness and superiority.

Key words: 3D Gaussian splatting (3DGS), animatable human avatars, monocular video, real-time rendering, parametric model

中图分类号:

TP391.9

钟雨攸,沈旭昆,胡勇 . 基于3DGS的可实时驱动人体化身生成研究[J]. 系统仿真学报, 2026, 38(1): 200-210.

Zhong Yuyou,Shen Xukun,Hu Yong . Research on Real-time Animatable Human Avatar Generation via 3D Gaussian Splatting[J]. Journal of System Simulation, 2026, 38(1): 200-210.

图/表 10

图1

图2

图3

图4

表1

对比实验结果

方法	NeuMan			ZJU-MoCap			People-Snapshot			FPS $↑$
方法	PSNR $↑$	SSIM $↑$	LPIPS $↓$	PSNR $↑$	SSIM $↑$	LPIPS $↓$	PSNR $↑$	SSIM $↑$	LPIPS $↓$	FPS $↑$
HumanNeRF	27.06	0.967	0.019	29.29	0.967	0.030	25.68	0.956	0.023	0.2
InstantAvatar	28.47	0.972	0.028	29.73	0.938	0.068	28.60	0.971	0.018	4.15
NeuMan	29.32	0.972	0.014	28.96	0.948	0.061	—	—	—	0.1
Vid2Avatar	31.18	0.982	0.025	15.72	0.942	0.060	28.70	0.975	0.042	0.005
GaussianAvatar	29.94	0.980	0.012	24.38	0.932	0.081	30.27	0.978	0.019	27.2
3DGS-Avatar	28.99	0.974	0.016	29.76	0.970	0.028	32.42	0.965	0.018	40.3
本文	32.09	0.982	0.012	31.65	0.989	0.011	32.82	0.982	0.017	45.8

表1

图5

图6

表2

参数化人体重建消融实验结果

参数是否优化	Epoch	PSNR $↑$	SSIM $↑$	LPIPS $↓$
不优化	100	27.82	0.966	0.020
不优化	300	31.05	0.974	0.015
优化	100	31.47	0.978	0.015
优化	300	32.09	0.982	0.012

表2

图7

图8

参考文献 26

[1]	Weng C Y, Curless B, Kemelmacher-Shlizerman I. Photo Wake-up: 3D Character Animation from a Single Photo[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2019: 5901-5910.
[2]	Zackariasson P, Wilson T L. The Video Game Industry: Formation, Present State, and Future[M]. New York: Routledge, 2012.
[3]	Li Nianlong, Zhang Zhengquan, Liu Can, et al. vMirror: Enhancing the Interaction with Occluded or Distant Objects in VR with Virtual Mirrors[C]//Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery, 2021: 132.
[4]	Liu Yang, Qiu Changzhen, Zhang Zhiyong. Deep Learning for 3D Human Pose Estimation and Mesh Recovery: A Survey[J]. Neurocomputing, 2024, 596: 128049.
[5]	Mildenhall B, Srinivasan P P, TANCIK M, et al. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis[C]//Computer Vision – ECCV 2020. Cham: Springer International Publishing, 2020: 405-421.
[6]	Kerbl B, Kopanas G, Leimkuehler T, et al. 3D Gaussian Splatting for Real-time Radiance Field Rendering[J]. ACM Transactions on Graphics, 2023, 42(4): 139.
[7]	Li Zhe, Zheng Zerong, Wang Lizhen, et al. Animatable Gaussians: Learning Pose-dependent Gaussian Maps for High-fidelity Human Avatar Modeling[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 19711-19722.
[8]	Hu Shoukang, Hu Tao, Liu Ziwei. GauHuman: Articulated Gaussian Splatting from Monocular Human Videos[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 20418-20431.
[9]	Hu Liangxiao, Zhang Hongwen, Zhang Yuxiang, et al. GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 634-644.
[10]	Bhatnagar B L, Sminchisescu C, Theobalt C, et al. LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2020: 12909-12922.
[11]	Loper M, Mahmood N, Romero J, et al. SMPL: A Skinned Multi-person Linear Model[J]. ACM Transactions on Graphics, 2015, 34(6): 248.
[12]	Weng C Y, Curless B, Srinivasan P P, et al. HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 16189-16199.
[13]	Jiang Wei, Yi K M, Samei G, et al. NeuMan: Neural Human Radiance Field from a Single Video[C]//Computer Vision – ECCV 2022. Cham: Springer Nature Switzerland, 2022: 402-418.
[14]	Guo Chen, Jiang Tianjian, Chen Xu, et al. Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2023: 12858-12868.
[15]	Peng Sen, Xie Weixing, Wang Zilong, et al. RMAvatar: Photorealistic Human Avatar Reconstruction from Monocular Video Based on Rectified Mesh-embedded Gaussians[EB/OL]. (2025-01-13) [2025-09-13]. .
[16]	Chen Jianchuan, Hu Jingchuan, Wang Gaige, et al. TaoAvatar: Real-time Lifelike Full-body Talking Avatars for Augmented Reality via 3D Gaussian Splatting[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2025: 10723-10734.
[17]	Pavlakos G, Choutas V, Ghorbani N, et al. Expressive Body Capture: 3D Hands, Face, and Body from a Single Image[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2019: 10967-10977.
[18]	Baradel F, Armando M, Galaaoui S, et al. Multi-HMR: Multi-person Whole-body Human Mesh Recovery in a Single Shot[C]//Computer Vision – ECCV 2024. Cham: Springer Nature Switzerland, 2025: 202-218.
[19]	Jiang Tao, Lu Peng, Zhang Li, et al. RTMPose: Real-time Multi-person Pose Estimation based on MMPose[EB/OL]. (2023-07-03) [2025-09-13]. .
[20]	Ravi N, Gabeur V, Hu Yuanting, et al. SAM 2: Segment Anything in Images and Videos[EB/OL]. (2024-10-28) [2025-09-13]. .
[21]	Jin Sheng, Xu Lumin, Xu Jin, et al. Whole-body Human Pose Estimation in the Wild[C]//Computer Vision – ECCV 2020. Cham: Springer International Publishing, 2020: 196-214.
[22]	Zhang R, Isola P, Efros A A, et al. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 586-595.
[23]	Peng Sida, Zhang Yuanqing, Xu Yinghao, et al. Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 9050-9059.
[24]	Alldieck T, Magnor M, Xu Weipeng, et al. Video Based Reconstruction of 3D People Models[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 8387-8397.
[25]	Jiang Tianjian, Chen Xu, Song Jie, et al. InstantAvatar: Learning Avatars from Monocular Video in 60 Seconds[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2023: 16922-16932.
[26]	Qian Zhiyin, Wang Shaofei, Mihajlovic M, et al. 3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 5020-5030.