基于引导对抗网络的人体深度图像修补方法

doi:10.16182/j.issn1004731x.joss.19-VR0462

系统仿真学报 ›› 2020, Vol. 32 ›› Issue (7): 1312-1321.doi: 10.16182/j.issn1004731x.joss.19-VR0462

基于引导对抗网络的人体深度图像修补方法

阴敬方¹, 朱登明^{1, 3, *}, 石敏², 王兆其^{1, 3}

1. 中国科学院计算技术研究所,北京 100190;
2. 华北电力大学,北京 102206;
3. 太仓中科信息技术研究院,江苏太仓 215400

收稿日期:2019-08-29 修回日期:2019-11-13 出版日期:2020-07-25 发布日期:2020-07-15
通讯作者: 朱登明(1973-),男,安徽太湖,博士,副研究员,研究方向为虚拟现实、计算机图形学。
作者简介:阴敬方(1997-),男,山东莱州,硕士生,研究方向为虚拟现实、计算机图像与图形

Human Depth Maps Restoration Based on Guided GAN

Yin Jingfang¹, Zhu Dengming^{1, 3, *}, Shi Min², Wang Zhaoqi^{1, 3}

1. Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China;
2. University of Chinese Academy of Sciences, Beijing 102206, China;
3. Taicang-CAS Institute of Information and Technology, Taicang 215400, China

Received:2019-08-29 Revised:2019-11-13 Online:2020-07-25 Published:2020-07-15

摘要/Abstract

摘要： 移动设备配备的小型深度相机采集到的人体深度图像存在严重的孔洞问题。针对该问题,提出基于深度学习的引导对抗网络。使用基于堆叠沙漏网络的引导器从RGB图像中提取人体部分分割特征和深度类别特征;在上述人体特征引导下,使用独特的生成器修复人体深度图像中的孔洞。为使结果更加逼真,加入判别器在网络训练过程中对生成器进行优化调整。实验结果显示,该方法在现有的人体数据集和小型深度相机采集的数据集上,都能很好解决孔洞问题,均取得比现有方法更好的效果。

关键词: 深度相机, 人体深度图像修复, 深度学习, 堆叠沙漏网络, 引导对抗网络

Abstract: The depth maps captured by a small depth camera on mobile devices suffer from the problem of severe holes. The Guided Generative Adversarial Network (Guided GAN) based on deep learning is proposed to restore human depth maps with above problems. The high-precision human segmentation features and depth class features are extracted from the monocular RGB image by the guider based on the stacked hourglass network. The holes in the human depth maps are filled by the special generator under the guidance of the extracted human features. In order to get the more realistic results, the discriminator is introduced to optimize the generator. The experimental results show that the proposed method can restore the human depth maps effectively in the existing human datasets and the dataset collected by the small depth camera. It achieves better results than the existing method.

Key words: RGBD camera, human depth data restoration, deep learning, two-stage stacked hourglass network, guided GAN

中图分类号:

TP391.9

阴敬方, 朱登明, 石敏, 王兆其. 基于引导对抗网络的人体深度图像修补方法[J]. 系统仿真学报, 2020, 32(7): 1312-1321.

Yin Jingfang, Zhu Dengming, Shi Min, Wang Zhaoqi. Human Depth Maps Restoration Based on Guided GAN[J]. Journal of System Simulation, 2020, 32(7): 1312-1321.

参考文献 33

[1]	Tukey J W.Exploratory Data Analysis (preliminary ed.)[M]. Reading MA: Addison-Wesley, 1971.
[2]	Bertalmio M, Sapiro G, Caselles V, et al.Image Inpainting[C]. In ACM Transactions on Graphics. USA: ACM Press/Addison-Wesley Publishing Co., 2000, 417-424.
[3]	Barnes C, Shechtman E, Finkelstein A, et al.Patch-Match: A Randomized Correspondence Algorithm for Structural Image Editing[J]. ACM Transactions on Graphics (S0730-0301), 2009, 28(3): 24-35.
[4]	Darabi S, Shechtman E, Barnes C, et al.Image Melding: Combining Inconsistent Images using Patch-based Synthesis[J]. ACM Transactions on Graphics (S0730-0301), 2012, 31(4): 82-92.
[5]	Köhler R, Schuler C, Schölkopf B, et al.Mask-specific inpainting with deep neural networks[C]. In German Conference on Pattern Recognition. German: Springer, Cham, 2014: 523-534.
[6]	Ren J S J, Xu L, Yan Q, et al. Shepard Convolutional Neural Networks[C]. In Conference on Neural Information Processing Systems. New York: Curran Associates, Inc., 2015: 901-909.
[7]	Xie J Y, Xu L L, Chen E H.Image Denoising and Inpainting with Deep Neural Networks[C]. In Conference on Neural Information Processing Systems. New York,NY: Curran Associates, Inc., 2012: 341-349.
[8]	Pathak D, Krähenbühl P, Donahue J, et al.Context Encoders: Feature Learning by Inpainting[C]. In IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 2536-2544.
[9]	Goodfellow I J, Pouget-Abadie J, Mirza M, et al.Generative Adversarial Nets[C]. In Conference on Neural Information Processing Systems. New York: Curran Associates, Inc., 2014: 2672-2680.
[10]	Newell A, Yang K, Deng J.Stacked hourglass networks for human pose estimation[C]. European Conference on Computer Vision. Cham: Springer, 2016: 483-499.
[11]	Dollár P, Wojek C, Schiele B, et al.Pedestrian Detection: A Benchmark[C]. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2009: 304-311.
[12]	Pishchulin L, Andriluka M, Gehler P, et al.Poselet conditioned pictorial structures[C]. In CVPR. Portland, Oregon: IEEE, 2013: 588-595.
[13]	Yang Y, Ramanan D.Articulated pose estimation with flexible mixtures-of-parts[C]. In CVPR. Providence, RI: IEEE, 2011. 1385-1392.
[14]	Chen X, Yuille A L.Articulated pose estimation by a graphical model with image dependent pairwise relations[C]. In NIPS. New York: Curran Associates, Inc., 2014. 1736-1744.
[15]	Chu X, Ouyang W, Li H, et al.Structured feature learning for pose estimation[C]. In CVPR. Las Vegas, NV: IEEE, 2016. 4715-4723.
[16]	Zhou X, Huang Q, Sun X, et al.Towards 3d human pose estimation in the wild: a weakly-supervised approach[C]. In IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 398-407.
[17]	Yang W, Ouyang W, Wang X, et al.3D Human Pose Estimation in the Wild by Adversarial Learning[C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT: IEEE, 2018, 5255-5264.
[18]	Long J, Shelhamer E, Darrell T.Fully Convolutional Networks for Semantic Segmentation[C]. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA: IEEE, 2015: 3431-3440.
[19]	Zhao J, Li J S, Cheng Y, et al.Understanding Humans in Crowded Scenes: Deep Nested Adversarial Learning and A New Benchmark for Multi-Human Parsing[C]. In MM '18 Proceedings of the 26th ACM international conference on Multimedia. New York: Association for Computing Machinery, 2018: 792-800.
[20]	Varol G, Romero J, Martin X, et al.Learning from synthetic humans[C]. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). Venice: IEEE, 2017: 109-117.
[21]	Jacques J J C S, Jung C R, Musse S R. Skeleton-based human segmentation in still images[C]. 19th IEEE International Conference on Image Processing, Orlando, FL: IEEE, 2012: 141-144.
[22]	Rother C, Kolmogorov V, Blake A."GrabCut": interactive foreground extraction using iterated graph cuts[J]. ACM Transactions on Graphics (S0730-0301), 2004, 23(3): 309-314.
[23]	Ionescu C, Papava D, Olaru V, et al.Human3.6M: Large scale datasets and predictive methods for 3d human sensing in natural environments[J]. IEEE transactions on pattern analysis and machine intelligence (S0162-8828). 2014, 36(7): 1325-1339.
[24]	Loper M, Mahmood N, Romero J, et al.SMPL: A Skinned Multi-Person Linear Model[J]. ACM Transactions on Graphics (S0730-0301), 2015, 34(6): 248.
[25]	Ma X, Geng Z, Bie Z.Depth estimation from single image using cnn-residual network[Z].
[26]	Chen W, Fu Z, Yang D, et al.Single-image depth perception in the wild[C]. Advances in Neural Information Processing Systems. New York: Curran Associates, Inc., 2016: 730-738.
[27]	Liu B, Gould S, Koller D.Single image depth estimation from predicted semantic labels[C]. Computer Vision and Pattern Recognition(CVPR), San Francisco : IEEE, 2010: 1253-1260.
[28]	Liu W, Chen X, Yang J, et al.Robust Color Guided Depth Map Restoration[J]. IEEE Transactions on Image Processing (S1057-7149), 2017, 26(1): 315-327.
[29]	Hays J, Efros A A.Scene Completion Using Millions of Photographs[J]. ACM Transactions on Graphics (S0730-0301), 2007, 26(3): 4-12.
[30]	Satoshi Iizuka, EDGAR SIMO-SERRA, HIROSHI ISHIKAWA.Globally and Locally Consistent Image Completion[J]. In ACM Transaction on Graphics (S0730-0301), 2017, 36(4): 1-14.
[31]	Chen Y, Tai Y, Liu X M, et al.FSRNet: End-to-End Learning Face Super-Resolution With Facial Priors[C]. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, UT: IEEE, 2018: 2492-2501.
[32]	Yu F, Koltun V.Multi-Scale Context Aggregation by Dilated Convolutions[C]. In International Conference on Learning Representations. San Juan, Puerto Rico: Computational and Biological Learning Society, 2016.
[33]	Pérez P, Gangnet M, Blake A.Poisson Image Editing[J]. ACM Transactions on Graphics (S0730-0301), 2003, 22(3): 313-318.

基于引导对抗网络的人体深度图像修补方法

Human Depth Maps Restoration Based on Guided GAN

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献 33

相关文章 15

编辑推荐

Metrics

本文评价

[1]	康旭, 张晓峰. 基于生成对抗神经网络的雷达遥感数据增广方法[J]. 系统仿真学报, 2022, 34(4): 920-927.
[2]	林硕, 安磊, 高治军, 单丹, 尚文利. 结合栈式自编码及长短时记忆的入侵检测研究[J]. 系统仿真学报, 2021, 33(6): 1288-1296.
[3]	程文聪, 史小康, 王志刚. 基于生成对抗网络的仿真卫星云图生成方法[J]. 系统仿真学报, 2021, 33(6): 1297-1306.
[4]	董书琴, 张斌. 面向不平衡数据的网络流量异常检测方法[J]. 系统仿真学报, 2021, 33(3): 679-689.
[5]	王霄汉, 张霖, 任磊, 谢堃钰, 王昆玉, 叶飞, 陈真. 基于强化学习的车间调度问题研究简述[J]. 系统仿真学报, 2021, 33(12): 2782-2791.
[6]	王步维, 王敏, 范谦, 王雅男, 章涵文, 乐云亮. 基于深度学习的晶体性质预测研究[J]. 系统仿真学报, 2021, 33(12): 2854-2863.
[7]	冯晓, 张辉, 周蕊, 乔璐, 魏东, 李丹丹, 张玉尧, 郑国清. 基于深度学习和籽粒双面特征的玉米品种识别[J]. 系统仿真学报, 2021, 33(12): 2983-2991.
[8]	杜金莲, 李淑飞, 金雪云. 三维烟雾流场超分辨率数据生成网络模型的研究[J]. 系统仿真学报, 2021, 33(10): 2381-2389.
[9]	邹序焱, 何汉武, 吴悦明, 邓景威. 基于手势交互的虚实融合实验仿真技术[J]. 系统仿真学报, 2021, 33(10): 2488-2498.
[10]	刘瑞军, 王向上, 张晨, 章博华. 基于深度学习的视觉SLAM综述[J]. 系统仿真学报, 2020, 32(7): 1244-1256.
[11]	戢晓峰, 戈艺澄. 基于深度学习的节假日高速公路交通流预测方法[J]. 系统仿真学报, 2020, 32(6): 1164-1171.
[12]	孔锐, 谢玮, 雷泰. 基于神经网络的图像描述方法研究[J]. 系统仿真学报, 2020, 32(4): 601-611.
[13]	秦胜伟, 李重, 李金锋, 陈梓浩, 丁靖骞, 刘万顺. 校园漫游互动AR系统设计与实现[J]. 系统仿真学报, 2019, 31(7): 1367-1376.
[14]	叶继华, 时淑霞, 李汉曦, 王仕民, 杨思渝. 基于深度学习的驾驶关注区域检测研究与实现[J]. 系统仿真学报, 2019, 31(7): 1421-1428.
[15]	叶继华, 时淑霞, 李汉曦, 左家莉, 王仕民. 一种结合深度学习和随机森林的地平线检测方法[J]. 系统仿真学报, 2018, 30(7): 2507-2514.