一种面向戏曲妆容细节生成的风格迁移网络

doi:10.16182/j.issn1004731x.joss.22-1499

摘要/Abstract

摘要：

为解决跨域图像仿真中局部风格细节丢失问题，以保护传统优秀文化为视角，设计了一种适用于戏曲人脸妆容的ChinOperaGAN网络框架。为了解决两个图像域内差异的风格迁移，提出了在生成对抗网络中使用多个重叠的局部对抗性鉴别器；考虑到成对的戏曲人脸妆容数据较难获取问题，设计了结合源图像妆容映射生成合成图像来指导图像间局部妆容细节的有效迁移方法；针对戏曲人脸妆容色彩浓重分明的特点，引入了损失函数以约束生成具有高频细节的妆容图像。在开源数据集和自建数据集上进行实验，通过与经典的方法进行定性和定量分析，均优于传统经典方法。结果表明，通过无监督的对抗性学习完成了妆容的迁移，很好地生成了具有高频细节的戏曲人脸妆容风格图像，能够实现图像特征一致、风格匹配的图像迁移，成果可应用于非遗文化的数字化系统仿真，有利于传统文化的传承和发展。

关键词: 戏曲人脸妆容迁移, 生成对抗网络, 局部特征提取, 细节生成, 深度学习

Abstract:

To address the problem of the loss of local style details in cross-domain image simulation, a ChinOperaGAN network framework suitable for opera makeup is designed from the perspective of protecting the excellent traditional culture. In order to solve the style translation of differences in two image domains, multiple overlapping local adversarial discriminators are proposed in the generative adversarial network. Since paired opera makeup data are difficult to obtain, a synthetic image is generated by combining the source image makeup mapping to effectively guide the transfer of local makeup details between images. In view of the characteristics of opera makeup with strong and distinct colors, a loss function is introduced to ensure the generation of makeup images with high-frequency details. Experiments are carried out on open-source datasets and self-built datasets, and the classical method is better than the traditional classical method through qualitative and quantitative analysis. The experimental results show that the proposed method transfers the makeup by unsupervised adversarial learning and generates opera makeup style images with high-frequency details well. It can realize image transfer with consistent image features and matching style and can be applied to digital system simulation of intangible cultural heritage.

Key words: opera makeup transfer, generative adversarial networks, local feature extraction, detail generation, deep learning

中图分类号:

TP391.41

张凤全,曹铎,马晓寒等 . 一种面向戏曲妆容细节生成的风格迁移网络[J]. 系统仿真学报, 2023, 35(9): 2064-2076.

Zhang Fengquan,Cao Duo,Ma Xiaohan,et al . Style Transfer Network for Generating Opera Makeup Details[J]. Journal of System Simulation, 2023, 35(9): 2064-2076.

图/表 13

图1

图2

图3

图4

图5

表1

图6

图7

图 8

图9

图10

图11

表2

参考文献 35

1	钱文华, 徐丹, 徐瑾, 等. 东巴画艺术风格绘制[J].系统仿真学报, 2020, 32(7): 1349-1359.
	Qian Wenhua, Xu Dan, Xu Jin, et al. Simulation of Dongba Art Style Painting[J], Journal of System Simulation, 2020, 32(7): 1349-1359.
2	唐传谦, 刘志强, 苏毅俊, 等. 基于Kinect的皮影动作实时匹配研究[J]. 系统仿真学报, 2022, 34(3): 633-639.
	Tang Chuanqian, Liu Zhiqiang, Su Yijun, et al. Research on Real-time Motion Matching of Shadow Play based on Kinect[J]. Journal of System Simulation, 2022, 34(3): 633-639.
3	Xiang Jianfeng, Chen Junliang, Liu Wenshuang, et al. RamGAN: Region Attentive Morphing GAN for Region-level Makeup Transfer[C]//Computer Vision-ECCV 2022. Cham: Springer Nature Switzerland, 2022: 719-735.
4	Zhao Xiaodong, Chen Junliang, Liu Minmin, et al. Multi-scale Attention-based Feature Pyramid Networks for Object Detection[C]//Image and Graphics. Cham: Springer International Publishing, 2021: 405-417.
5	Zhang Xiaokang, Zhu Yuanlue, Chen Wenting, et al. Gated SwitchGAN for Multi-domain Facial Image Translation[J]. IEEE Transactions on Multimedia, 2022, 24: 1990-2003.
6	Nguyen T, Tran A, Hoai M. Lipstick Ain't Enough: Beyond Color Matching for In-the-wild Makeup Transfer[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2021: 13300-13309.
7	Liu Wenshuang, Chen Wenting, Yang Zhanjia, et al. Translate the Facial Regions You Like Using Self-adaptive Region Translation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2021: 2180-2188.
8	Liu Si, Jiang Wentao, Gao Chen, et al. PSGAN++: Robust Detail-preserving Makeup Transfer and Removal[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(11): 8538-8551.
9	Lee H Y, Tseng H Y, Mao Qi, et al. DRIT++: Diverse Image-to-image Translation via Disentangled Representations[J]. International Journal of Computer Vision, 2020, 128(10): 2402-2417.
10	Sun Zhaoyang, Chen Yaxiong, Xiong Shengwu. SSAT: A Symmetric Semantic-aware Transformer Network for Makeup Transfer and Removal[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2022: 2325-2334.
11	Isola P, Zhu Junyan, Zhou Tinghui, et al. Image-to-image Translation with Conditional Adversarial Networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2017: 5967-5976.
12	Kim T, Kim B, Cha M, et al. Unsupervised Visual Attribute Transfer with Reconfigurable Generative Adversarial Networks[EB/OL]. (2017-07-31) [2022-10-09]. .
13	Li Tingting, Qian Ruihe, Dong Chao, et al. BeautyGAN: Instance-level Facial Makeup Transfer with Deep Generative Adversarial Network[C]//Proceedings of the 26th ACM International Conference on Multimedia. New York, USA: Association for Computing Machinery, 2018: 645-653.
14	Jiang Wentao, Liu Si, Gao Chen, et al. PSGAN: Pose and Expression Robust Spatial-aware GAN for Customizable Makeup Transfer[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2020: 5193-5201.
15	Nirkin Y, Keller Y, Hassner T. FSGAN: Subject Agnostic Face Swapping and Reenactment[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway, NJ, USA: IEEE, 2019: 7183-7192.
16	Deng Han, Han Chu, Cai Hongmin, et al. Spatially-invariant Style-codes Controlled Makeup Transfer[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2021: 6545-6553.
17	Liu Si, Xinyu Ou, Qian Ruihe, et al. Makeup Like a Superstar: Deep Localized Makeup Transfer Network[C]//Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2016: 2568-2575.
18	Moniz J R A, Beckham C, Rajotte S, et al. Unsupervised Depth Estimation, 3D Face Rotation and Replacement[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates Inc., 2018: 9759-9769.
19	Zhang Hang, Dana K. Multi-style Generative Network for Real-time Transfer[C]//Computer Vision-ECCV 2018 Workshops. Cham: Springer International Publishing, 2019: 349-365.
20	Liu Hanwei, Cai Huiling, Lin Qincheng, et al. FEDA: Fine-grained Emotion Difference Analysis for Facial Expression Recognition[J]. Biomedical Signal Processing and Control, 2023, 79, Part 2: 104209.
21	Ho S T, Nguyen V T, Ngo T D. Interpolation Based Anime Face Style Transfer[C]//2020 International Conference on Multimedia Analysis and Pattern Recognition(MAPR). Piscataway, NJ, USA: IEEE, 2020: 1-6.
22	Tzaban R, Mokady R, Gal R, et al. Stitch It in Time: GAN-based Facial Editing of Real Videos[C]//SIGGRAPH Asia 2022 Conference Papers. New York, NY, USA: Association for Computing Machinery, 2022: 29.
23	Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Generative Adversarial Nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 2014: 2672-2680.
24	Ji Wei, Guo Jing, Li Yun. Multi-head Mutual-attention CycleGAN for Unpaired Image-to-image Translation[J]. IET Image Processing, 2020, 14(11): 2395-2402.
25	Wu Zongze, Lischinski D, Shechtman E. StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2021: 12858-12867.
26	Karras T, Laine S, Aittala M, et al. Analyzing and Improving the Image Quality of StyleGAN[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE. 2020: 8107-8116.
27	Cai Yali, Wang Xiaoru, Yu Zhihong, et al. Dualattn-GAN: Text to Image Synthesis with Dual Attentional Generative Adversarial Network[J]. IEEE Access, 2019, 7: 183706-183716.
28	Radford A, Metz L, Chintala S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[EB/OL]. (2016-01-07) [2022-09-11]. .
29	Wang Tingchun, Liu Mingyu, Zhu Junyan, et al. High-resolution Image Synthesis and Semantic Manipulation with Conditional GANs[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2018: 8798-8807.
30	Ma Yifei, Dong Yuntai, Li Kang, et al. A Survey of Chinese Character Style Transfer[C]//Image and Graphics Technologies and Applications. Singapore: Springer Singapore, 2019: 392-404.
31	Zhang Fengquan, Gao Huaming, Lai Yuping. Detail-preserving CycleGAN-adain Framework for Image-to-ink Painting Translation[J]. IEEE Access, 2020, 8: 132002-132011.
32	Takahashi N, Singh M K, Mitsufuji Y. Cross-modal Face- and Voice-style Transfer[EB/OL]. (2023-03-01) [2023-03-03]. .
33	Cao Junyan, Hong Yan, Niu Li. Painterly Image Harmonization in Dual Domains[EB/OL]. (2023-02-28) [2023-03-03]. .
34	Tong Waishun, Tang C K, Brown M S, et al. Example-based Cosmetic Transfer[C]//15th Pacific Conference on Computer Graphics and Applications (PG'07). Piscataway, NJ, USA: IEEE, 2007: 211-218.
35	Guo Dong, Sim T. Digital Face Makeup by Example[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2009: 73-79.

方法		眉毛	眼影	脸颊	嘴唇	总和
Digital Face Makeup	颜色	(30,28,15)	(166,53,47)	(163,120,114)	(196,84,68)
Digital Face Makeup	色差	23.52	46.90	68.25	38.18	176.85
CycleGAN	颜色	(36,24,12)	(73,53,42)	(197,150,134)	(185,96,114)
CycleGAN	色差	24.78	129.59	32.16	71.62	258.14
BeautyGAN	颜色	(40,29,25)	(140,70,62)	(196,147,133)	(188,47,55)
BeautyGAN	色差	10.39	59.35	33.06	7.35	110.15
ChinOperaGAN(Ours)	颜色	(58,23,17)	(182,57,63)	(191,147,148)	(181,54,37)
ChinOperaGAN(Ours)	色差	25.38	24.17	19.31	23.62	92.48

[1]	刘书刚, 张林坤, 杜昊东, 王洪涛. 雾天条件下改进YOLOv4的目标检测[J]. 系统仿真学报, 2023, 35(8): 1681-1691.
[2]	蔡兴泉, 李治均, 奚梦瑶, 孙海燕. 基于神经网络的手绘服饰图纹上色及风格迁移[J]. 系统仿真学报, 2023, 35(3): 604-615.
[3]	杜宇, 杨新权, 张建华, 袁素春, 肖化超, 袁晶晶. 基于循环谱截面智能分析的混合信号调制识别方法[J]. 系统仿真学报, 2023, 35(1): 146-157.
[4]	张大永, 杨镜宇, 吴曦. 兵棋推演空中任务智能预测方法研究[J]. 系统仿真学报, 2023, 35(1): 212-220.
[5]	柴慧敏, 张勇, 李欣粤, 宋雅楠. 基于深度学习的空中目标威胁评估方法[J]. 系统仿真学报, 2022, 34(7): 1459-1467.
[6]	曹建芳, 贾一鸣, 闫敏敏, 田晓东. 稳定增强生成对抗网络在壁画的超分辨率重建[J]. 系统仿真学报, 2022, 34(5): 1076-1089.
[7]	康旭, 张晓峰. 基于生成对抗神经网络的雷达遥感数据增广方法[J]. 系统仿真学报, 2022, 34(4): 920-927.
[8]	刘家瑞, 杨国田, 王孝伟. 基于孪生深度神经网络的风电机组故障诊断方法[J]. 系统仿真学报, 2022, 34(11): 2348-2358.
[9]	陆荣秀, 张笔豪, 莫振龙. 基于脸部特征和头部姿态的疲劳检测方法[J]. 系统仿真学报, 2022, 34(10): 2279-2292.
[10]	仝卫国, 曾世超, 张立峰, 侯哲, 郭佳跃. 基于深度残差神经网络的电阻层析成像及流型辨识方法[J]. 系统仿真学报, 2022, 34(09): 2028-2036.
[11]	吴曦, 孟祥林, 杨镜宇. 下一代战略博弈推演系统研究[J]. 系统仿真学报, 2021, 33(9): 2017-2024.
[12]	林硕, 安磊, 高治军, 单丹, 尚文利. 结合栈式自编码及长短时记忆的入侵检测研究[J]. 系统仿真学报, 2021, 33(6): 1288-1296.
[13]	程文聪, 史小康, 王志刚. 基于生成对抗网络的仿真卫星云图生成方法[J]. 系统仿真学报, 2021, 33(6): 1297-1306.
[14]	董书琴, 张斌. 面向不平衡数据的网络流量异常检测方法[J]. 系统仿真学报, 2021, 33(3): 679-689.
[15]	王霄汉, 张霖, 任磊, 谢堃钰, 王昆玉, 叶飞, 陈真. 基于强化学习的车间调度问题研究简述[J]. 系统仿真学报, 2021, 33(12): 2782-2791.