Style Transfer Network for Generating Opera Makeup Details

doi:10.16182/j.issn1004731x.joss.22-1499

Abstract

Abstract:

To address the problem of the loss of local style details in cross-domain image simulation, a ChinOperaGAN network framework suitable for opera makeup is designed from the perspective of protecting the excellent traditional culture. In order to solve the style translation of differences in two image domains, multiple overlapping local adversarial discriminators are proposed in the generative adversarial network. Since paired opera makeup data are difficult to obtain, a synthetic image is generated by combining the source image makeup mapping to effectively guide the transfer of local makeup details between images. In view of the characteristics of opera makeup with strong and distinct colors, a loss function is introduced to ensure the generation of makeup images with high-frequency details. Experiments are carried out on open-source datasets and self-built datasets, and the classical method is better than the traditional classical method through qualitative and quantitative analysis. The experimental results show that the proposed method transfers the makeup by unsupervised adversarial learning and generates opera makeup style images with high-frequency details well. It can realize image transfer with consistent image features and matching style and can be applied to digital system simulation of intangible cultural heritage.

Key words: opera makeup transfer, generative adversarial networks, local feature extraction, detail generation, deep learning

CLC Number:

TP391.41

Zhang Fengquan, Cao Duo, Ma Xiaohan, Chen Baijun, Zhang Jiangxiao. Style Transfer Network for Generating Opera Makeup Details[J]. Journal of System Simulation, 2023, 35(9): 2064-2076.

Figures/Tables 13

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Table 1

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Table 2

References 35

1	钱文华, 徐丹, 徐瑾, 等. 东巴画艺术风格绘制[J].系统仿真学报, 2020, 32(7): 1349-1359.
	Qian Wenhua, Xu Dan, Xu Jin, et al. Simulation of Dongba Art Style Painting[J], Journal of System Simulation, 2020, 32(7): 1349-1359.
2	唐传谦, 刘志强, 苏毅俊, 等. 基于Kinect的皮影动作实时匹配研究[J]. 系统仿真学报, 2022, 34(3): 633-639.
	Tang Chuanqian, Liu Zhiqiang, Su Yijun, et al. Research on Real-time Motion Matching of Shadow Play based on Kinect[J]. Journal of System Simulation, 2022, 34(3): 633-639.
3	Xiang Jianfeng, Chen Junliang, Liu Wenshuang, et al. RamGAN: Region Attentive Morphing GAN for Region-level Makeup Transfer[C]//Computer Vision-ECCV 2022. Cham: Springer Nature Switzerland, 2022: 719-735.
4	Zhao Xiaodong, Chen Junliang, Liu Minmin, et al. Multi-scale Attention-based Feature Pyramid Networks for Object Detection[C]//Image and Graphics. Cham: Springer International Publishing, 2021: 405-417.
5	Zhang Xiaokang, Zhu Yuanlue, Chen Wenting, et al. Gated SwitchGAN for Multi-domain Facial Image Translation[J]. IEEE Transactions on Multimedia, 2022, 24: 1990-2003.
6	Nguyen T, Tran A, Hoai M. Lipstick Ain't Enough: Beyond Color Matching for In-the-wild Makeup Transfer[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2021: 13300-13309.
7	Liu Wenshuang, Chen Wenting, Yang Zhanjia, et al. Translate the Facial Regions You Like Using Self-adaptive Region Translation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2021: 2180-2188.
8	Liu Si, Jiang Wentao, Gao Chen, et al. PSGAN++: Robust Detail-preserving Makeup Transfer and Removal[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(11): 8538-8551.
9	Lee H Y, Tseng H Y, Mao Qi, et al. DRIT++: Diverse Image-to-image Translation via Disentangled Representations[J]. International Journal of Computer Vision, 2020, 128(10): 2402-2417.
10	Sun Zhaoyang, Chen Yaxiong, Xiong Shengwu. SSAT: A Symmetric Semantic-aware Transformer Network for Makeup Transfer and Removal[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2022: 2325-2334.
11	Isola P, Zhu Junyan, Zhou Tinghui, et al. Image-to-image Translation with Conditional Adversarial Networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2017: 5967-5976.
12	Kim T, Kim B, Cha M, et al. Unsupervised Visual Attribute Transfer with Reconfigurable Generative Adversarial Networks[EB/OL]. (2017-07-31) [2022-10-09]. .
13	Li Tingting, Qian Ruihe, Dong Chao, et al. BeautyGAN: Instance-level Facial Makeup Transfer with Deep Generative Adversarial Network[C]//Proceedings of the 26th ACM International Conference on Multimedia. New York, USA: Association for Computing Machinery, 2018: 645-653.
14	Jiang Wentao, Liu Si, Gao Chen, et al. PSGAN: Pose and Expression Robust Spatial-aware GAN for Customizable Makeup Transfer[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2020: 5193-5201.
15	Nirkin Y, Keller Y, Hassner T. FSGAN: Subject Agnostic Face Swapping and Reenactment[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway, NJ, USA: IEEE, 2019: 7183-7192.
16	Deng Han, Han Chu, Cai Hongmin, et al. Spatially-invariant Style-codes Controlled Makeup Transfer[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2021: 6545-6553.
17	Liu Si, Xinyu Ou, Qian Ruihe, et al. Makeup Like a Superstar: Deep Localized Makeup Transfer Network[C]//Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2016: 2568-2575.
18	Moniz J R A, Beckham C, Rajotte S, et al. Unsupervised Depth Estimation, 3D Face Rotation and Replacement[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates Inc., 2018: 9759-9769.
19	Zhang Hang, Dana K. Multi-style Generative Network for Real-time Transfer[C]//Computer Vision-ECCV 2018 Workshops. Cham: Springer International Publishing, 2019: 349-365.
20	Liu Hanwei, Cai Huiling, Lin Qincheng, et al. FEDA: Fine-grained Emotion Difference Analysis for Facial Expression Recognition[J]. Biomedical Signal Processing and Control, 2023, 79, Part 2: 104209.
21	Ho S T, Nguyen V T, Ngo T D. Interpolation Based Anime Face Style Transfer[C]//2020 International Conference on Multimedia Analysis and Pattern Recognition(MAPR). Piscataway, NJ, USA: IEEE, 2020: 1-6.
22	Tzaban R, Mokady R, Gal R, et al. Stitch It in Time: GAN-based Facial Editing of Real Videos[C]//SIGGRAPH Asia 2022 Conference Papers. New York, NY, USA: Association for Computing Machinery, 2022: 29.
23	Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Generative Adversarial Nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 2014: 2672-2680.
24	Ji Wei, Guo Jing, Li Yun. Multi-head Mutual-attention CycleGAN for Unpaired Image-to-image Translation[J]. IET Image Processing, 2020, 14(11): 2395-2402.
25	Wu Zongze, Lischinski D, Shechtman E. StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2021: 12858-12867.
26	Karras T, Laine S, Aittala M, et al. Analyzing and Improving the Image Quality of StyleGAN[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE. 2020: 8107-8116.
27	Cai Yali, Wang Xiaoru, Yu Zhihong, et al. Dualattn-GAN: Text to Image Synthesis with Dual Attentional Generative Adversarial Network[J]. IEEE Access, 2019, 7: 183706-183716.
28	Radford A, Metz L, Chintala S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[EB/OL]. (2016-01-07) [2022-09-11]. .
29	Wang Tingchun, Liu Mingyu, Zhu Junyan, et al. High-resolution Image Synthesis and Semantic Manipulation with Conditional GANs[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2018: 8798-8807.
30	Ma Yifei, Dong Yuntai, Li Kang, et al. A Survey of Chinese Character Style Transfer[C]//Image and Graphics Technologies and Applications. Singapore: Springer Singapore, 2019: 392-404.
31	Zhang Fengquan, Gao Huaming, Lai Yuping. Detail-preserving CycleGAN-adain Framework for Image-to-ink Painting Translation[J]. IEEE Access, 2020, 8: 132002-132011.
32	Takahashi N, Singh M K, Mitsufuji Y. Cross-modal Face- and Voice-style Transfer[EB/OL]. (2023-03-01) [2023-03-03]. .
33	Cao Junyan, Hong Yan, Niu Li. Painterly Image Harmonization in Dual Domains[EB/OL]. (2023-02-28) [2023-03-03]. .
34	Tong Waishun, Tang C K, Brown M S, et al. Example-based Cosmetic Transfer[C]//15th Pacific Conference on Computer Graphics and Applications (PG'07). Piscataway, NJ, USA: IEEE, 2007: 211-218.
35	Guo Dong, Sim T. Digital Face Makeup by Example[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2009: 73-79.

方法		眉毛	眼影	脸颊	嘴唇	总和
Digital Face Makeup	颜色	(30,28,15)	(166,53,47)	(163,120,114)	(196,84,68)
Digital Face Makeup	色差	23.52	46.90	68.25	38.18	176.85
CycleGAN	颜色	(36,24,12)	(73,53,42)	(197,150,134)	(185,96,114)
CycleGAN	色差	24.78	129.59	32.16	71.62	258.14
BeautyGAN	颜色	(40,29,25)	(140,70,62)	(196,147,133)	(188,47,55)
BeautyGAN	色差	10.39	59.35	33.06	7.35	110.15
ChinOperaGAN(Ours)	颜色	(58,23,17)	(182,57,63)	(191,147,148)	(181,54,37)
ChinOperaGAN(Ours)	色差	25.38	24.17	19.31	23.62	92.48

[1]	Yu Du, Xinquan Yang, Jianhua Zhang, Suchun Yuan, Huachao Xiao, Jingjing Yuan. Modulation Recognition Method of Mixed Signal Based on Intelligent Analysis of Cyclic Spectrum Section [J]. Journal of System Simulation, 2023, 35(1): 146-157.
[2]	Dayong Zhang, Jingyu Yang, Xi Wu. Research on Intelligent Prediction Method of Wargaming Air Mission [J]. Journal of System Simulation, 2023, 35(1): 212-220.
[3]	Huimin Chai, Yong Zhang, Xinyue Li, Yanan Song. Aerial Target Threat Assessment Method based on Deep Learning [J]. Journal of System Simulation, 2022, 34(7): 1459-1467.
[4]	Xu Kang, Xiaofeng Zhang. Radar Remote Sensing Data Augmentation Method Based on Generative Adversarial Network [J]. Journal of System Simulation, 2022, 34(4): 920-927.
[5]	Jiarui Liu, Guotian Yang, Xiaowei Wang. A Wind Turbine Fault Diagnosis Method Based on Siamese Deep Neural Network [J]. Journal of System Simulation, 2022, 34(11): 2348-2358.
[6]	Rongxiu Lu, Bihao Zhang, Zhenlong Mo. Fatigue Detection Method Based on Facial Features and Head Posture [J]. Journal of System Simulation, 2022, 34(10): 2279-2292.
[7]	Weiguo Tong, Shichao Zeng, Lifeng Zhang, Zhe Hou, Jiayue Guo. Electrical Resistance Tomography and Flow Pattern Identification Method Based on Deep Residual Neural Network [J]. Journal of System Simulation, 2022, 34(09): 2028-2036.
[8]	Lin Shuo, An Lei, Gao Zhijun, Shan Dan, Shang Wenli. Research on Intrusion Detection Based on Stacked Autoencoder and Long-short Memory [J]. Journal of System Simulation, 2021, 33(6): 1288-1296.
[9]	Cheng Wencong, Shi Xiaokang, Wang Zhigang. Creating Synthetic Satellite Cloud Data Based on GAN Method [J]. Journal of System Simulation, 2021, 33(6): 1297-1306.
[10]	Li Xinli, Zou Changming, Yang Guotian, Liu He. Research of Super-resolution Processing of Invoice Image Based on Generative Adversarial Network [J]. Journal of System Simulation, 2021, 33(6): 1307-1314.
[11]	Dong Shuqin, Zhang Bin. Network Traffic Anomaly Detection Method for Imbalanced Data [J]. Journal of System Simulation, 2021, 33(3): 679-689.
[12]	Wang Xiaohan, Zhang Lin, Ren Lei, Xie Kunyu, Wang Kunyu, Ye Fei, Chen Zhen. Brief Review on Applying Reinforcement Learning to Job Shop Scheduling Problems [J]. Journal of System Simulation, 2021, 33(12): 2782-2791.
[13]	Wang Buwei, Wang Min, Fan Qian, Wang Ya'nan, zhang hanwen, Yue Yunliang. Study on Prediction of Crystal Properties Based on Deep Learning [J]. Journal of System Simulation, 2021, 33(12): 2854-2863.
[14]	Feng Xiao, Zhang Hui, Zhou Rui, Qiao Lu, Wei Dong, Li Dandan, Zhang Yuyao, Zheng Guoqing. Variety Recognition Based on Deep Learning and Double-Sided Characteristics of Maize Kernel [J]. Journal of System Simulation, 2021, 33(12): 2983-2991.
[15]	Du Jinlian, Li Shufei, Jin Xueyun. Research on the Network of 3D Smoke Flow Super-Resolution Data Generation [J]. Journal of System Simulation, 2021, 33(10): 2381-2389.