Style Transfer Network for Generating Opera Makeup Details

doi:10.16182/j.issn1004731x.joss.22-1499

Abstract

Abstract:

To address the problem of the loss of local style details in cross-domain image simulation, a ChinOperaGAN network framework suitable for opera makeup is designed from the perspective of protecting the excellent traditional culture. In order to solve the style translation of differences in two image domains, multiple overlapping local adversarial discriminators are proposed in the generative adversarial network. Since paired opera makeup data are difficult to obtain, a synthetic image is generated by combining the source image makeup mapping to effectively guide the transfer of local makeup details between images. In view of the characteristics of opera makeup with strong and distinct colors, a loss function is introduced to ensure the generation of makeup images with high-frequency details. Experiments are carried out on open-source datasets and self-built datasets, and the classical method is better than the traditional classical method through qualitative and quantitative analysis. The experimental results show that the proposed method transfers the makeup by unsupervised adversarial learning and generates opera makeup style images with high-frequency details well. It can realize image transfer with consistent image features and matching style and can be applied to digital system simulation of intangible cultural heritage.

Key words: opera makeup transfer, generative adversarial networks, local feature extraction, detail generation, deep learning

CLC Number:

TP391.41

Zhang Fengquan, Cao Duo, Ma Xiaohan, Chen Baijun, Zhang Jiangxiao. Style Transfer Network for Generating Opera Makeup Details[J]. Journal of System Simulation, 2023, 35(9): 2064-2076.

Figures/Tables 13

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Table 1

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Table 2

References 35

1	钱文华, 徐丹, 徐瑾, 等. 东巴画艺术风格绘制[J].系统仿真学报, 2020, 32(7): 1349-1359.
	Qian Wenhua, Xu Dan, Xu Jin, et al. Simulation of Dongba Art Style Painting[J], Journal of System Simulation, 2020, 32(7): 1349-1359.
2	唐传谦, 刘志强, 苏毅俊, 等. 基于Kinect的皮影动作实时匹配研究[J]. 系统仿真学报, 2022, 34(3): 633-639.
	Tang Chuanqian, Liu Zhiqiang, Su Yijun, et al. Research on Real-time Motion Matching of Shadow Play based on Kinect[J]. Journal of System Simulation, 2022, 34(3): 633-639.
3	Xiang Jianfeng, Chen Junliang, Liu Wenshuang, et al. RamGAN: Region Attentive Morphing GAN for Region-level Makeup Transfer[C]//Computer Vision-ECCV 2022. Cham: Springer Nature Switzerland, 2022: 719-735.
4	Zhao Xiaodong, Chen Junliang, Liu Minmin, et al. Multi-scale Attention-based Feature Pyramid Networks for Object Detection[C]//Image and Graphics. Cham: Springer International Publishing, 2021: 405-417.
5	Zhang Xiaokang, Zhu Yuanlue, Chen Wenting, et al. Gated SwitchGAN for Multi-domain Facial Image Translation[J]. IEEE Transactions on Multimedia, 2022, 24: 1990-2003.
6	Nguyen T, Tran A, Hoai M. Lipstick Ain't Enough: Beyond Color Matching for In-the-wild Makeup Transfer[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2021: 13300-13309.
7	Liu Wenshuang, Chen Wenting, Yang Zhanjia, et al. Translate the Facial Regions You Like Using Self-adaptive Region Translation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2021: 2180-2188.
8	Liu Si, Jiang Wentao, Gao Chen, et al. PSGAN++: Robust Detail-preserving Makeup Transfer and Removal[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(11): 8538-8551.
9	Lee H Y, Tseng H Y, Mao Qi, et al. DRIT++: Diverse Image-to-image Translation via Disentangled Representations[J]. International Journal of Computer Vision, 2020, 128(10): 2402-2417.
10	Sun Zhaoyang, Chen Yaxiong, Xiong Shengwu. SSAT: A Symmetric Semantic-aware Transformer Network for Makeup Transfer and Removal[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2022: 2325-2334.
11	Isola P, Zhu Junyan, Zhou Tinghui, et al. Image-to-image Translation with Conditional Adversarial Networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2017: 5967-5976.
12	Kim T, Kim B, Cha M, et al. Unsupervised Visual Attribute Transfer with Reconfigurable Generative Adversarial Networks[EB/OL]. (2017-07-31) [2022-10-09]. .
13	Li Tingting, Qian Ruihe, Dong Chao, et al. BeautyGAN: Instance-level Facial Makeup Transfer with Deep Generative Adversarial Network[C]//Proceedings of the 26th ACM International Conference on Multimedia. New York, USA: Association for Computing Machinery, 2018: 645-653.
14	Jiang Wentao, Liu Si, Gao Chen, et al. PSGAN: Pose and Expression Robust Spatial-aware GAN for Customizable Makeup Transfer[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2020: 5193-5201.
15	Nirkin Y, Keller Y, Hassner T. FSGAN: Subject Agnostic Face Swapping and Reenactment[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway, NJ, USA: IEEE, 2019: 7183-7192.
16	Deng Han, Han Chu, Cai Hongmin, et al. Spatially-invariant Style-codes Controlled Makeup Transfer[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2021: 6545-6553.
17	Liu Si, Xinyu Ou, Qian Ruihe, et al. Makeup Like a Superstar: Deep Localized Makeup Transfer Network[C]//Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. Palo Alto, CA, USA: AAAI Press, 2016: 2568-2575.
18	Moniz J R A, Beckham C, Rajotte S, et al. Unsupervised Depth Estimation, 3D Face Rotation and Replacement[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates Inc., 2018: 9759-9769.
19	Zhang Hang, Dana K. Multi-style Generative Network for Real-time Transfer[C]//Computer Vision-ECCV 2018 Workshops. Cham: Springer International Publishing, 2019: 349-365.
20	Liu Hanwei, Cai Huiling, Lin Qincheng, et al. FEDA: Fine-grained Emotion Difference Analysis for Facial Expression Recognition[J]. Biomedical Signal Processing and Control, 2023, 79, Part 2: 104209.
21	Ho S T, Nguyen V T, Ngo T D. Interpolation Based Anime Face Style Transfer[C]//2020 International Conference on Multimedia Analysis and Pattern Recognition(MAPR). Piscataway, NJ, USA: IEEE, 2020: 1-6.
22	Tzaban R, Mokady R, Gal R, et al. Stitch It in Time: GAN-based Facial Editing of Real Videos[C]//SIGGRAPH Asia 2022 Conference Papers. New York, NY, USA: Association for Computing Machinery, 2022: 29.
23	Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Generative Adversarial Nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 2014: 2672-2680.
24	Ji Wei, Guo Jing, Li Yun. Multi-head Mutual-attention CycleGAN for Unpaired Image-to-image Translation[J]. IET Image Processing, 2020, 14(11): 2395-2402.
25	Wu Zongze, Lischinski D, Shechtman E. StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2021: 12858-12867.
26	Karras T, Laine S, Aittala M, et al. Analyzing and Improving the Image Quality of StyleGAN[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE. 2020: 8107-8116.
27	Cai Yali, Wang Xiaoru, Yu Zhihong, et al. Dualattn-GAN: Text to Image Synthesis with Dual Attentional Generative Adversarial Network[J]. IEEE Access, 2019, 7: 183706-183716.
28	Radford A, Metz L, Chintala S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[EB/OL]. (2016-01-07) [2022-09-11]. .
29	Wang Tingchun, Liu Mingyu, Zhu Junyan, et al. High-resolution Image Synthesis and Semantic Manipulation with Conditional GANs[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2018: 8798-8807.
30	Ma Yifei, Dong Yuntai, Li Kang, et al. A Survey of Chinese Character Style Transfer[C]//Image and Graphics Technologies and Applications. Singapore: Springer Singapore, 2019: 392-404.
31	Zhang Fengquan, Gao Huaming, Lai Yuping. Detail-preserving CycleGAN-adain Framework for Image-to-ink Painting Translation[J]. IEEE Access, 2020, 8: 132002-132011.
32	Takahashi N, Singh M K, Mitsufuji Y. Cross-modal Face- and Voice-style Transfer[EB/OL]. (2023-03-01) [2023-03-03]. .
33	Cao Junyan, Hong Yan, Niu Li. Painterly Image Harmonization in Dual Domains[EB/OL]. (2023-02-28) [2023-03-03]. .
34	Tong Waishun, Tang C K, Brown M S, et al. Example-based Cosmetic Transfer[C]//15th Pacific Conference on Computer Graphics and Applications (PG'07). Piscataway, NJ, USA: IEEE, 2007: 211-218.
35	Guo Dong, Sim T. Digital Face Makeup by Example[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2009: 73-79.

方法		眉毛	眼影	脸颊	嘴唇	总和
Digital Face Makeup	颜色	(30,28,15)	(166,53,47)	(163,120,114)	(196,84,68)
Digital Face Makeup	色差	23.52	46.90	68.25	38.18	176.85
CycleGAN	颜色	(36,24,12)	(73,53,42)	(197,150,134)	(185,96,114)
CycleGAN	色差	24.78	129.59	32.16	71.62	258.14
BeautyGAN	颜色	(40,29,25)	(140,70,62)	(196,147,133)	(188,47,55)
BeautyGAN	色差	10.39	59.35	33.06	7.35	110.15
ChinOperaGAN(Ours)	颜色	(58,23,17)	(182,57,63)	(191,147,148)	(181,54,37)
ChinOperaGAN(Ours)	色差	25.38	24.17	19.31	23.62	92.48

[1]	Wang Bingheng, Liu Tingrui, Yang Fan, Zhang Huan, Li Wei, Ma Ping, Yang Ming. Research on Requirements and Methods for Intelligent Assessment of Simulation Credibility [J]. Journal of System Simulation, 2025, 37(7): 1710-1722.
[2]	Chen Kun, Chen Liang, Xie Jiming, Liu Fengbo, Chen Taixiong, Wei Lukuan. Simulation Study on Adaptive Signal Control of Deformed Intersection Based on LSTM-GNN [J]. Journal of System Simulation, 2025, 37(6): 1343-1351.
[3]	Jiang Dawei, Dong Yangyang, Zhang Lidong, Lu Xiao, Dong Chunxi. Research on Air Target Threat Assessment Technology Based on Deep Learning [J]. Journal of System Simulation, 2025, 37(3): 791-802.
[4]	Wang Xiao, Li Xiangyang, Liang Feng, Zhang Zhili. Research on Infrared and Visible Light Fusion Method Based on ResNet-50 and Laplacian Filtering [J]. Journal of System Simulation, 2025, 37(12): 3202-3211.
[5]	Hu yang, Li Zihao, Fu Deyi, Song Ziqiu, Fang Fang, Liu Jizhen. Deep Learning Modeling of Multi-scale Characteristics of Large-scale Wind Turbine Gearbox [J]. Journal of System Simulation, 2025, 37(10): 2454-2468.
[6]	Gu Hao, Wang Jiayu, Xiong Weili. Soft Sensor Modeling Based on Improved Transformer in Dual-stream Framework [J]. Journal of System Simulation, 2025, 37(10): 2594-2604.
[7]	Guo Yecai, Tong Shuang. A Multimodal Residual Spatial-temporal Fusion Model Based on Automatic Sleep Classification [J]. Journal of System Simulation, 2024, 36(9): 2065-2074.
[8]	Liu Zesen, Bi Sheng, Guo Chuanhong, Wang Yankui, Dong Min. Deep Learning Based Local Path Planning Method for Moving Robots [J]. Journal of System Simulation, 2024, 36(5): 1199-1210.
[9]	Wei Jinyang, Wang Keping, Yang Yi, Fei Shumin. Incremental Image Dehazing Algorithm Based on Multiple Transfer Attention [J]. Journal of System Simulation, 2024, 36(4): 969-980.
[10]	Yang Zhe, Cui Yinghan, Guo Lingxi, Li Jiaxin, Wu Xusheng. Search Technology for Aircraft Debris Integrating Data Augmentation and Deep Learning Algorithm [J]. Journal of System Simulation, 2024, 36(10): 2238-2245.
[11]	Li Chen, He Ming, Dong Chen, Li Wei. Action Recognition Model of Directed Attention Based on Cosine Similarity [J]. Journal of System Simulation, 2024, 36(1): 67-82.
[12]	Yu Du, Xinquan Yang, Jianhua Zhang, Suchun Yuan, Huachao Xiao, Jingjing Yuan. Modulation Recognition Method of Mixed Signal Based on Intelligent Analysis of Cyclic Spectrum Section [J]. Journal of System Simulation, 2023, 35(1): 146-157.
[13]	Dayong Zhang, Jingyu Yang, Xi Wu. Research on Intelligent Prediction Method of Wargaming Air Mission [J]. Journal of System Simulation, 2023, 35(1): 212-220.
[14]	Huimin Chai, Yong Zhang, Xinyue Li, Yanan Song. Aerial Target Threat Assessment Method based on Deep Learning [J]. Journal of System Simulation, 2022, 34(7): 1459-1467.
[15]	Xu Kang, Xiaofeng Zhang. Radar Remote Sensing Data Augmentation Method Based on Generative Adversarial Network [J]. Journal of System Simulation, 2022, 34(4): 920-927.