Research on Latent Space-based Anime Face Style Transfer and Editing Techniques

doi:10.16182/j.issn1004731x.joss.24-FZ0797

Abstract

Abstract:

To address issues such as image distortion and style uniformity in existing anime style transfer networks within the field of image simulation, we propose the TGFE-TrebleStyleGAN (text-guided facial editing with TrebleStyleGAN) for anime facial style transfer and editing. This framework leverages vector guidance within the latent space to generate facial imagery and incorporates a detail control module and a feature control module to constrain the aesthetic attributes of the generated images. The images generated by the transfer network serve as style control signals and constraints for fine-grained segmentation. Text-to-image generation technology captures correlations between style-transferred images and semantic information. Experimental results on both open-source datasets and self-constructed datasets with paired attribute tags for anime faces demonstrate that the proposed model reduces the FID score by 2.819 compared to DualStyleGAN, improve the SSIM and NIMA scores by 0.028 and 0.074 respectively. Combining style transfer and editing retains anime facial details while allowing flexible adjustments, minimizing distortion and enhancing feature consistency and style similarity.

Key words: anime style transfer, GAN, latent space, anime facial editing, text-guided image generation

CLC Number:

TP391.9

Deng Haixin, Zhang Fengquan, Wang Nan, Zhang Wancai, Lei Jierui. Research on Latent Space-based Anime Face Style Transfer and Editing Techniques[J]. Journal of System Simulation, 2024, 36(12): 2834-2849.

Figures/Tables 19

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Table 1

Fig. 12

Fig. 13

Table 2

Fig. 14

Fig. 15

Fig. 16

Fig. 17

References 31

1	Karras T, Laine S, Aila T. A Style-based Generator Architecture for Generative Adversarial Networks[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2019: 4396-4405.
2	Rombach Robin, Blattmann Andreas, Lorenz Dominik, et al. High-resolution Image Synthesis with Latent Diffusion Models[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 10674-10685.
3	Jang Wonjong, Ju Gwangjin, Jung Yucheol, et al. StyleCariGAN: Caricature Generation via StyleGAN Feature Map Modulation[J]. ACM Transactions on Graphics, 2021, 40(4): 116.
4	Gatys Leon A, Ecker Alexander S, Bethge Matthias. Image Style Transfer Using Convolutional Neural Networks[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2016: 2414-2423.
5	Creswell A, White Tom, Dumoulin Vincent, et al. Generative Adversarial Networks: An Overview[J]. IEEE Signal Processing Magazine, 2018, 35(1): 53-65.
6	Kim Junho, Kim Minjae, Kang Hyeonwoo, et al. U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-instance Normalization for Image-to-image Translation[EB/OL]. (2020-04-08) [2024-07-01]. .
7	Cho Hansam, Lee Jonghyun, Chang Seunggyu, et al. One-shot Structure-aware Stylized Image Synthesis[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 8302-8311.
8	Liu Songhua, Lin Tianwei, He Dongliang, et al. AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2021: 6629-6638.
9	Chong M J, Forsyth D. GANs N' Roses: Stable, Controllable, Diverse Image to Image Translation (works for videos too!)[EB/OL]. (2021-06-11) [2024-06-17]. .
10	Yang Shuai, Jiang Liming, Liu Ziwei, et al. Pastiche Master: Exemplar-based High-resolution Portrait Style Transfer[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 7683-7692.
11	Zeng Wei, Ren Xiaozhe, Su Teng, et al. PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation[EB/OL]. (2021-04-26) [2024-06-17]. .
12	Radford A, Kim J W, Hallacy C, et al. Learning Transferable Visual Models from Natural Language Supervision[C]//Proceedings of the 38th International Conference on Machine Learning. Chia Laguna Resort: PMLR, 2021: 8748-8763.
13	Ramesh A, Dhariwal P, Nichol A, et al. Hierarchical Text-conditional Image Generation with CLIP Latents[EB/OL]. (2022-04-13) [2024-06-18]. .
14	Yu Tao, Feng Runseng, Feng Ruoyu, et al. Inpaint Anything: Segment Anything Meets Image Inpainting[EB/OL]. (2023-04-13) [2024-07-05]. .
15	Kirillov A, Mintun E, Ravi N, et al. Segment Anything[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2023: 3992-4003.
16	Suvorov Roman, Logacheva Elizaveta, Mashikhin Anton, et al. Resolution-robust Large Mask Inpainting with Fourier Convolutions[C]//2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Piscataway: IEEE, 2022: 3172-3182.
17	Collins Edo, Bala R, Price B, et al. Editing in Style: Uncovering the Local Semantics of GANs[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2020: 5770-5779.
18	Shen Yujun, Zhou Bolei. Closed-form Factorization of Latent Semantics in GANs[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 1532-1540.
19	Kim Gwanghyun, Kwon Taesung, Chul Ye Jong. DiffusionCLIP: Text-guided Diffusion Models for Robust Image Manipulation[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 2416-2425.
20	Zhang Lümin, Rao Anyi, Agrawala M. Adding Conditional Control to Text-to-image Diffusion Models[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2023: 3813-3824.
21	Ronneberger Olaf, Fischer Philipp, Brox Thomas. U-net: Convolutional Networks for Biomedical Image Segmentation[C]//Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015. Cham: Springer International Publishing, 2015: 234-241.
22	Huang Ziqi, C K Chan Kelvin, Jiang Yuming, et al. Collaborative Diffusion for Multi-modal Face Generation and Editing[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2023: 6080-6090.
23	Pinkney J N M, Adler D. Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains[EB/OL]. (2020-11-21) [2024-06-18]. .
24	Johnson J, Alahi A, Li Feifei. Perceptual Losses for Real-time Style Transfer and Super-resolution[C]//Computer Vision-ECCV 2016. Cham: Springer International Publishing, 2016: 694-711.
25	Deng Jiankang, Guo Jia, Xue Niannan, et al. ArcFace: Additive Angular Margin Loss for Deep Face Recognition[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2019: 4685-4694.
26	Song Shuang, Liang Yuanbang, Wu Jing, et al. Feature Proliferation-the "Cancer" in StyleGAN and Its Treatments[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2023: 2360-2370.
27	Kingma Diederik P, Welling Max. Auto-encoding Variational Bayes[EB/OL]. (2022-12-10) [2024-07-05]. .
28	Liu Shilong, Zeng Zhaoyang, Ren Tianhe, et al. Grounding DINO: Marrying DINO with Grounded Pre-training for Open-set Object Detection[EB/OL]. (2024-07-19) [2024-07-06]. .
29	Hu E J, Shen Yelong, Wallis P, et al. LoRA: Low-rank Adaptation of Large Language Models[EB/OL]. (2021-10-16) [2024-07-06]. .
30	Branwen Gwern, Arfafax, Presser Shawn, et al. Anime Crop Datasets: Faces, Figures, & Hands[EB/OL]. (2020-08-05) [2024-07-10]. .
31	Talebi H, Milanfar P. NIMA: Neural Image Assessment[J]. IEEE Transactions on Image Processing, 2018, 27(8): 3998-4011.

方法	FID	SSIM	NIMA
本文	158.564	0.821	4.682
DualStyle-GAN	161.383	0.793	4.608
U-GAT-IT	206.971	0.699	4.215
GNR	188.846	0.568	4.318
Diffusion	167.615	0.761	4.752

方法	风格相似	结构一致	编辑合理
本文	8.4	9.1	8.8
DualStyle-GAN	8.3	8.8	6.7
StyleGAN2	6.3	7.7	6.8
Diffusion	5.6	5.1	7.3

[1]	Liu Zewei, Ding Yishan, Lin Tingyu, Ke Mingxing, Guo Liqing, Xiao Yingying, Zhao Zhilong, Li Yan, xuan Lü. Research on Simulation Resource Management Based on Graph Association Organization [J]. Journal of System Simulation, 2024, 36(9): 1983-1994.
[2]	Wu Yunpeng, Fu Yingxiong, Shen Lijun, Cui Feng. Traffic Sign Recognition Model with Long-Tail Distribution Based on YOLOX-Tiny [J]. Journal of System Simulation, 2024, 36(11): 2503-2516.
[3]	Shugang Liu, Linkun Zhang, Haodong Du, Hongtao Wang. Improved Object Detection of YOLOv4 in Foggy Conditions [J]. Journal of System Simulation, 2023, 35(8): 1681-1691.
[4]	Guangqiu Huang, Xixuan Zhao, Qiuqin Lu. Calculation of Optimal VOCs Emission Reduction Based on Improved SEIRS Model in Cloud Environment [J]. Journal of System Simulation, 2023, 35(3): 632-645.
[5]	Gao Yan, Li Jimeng, Xu Jianzhong, Quan Hongyan. Terrain Surface Texture Generation Networks for User Semantics Customization [J]. Journal of System Simulation, 2023, 35(10): 2077-2086.
[6]	Guangqiu Huang, He Wang. Dynamic Risk Assessment of VOCs Cross Regional Flow Based on Petri Nets [J]. Journal of System Simulation, 2023, 35(1): 57-68.
[7]	Kan Li, Yunpeng Li, Jiangbo Zhao. Design and Implementation of UAV Swarm Self-organizing Search Model [J]. Journal of System Simulation, 2022, 34(8): 1820-1833.
[8]	Guangqiu Huang, Tiantian Wu. Cause Analysis of VOCs Hazards in Related Areas Based on Object Function Petri Net [J]. Journal of System Simulation, 2022, 34(3): 470-481.
[9]	Li Xinli, Zou Changming, Yang Guotian, Liu He. Research of Super-resolution Processing of Invoice Image Based on Generative Adversarial Network [J]. Journal of System Simulation, 2021, 33(6): 1307-1314.
[10]	Zhao Dan, Niu Xuejun, Zhang Shuhao, Wei Jiaxu. Study and Effect Evaluation on the Setting of Contraflow Left-Turn at Intersections [J]. Journal of System Simulation, 2021, 33(12): 2975-2982.
[11]	Li Fei, Su Zhong. Intelligent Control of Wastewater Treatment Processes Based on Adaptive Immune Optimization [J]. Journal of System Simulation, 2021, 33(12): 3012-3020.
[12]	Du Jinlian, Li Shufei, Jin Xueyun. Research on the Network of 3D Smoke Flow Super-Resolution Data Generation [J]. Journal of System Simulation, 2021, 33(10): 2381-2389.
[13]	Cao Cejun, Li Congdong. Overview of Optimization Problems Regarding Emergency Organization Allocation Based on Mathematical Programming [J]. Journal of System Simulation, 2021, 33(1): 1-12.
[14]	Yin Jingfang, Zhu Dengming, Shi Min, Wang Zhaoqi. Human Depth Maps Restoration Based on Guided GAN [J]. Journal of System Simulation, 2020, 32(7): 1312-1321.
[15]	Yang Wei, Zeng Liang, Kang Xinchen. Self-organizing Aggregation Behavior Modeling of Swarm Robots Based on Ant Colony Algorithm [J]. Journal of System Simulation, 2020, 32(2): 191-200.