Costume Pattern Sketch Colorization and Style Transfer Based on Neural Network

doi:10.16182/j.issn1004731x.joss.21-1154

Abstract

Abstract:

Aiming at the problems of color overflow in pattern sketch colorization and lack of fabric texture features in style transfer, this paper proposes a method of costume pattern sketch colorization and style transfer based on neural network. This paper initializes the data set, collects the costume pattern image, extracts the costume pattern sketch, synthesizes the costume pattern sketch with color features and constructs the style data set. The research builds the conditional generative adversarial nets and achieves the costume pattern sketch with color features colorization based on the generator. The study constructs a convolutional neural network model, uses the model to calculate the content features of the content map and uses the Gram matrix to calculate the style features of the style map, introduces weight parameters to optimize the loss function, and outputs a satisfactory costume pattern transfer image. The experimental results show that the generated image has a real costume pattern color distribution and a good sense of fabric material.

Key words: costume pattern sketch, sketch colorization, style transfer, conditional generative adversarial network, convolutional neural network

CLC Number:

TP391.9

Xingquan Cai, Zhijun Li, Mengyao Xi, Haiyan Sun. Costume Pattern Sketch Colorization and Style Transfer Based on Neural Network[J]. Journal of System Simulation, 2023, 35(3): 604-615.

Figures/Tables 19

Fig. 1

Table 1

Table 2

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Fig. 14

Fig. 15

Fig. 16

Table 3

References 26

1	梁培俊, 刘怡俊. 基于条件生成对抗网络的漫画手绘图上色方法[J]. 计算机应用研究, 2019, 36(1): 308-311.
	Liang Peijun, Liu Yijun. Caricature Hand Drawing Coloring Method Based on Conditional Generative Con-frontation Network[J]. Application Research of Computers, 2019, 36(1): 308-311.
2	李媛, 陈昭炯, 叶东毅. 注意力引导的交互式工笔花卉上色算法[J]. 模式识别与人工智能, 2020, 33(7): 575-587.
	Li Yuan, Chen Zhaojiong, Ye Dongyi. Attention-Guided Interactive Meticulous Flower Coloring Algorithm[J]. Pattern Recognition and Artificial Intelligence, 2020, 33(7): 575-587.
3	Cheng Z, Yang Q, Sheng B. Deep Colorization[C]// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Santiago, Chile: IEEE Press, 2015: 415-423.
4	Yoo S, Bahng H, Chung S, et al. Coloring with Limited Data: Few-Shot Colorization Via Memory-Augmented Networks[C]// IEEE Conference on Computer Vision and Pattern Recognition. Los Angeles, USA: IEEE Press, 2019: 11283-11292.
5	Sangkloy P, Lu J, Fang C, et al. Scribbler: Controlling Deep Image Synthesis with Sketch and Color[C]// IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE Press, 2016: 5400-5409.
6	Güclütürk Y. Convolutional sketch inversion[C]// ECCV Workshops Computer Vision. Amsterdam, Netherlands: ECCV Press, 2016: 810-824.
7	Ci Y, Ma X, Wang Z, et al. User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks[C]// 26th ACM International Conference on Multimedia. Seoul, Korea: ACM Press, 2018: 1536-1544.
8	Mirza M, Osindero S. Conditional Generative Adversarial Nets[J]. Computer Science(S0098-3004), 2014, 48(4): 2672-2680.
9	Kron A, Nestmann F, SchlTer I, et al. Operational Flood Management under Large-Scale Extreme Conditions, Using the Example of the Middle Elbe[J]. Natural Hazards and Earth System Sciences (S1561-8633), 2010, 10(6): 1171-1181.
10	常佳, 王玉德, 吉燕妮. 基于改进的U-Net生成对抗网络的图像翻译算法[J]. 通信技术, 2020, 53(2): 327-334.
	Chang Jia, Wang Yude, Ji Yanni. Image Translation Algorithm Based on Improved U-Net Generative Adversarial Network[J]. Communication Technology, 2020, 53(2): 327-334.
11	Lee K H. A New Image Completion Method Inserting an Image Generated by Sketch Image[J]. International Journal of Innovative Technology and Exploring Engineering (S2278-3075), 2021, 10(4): 14-18.
12	Xie Minshan, Li Chengze, Liu Xueting, et al. Manga Filling Style Conversion with Screentone Variational Autoencoder[J]. ACM Transactions on Graphics (S0730-0301), 2020, 39(6): 22601-22615.
13	Frans K. Unsupervised Adversarial Defense through Tandem Deep Image Priors[J]. Electronics (S1087-4423), 2020, 9(11): 1957.
14	Aarti E. Generative Adversarial Networks and Their Variants[J]. Information Science (S0020-0255), 2021, 16(7): 59-80.
15	Zhang L, Li C, Wong T T, et al. Two-stage Sketch Colorization[J]. ACM Transactions on Graphics (S0730-0301), 2018, 37(6): 26101-26114.
16	陈淑環, 韦玉科, 徐乐. 基于深度学习的图像风格迁移研究综述[J]. 计算机应用研究, 2019, 36(8): 2250-2255.
	Chen Shuhuan, Wei Yuke, Xu Le. A Review of Research on Image Style Transfer Based on Deep Learning[J]. Application Research of Computers, 2019, 36(8): 2250-2255.
17	丁晓龙. 基于深度学习的图像风格迁移技术的前沿进展[J]. 电子制作, 2018, 18(1): 86-87.
	Ding Xiaolong. The Cutting-Edge Development of Image Style Transfer Technology Based on Deep Learning[J]. Electronic production, 2018, 18(1): 86-87.
18	张海嵩, 尹小勤. 实时绘制3D中国画效果[J]. 计算机辅助设计与图形学学报, 2004, 11(2): 1485-1489.
	Zhang Haisong, Yin Xiaoqin. Real-Time Rendering of 3D Chinese Painting Effects[J]. Journal of Computer Aided Design and Graphics, 2004, 11(2): 1485-1489.
19	苗序娟, 余浩, 王露. 图像风格迁移技术概况及研究现状[J]. 现代计算机, 2021, 6(2): 67-72.
	Miao Xujuan, Yu Hao, Wang Lu. General Situation and Research Status of Image Style Transfer Technology[J]. Modern Computer, 2021, 6(2): 67-72.
20	张菊, 郭永峰. 深度学习研究综述[J]. 教学研究, 2021, 44(3): 6-11.
	Zhang Ju, Guo Yongfeng. A Survey of Deep Learning Research[J]. Teaching Research, 2021, 44(3): 6-11.
21	Gatys L A, Ecker A S, Bethge M. Image Style Transfer Using Convolutional Neural Networks[C]// IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE Press, 2016: 2414-2423.
22	Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition[J]. Computer Science (S0098-3004), 2014, 15(4): 8570-8594.
23	Johnson J, Alahi A, Fei L. Perceptual Losses for Real-Time Style Transfer and Super-Resolution[C]// European Conference on Computer Vision. Amsterdam, Netherlands: ECCV Press, 2016: 694-711.
24	Huang X, Belongie S. Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization[C]// IEEE International Conference on Computer Vision. Venice, Italy: IEEE Press, 2017: 1510-1519.
25	Tian Q C, Schmidt M. Fast Patch-Based Style Transfer of Arbitrary Style[C]// IEEE Conference on Computer Vision and Pattern Recognition. Los Angeles, USA: IEEE Press, 2019: 5873-5881.
26	Zhu J, Park T, Isola P, et al. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks[C]// IEEE International Conference on Computer Vision. Venice, Italy: IEEE Press, 2017: 2242-225.

层数	结构	卷积核个数	输出尺寸
1	Conv	64	128×128
2	LReLU, Conv	128	64×64
3	LReLU, Conv	256	32×32
4~8	LReLU, Conv	512	16×16
9~12	ReLU, DeConv	512	32×32
13	ReLU, DeConv	256	64×64
14	ReLU, DeConv	128	128×128
15	ReLU, DeConv	64	256×256
16	ReLU, DeConv, Tanh	3	256×256

层数	结构	卷积核个数	输出尺寸
1	Conv	64	128×128
2	LReLU, Conv	128	64×64
3	LReLU, Conv	256	32×32
4	LReLU, Conv	512	32×32
5	ReLU, Conv, Sigmoid	1	32×32

图像	SSIM	PSNR
PanintsChainer	0.64	21.71
Style2Paint	0.65	22.72
CycleGan	0.61	22.87
本文方法	0.71	23.37

[1]	Fengwei Yang, Peng Chen, Kai Xi, Hualin Pu, Xueyin Liu. Online Classification Method for Motor Imagery EEG with Spatial Information [J]. Journal of System Simulation, 2023, 35(2): 254-267.
[2]	Lifeng Zhang, Huiren Wang. Image Reconstruction of Electrical Capacitance Tomography Based on Convolutional Neural Network and Finite Element Simulation [J]. Journal of System Simulation, 2022, 34(4): 712-718.
[3]	Bangqian Ao, Sha Yang, Jinqing Linghu, zhenhuan Ye. Design of Fatigue Driving Detection System Based on Cascaded Neural Network [J]. Journal of System Simulation, 2022, 34(2): 323-333.
[4]	Yaru Wang, Kai Yang, Yongjie Zhai, Congbin Guo, Wenqing Zhao, Jie Su. Transmission Line Insulator Recognition Based on Artificial Images Data Expansion [J]. Journal of System Simulation, 2022, 34(11): 2337-2347.
[5]	Jiale Xu, Haidong Zhang, Donghai Zhao, Wancheng Ni. Tactical Maneuver Strategy Learning from Land Wargame Replay Based on Convolutional Neural Network [J]. Journal of System Simulation, 2022, 34(10): 2181-2193.
[6]	Yecai Guo, Qingwei Wang. Modulation Recognition Algorithm Based on Truncated Migration and Parallel ResNet [J]. Journal of System Simulation, 2022, 34(09): 2009-2018.
[7]	Zhao Ying, Lu Yao, Zhang Jian, Liang Qidi, Long Wei. Multi-view Human Action Recognition Based on Deep Neural Network [J]. Journal of System Simulation, 2021, 33(5): 1019-1030.
[8]	Tong Weiguo, Pang Xuechun, Zhu Genghong. Gas-liquid Two-phase Flow Pattern Recognition Method Based on Convolutional Neural Network [J]. Journal of System Simulation, 2021, 33(4): 883-891.
[9]	Huang Xin, Fang Yu, Gu Mengdan. Classification of Chest X-ray Disease Based on Convolutional Neural Network [J]. Journal of System Simulation, 2020, 32(6): 1188-1194.
[10]	Fan Minglu, Wang Yan, Ji Zhicheng. Fault Diagnosis for Bearings of Unbalanced Data Based on Feature Generation [J]. Journal of System Simulation, 2020, 32(12): 2438-2448.
[11]	Wang Hongjun, Li Zouzou, Zou Xiangjun. Wood Surface Defect Detection Based on Adaboost and CNN [J]. Journal of System Simulation, 2019, 31(8): 1636-1645.
[12]	Lu Yuqiu, Sun Jinyu, Ma Shiwei. Moving Object Detection Based on Deep Convolutional Neural Network [J]. Journal of System Simulation, 2019, 31(11): 2275-2280.
[13]	Xia Min, Shen Maoyang, Wang Jianfeng, Wang Yangguang. Cloud Fraction of Satellite Imagery Based On Convolutional Neural Networks [J]. Journal of System Simulation, 2018, 30(5): 1623-1630.
[14]	Wu Yunpu, Jin Weidong, Huang Yingkun. Fault Diagnosis of High Speed Train Bogie Based on Multi-domain Fusion CNN [J]. Journal of System Simulation, 2018, 30(11): 4492-4497.
[15]	Wang Song, Liu Fuchang, Huang Ji, Xu Weiwei, Dong Hongwei. Pose Estimation Using Convolutional Neural Network with Synthesis Depth Data [J]. Journal of System Simulation, 2017, 29(11): 2618-2623.