基于神经网络的手绘服饰图纹上色及风格迁移

doi:10.16182/j.issn1004731x.joss.21-1154

摘要/Abstract

摘要：

针对图纹上色容易色彩溢出、风格迁移缺少布料纹理特征等问题，提出基于神经网络的手绘服饰图纹上色及风格迁移方法。初始化数据集，收集服饰图纹图像，提取服饰图纹黑白线稿，合成具有颜色特征的手绘图像，构建风格数据集；构建条件生成对抗网络模型，基于该生成器模型实现对具有颜色信息的线稿图像上色；构建卷积神经网络模型，利用该模型计算内容图的内容特征并结合Gram矩阵计算风格图的风格特征，输出令人满意的服饰图纹迁移图像。实验结果表明，该方法生成的图像具有真实的服饰图纹颜色分布，具有较好的布料材质感。

关键词: 手绘服饰图纹, 线稿图像上色, 风格迁移, 条件生成对抗网络, 卷积神经网络

Abstract:

Aiming at the problems of color overflow in pattern sketch colorization and lack of fabric texture features in style transfer, this paper proposes a method of costume pattern sketch colorization and style transfer based on neural network. This paper initializes the data set, collects the costume pattern image, extracts the costume pattern sketch, synthesizes the costume pattern sketch with color features and constructs the style data set. The research builds the conditional generative adversarial nets and achieves the costume pattern sketch with color features colorization based on the generator. The study constructs a convolutional neural network model, uses the model to calculate the content features of the content map and uses the Gram matrix to calculate the style features of the style map, introduces weight parameters to optimize the loss function, and outputs a satisfactory costume pattern transfer image. The experimental results show that the generated image has a real costume pattern color distribution and a good sense of fabric material.

Key words: costume pattern sketch, sketch colorization, style transfer, conditional generative adversarial network, convolutional neural network

中图分类号:

TP391.9

蔡兴泉, 李治均, 奚梦瑶, 孙海燕. 基于神经网络的手绘服饰图纹上色及风格迁移[J]. 系统仿真学报, 2023, 35(3): 604-615.

Xingquan Cai, Zhijun Li, Mengyao Xi, Haiyan Sun. Costume Pattern Sketch Colorization and Style Transfer Based on Neural Network[J]. Journal of System Simulation, 2023, 35(3): 604-615.

图/表 19

图1

表1

表2

图2

图3

图4

图5

图6

图7

图8

图9

图10

图11

图12

图13

图14

图15

图16

表3

参考文献 26

1	梁培俊, 刘怡俊. 基于条件生成对抗网络的漫画手绘图上色方法[J]. 计算机应用研究, 2019, 36(1): 308-311.
	Liang Peijun, Liu Yijun. Caricature Hand Drawing Coloring Method Based on Conditional Generative Con-frontation Network[J]. Application Research of Computers, 2019, 36(1): 308-311.
2	李媛, 陈昭炯, 叶东毅. 注意力引导的交互式工笔花卉上色算法[J]. 模式识别与人工智能, 2020, 33(7): 575-587.
	Li Yuan, Chen Zhaojiong, Ye Dongyi. Attention-Guided Interactive Meticulous Flower Coloring Algorithm[J]. Pattern Recognition and Artificial Intelligence, 2020, 33(7): 575-587.
3	Cheng Z, Yang Q, Sheng B. Deep Colorization[C]// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Santiago, Chile: IEEE Press, 2015: 415-423.
4	Yoo S, Bahng H, Chung S, et al. Coloring with Limited Data: Few-Shot Colorization Via Memory-Augmented Networks[C]// IEEE Conference on Computer Vision and Pattern Recognition. Los Angeles, USA: IEEE Press, 2019: 11283-11292.
5	Sangkloy P, Lu J, Fang C, et al. Scribbler: Controlling Deep Image Synthesis with Sketch and Color[C]// IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE Press, 2016: 5400-5409.
6	Güclütürk Y. Convolutional sketch inversion[C]// ECCV Workshops Computer Vision. Amsterdam, Netherlands: ECCV Press, 2016: 810-824.
7	Ci Y, Ma X, Wang Z, et al. User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks[C]// 26th ACM International Conference on Multimedia. Seoul, Korea: ACM Press, 2018: 1536-1544.
8	Mirza M, Osindero S. Conditional Generative Adversarial Nets[J]. Computer Science(S0098-3004), 2014, 48(4): 2672-2680.
9	Kron A, Nestmann F, SchlTer I, et al. Operational Flood Management under Large-Scale Extreme Conditions, Using the Example of the Middle Elbe[J]. Natural Hazards and Earth System Sciences (S1561-8633), 2010, 10(6): 1171-1181.
10	常佳, 王玉德, 吉燕妮. 基于改进的U-Net生成对抗网络的图像翻译算法[J]. 通信技术, 2020, 53(2): 327-334.
	Chang Jia, Wang Yude, Ji Yanni. Image Translation Algorithm Based on Improved U-Net Generative Adversarial Network[J]. Communication Technology, 2020, 53(2): 327-334.
11	Lee K H. A New Image Completion Method Inserting an Image Generated by Sketch Image[J]. International Journal of Innovative Technology and Exploring Engineering (S2278-3075), 2021, 10(4): 14-18.
12	Xie Minshan, Li Chengze, Liu Xueting, et al. Manga Filling Style Conversion with Screentone Variational Autoencoder[J]. ACM Transactions on Graphics (S0730-0301), 2020, 39(6): 22601-22615.
13	Frans K. Unsupervised Adversarial Defense through Tandem Deep Image Priors[J]. Electronics (S1087-4423), 2020, 9(11): 1957.
14	Aarti E. Generative Adversarial Networks and Their Variants[J]. Information Science (S0020-0255), 2021, 16(7): 59-80.
15	Zhang L, Li C, Wong T T, et al. Two-stage Sketch Colorization[J]. ACM Transactions on Graphics (S0730-0301), 2018, 37(6): 26101-26114.
16	陈淑環, 韦玉科, 徐乐. 基于深度学习的图像风格迁移研究综述[J]. 计算机应用研究, 2019, 36(8): 2250-2255.
	Chen Shuhuan, Wei Yuke, Xu Le. A Review of Research on Image Style Transfer Based on Deep Learning[J]. Application Research of Computers, 2019, 36(8): 2250-2255.
17	丁晓龙. 基于深度学习的图像风格迁移技术的前沿进展[J]. 电子制作, 2018, 18(1): 86-87.
	Ding Xiaolong. The Cutting-Edge Development of Image Style Transfer Technology Based on Deep Learning[J]. Electronic production, 2018, 18(1): 86-87.
18	张海嵩, 尹小勤. 实时绘制3D中国画效果[J]. 计算机辅助设计与图形学学报, 2004, 11(2): 1485-1489.
	Zhang Haisong, Yin Xiaoqin. Real-Time Rendering of 3D Chinese Painting Effects[J]. Journal of Computer Aided Design and Graphics, 2004, 11(2): 1485-1489.
19	苗序娟, 余浩, 王露. 图像风格迁移技术概况及研究现状[J]. 现代计算机, 2021, 6(2): 67-72.
	Miao Xujuan, Yu Hao, Wang Lu. General Situation and Research Status of Image Style Transfer Technology[J]. Modern Computer, 2021, 6(2): 67-72.
20	张菊, 郭永峰. 深度学习研究综述[J]. 教学研究, 2021, 44(3): 6-11.
	Zhang Ju, Guo Yongfeng. A Survey of Deep Learning Research[J]. Teaching Research, 2021, 44(3): 6-11.
21	Gatys L A, Ecker A S, Bethge M. Image Style Transfer Using Convolutional Neural Networks[C]// IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE Press, 2016: 2414-2423.
22	Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition[J]. Computer Science (S0098-3004), 2014, 15(4): 8570-8594.
23	Johnson J, Alahi A, Fei L. Perceptual Losses for Real-Time Style Transfer and Super-Resolution[C]// European Conference on Computer Vision. Amsterdam, Netherlands: ECCV Press, 2016: 694-711.
24	Huang X, Belongie S. Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization[C]// IEEE International Conference on Computer Vision. Venice, Italy: IEEE Press, 2017: 1510-1519.
25	Tian Q C, Schmidt M. Fast Patch-Based Style Transfer of Arbitrary Style[C]// IEEE Conference on Computer Vision and Pattern Recognition. Los Angeles, USA: IEEE Press, 2019: 5873-5881.
26	Zhu J, Park T, Isola P, et al. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks[C]// IEEE International Conference on Computer Vision. Venice, Italy: IEEE Press, 2017: 2242-225.

层数	结构	卷积核个数	输出尺寸
1	Conv	64	128×128
2	LReLU, Conv	128	64×64
3	LReLU, Conv	256	32×32
4~8	LReLU, Conv	512	16×16
9~12	ReLU, DeConv	512	32×32
13	ReLU, DeConv	256	64×64
14	ReLU, DeConv	128	128×128
15	ReLU, DeConv	64	256×256
16	ReLU, DeConv, Tanh	3	256×256

层数	结构	卷积核个数	输出尺寸
1	Conv	64	128×128
2	LReLU, Conv	128	64×64
3	LReLU, Conv	256	32×32
4	LReLU, Conv	512	32×32
5	ReLU, Conv, Sigmoid	1	32×32

图像	SSIM	PSNR
PanintsChainer	0.64	21.71
Style2Paint	0.65	22.72
CycleGan	0.61	22.87
本文方法	0.71	23.37

[1]	杨丰玮, 陈鹏, 郗凯, 蒲华林, 刘雪垠. 融合空间信息的运动想象脑电在线分类方法[J]. 系统仿真学报, 2023, 35(2): 254-267.
[2]	张立峰, 王会忍. 基于卷积神经网络及有限元仿真的电容层析成像图像重建[J]. 系统仿真学报, 2022, 34(4): 712-718.
[3]	敖邦乾, 杨莎, 令狐金卿, 叶振环. 基于级联神经网络疲劳驾驶检测系统设计[J]. 系统仿真学报, 2022, 34(2): 323-333.
[4]	王亚茹, 杨凯, 翟永杰, 郭聪彬, 赵文清, 苏杰. 基于人工图像数据扩充的输电线路绝缘子识别[J]. 系统仿真学报, 2022, 34(11): 2337-2347.
[5]	徐佳乐, 张海东, 赵东海, 倪晚成. 基于卷积神经网络的陆战兵棋战术机动策略学习[J]. 系统仿真学报, 2022, 34(10): 2181-2193.
[6]	郭业才, 王庆伟. 基于截断迁移与并行残差网络的调制识别算法[J]. 系统仿真学报, 2022, 34(09): 2009-2018.
[7]	赵瑛, 陆耀, 张健, 梁启弟, 龙炜. 基于深度神经网络的多视角人体动作识别[J]. 系统仿真学报, 2021, 33(5): 1019-1030.
[8]	仝卫国, 庞雪纯, 朱赓宏. 基于卷积神经网络的气液两相流流型识别方法[J]. 系统仿真学报, 2021, 33(4): 883-891.
[9]	马榕, 陈秋瑞, 张晗, 梅铮, 王锐, 魏伟. 基于单目深度估计的低功耗视觉里程计[J]. 系统仿真学报, 2021, 33(12): 3001-3011.
[10]	黄欣, 方钰, 顾梦丹. 基于卷积神经网络的X线胸片疾病分类研究[J]. 系统仿真学报, 2020, 32(6): 1188-1194.
[11]	樊名鲁, 王艳, 纪志成. 基于特征生成的轴承不均衡数据故障诊断[J]. 系统仿真学报, 2020, 32(12): 2438-2448.
[12]	王红军, 黎邹邹, 邹湘军. 基于Adaboost与CNN的木材表面缺陷检测[J]. 系统仿真学报, 2019, 31(8): 1636-1645.
[13]	卢裕秋, 孙金玉, 马世伟. 基于深度卷积神经网络的运动目标检测方法[J]. 系统仿真学报, 2019, 31(11): 2275-2280.
[14]	石祥滨, 钟健, 刘翠微, 刘芳, 张德园. 小样本高光谱遥感图像深度学习方法[J]. 系统仿真学报, 2018, 30(7): 2744-2752.
[15]	夏旻, 申茂阳, 王舰锋, 王阳光. 基于卷积神经网络的卫星云图云量计算[J]. 系统仿真学报, 2018, 30(5): 1623-1630.