Murals Super-resolution Reconstruction with the Stable Enhanced Generative Adversarial Network

doi:10.16182/j.issn1004731x.joss.20-0989

Abstract

Abstract:

Aiming at the problems of low resolution and unclear texture details of ancient murals, which led to insufficient viewing of murals and low research value, a stable enhanced super-resolution generative adversarial networks (SESRGAN) reconstruction algorithm is proposed. Based on the generative adversarial network, the generative network uses dense residual blocks to extract mural features, and uses the visual geometry group (VGG) network as the basic framework of the discriminating network to determine the authenticity of the input mural, and introduces perception loss, content loss and penalty loss to jointly optimize the model. Experimental results show that, compared with other related super-resolution algorithms, the peak signal-to-noise ratio (PSNR) is improved by 0.4~2.62 dB on average, the structural similarity is improved by 0.013~0.027, and the subjective perception evaluation is also improved.

Key words: ancient mural, super-resolution reconstruction, generation adversarial network, dense residual block, penalty loss

CLC Number:

TP391.47

Jianfang Cao, Yiming Jia, Minmin Yan, Xiaodong Tian. Murals Super-resolution Reconstruction with the Stable Enhanced Generative Adversarial Network[J]. Journal of System Simulation, 2022, 34(5): 1076-1089.

Figures/Tables 20

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Table 1

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Table 2

Fig. 11

Fig. 12

Fig. 13

Table 3

Fig. 14

Table 4

Fig. 15

Fig. 16

References 20

[1]	师晓波. 毁损古墓葬壁画虚拟修复技术研究[D]. 太原: 中北大学, 2018.
	Shi Xiaobo. Research on Virtual Repair Technology of Damaged Ancient Tomb Murals[D]. Taiyuan: North University of China, 2018.
[2]	Wang H. Inpainting of Potala Palace Murals Based on Sparse Representation[C]//International Conference on Biomedical Engineering and Informatics, USA: IEEE, 2015: 737-741
[3]	曹建芳, 张自邦, 赵爱迪. 增强一致性生成对抗网络在壁画修复上的应用[J]. 计算机辅助设计与图形学学报, 2020, 32(8): 1315-1323.
	Cao Jianfang, Zhang Zibang, Zhao Aidi. The Application of Enhanced Consistency Generative Adversarial Network in Mural Repair[J]. Journal of Computer Aided Design and Graphics, 2020, 32(8): 1315-1323.
[4]	马强. 基于稀疏表示的单幅彩色图像超分辨率重建研究[D]. 兰州: 兰州理工大学, 2019.
	Ma Qiang. Research on Super-resolution Reconstruction of a Single Color Image Based on Sparse Representation[D]. Lanzhou: Lanzhou University of Technology, 2019.
[5]	徐志刚, 闫娟娟, 朱红蕾. 基于多尺度残差注意力网络的壁画图像超分辨率重建算法[J]. 激光与光电子学进展, 2020, 57(16): 152-159.
	Xu Zhigang, Yan Juanjuan, Zhu Honglei. Super-resolution Reconstruction Algorithm of Mural Image Based on Multi-Scale Residual Attention Network[J]. Progress in Laser and Optoelectronics, 2020, 57(16): 152-159.
[6]	张洋. 基于双线性插值法的图像缩放算法的设计与实现[J]. 电子设计工程, 2016, 24(3): 169-170, 175.
	Zhang Yang. Design and Implementation of Image Scaling Algorithm Based on Bilinear Interpolation[J]. Electronic Design Engineering, 2016, 24(3): 169-170, 175.
[7]	练秋生, 张伟. 基于图像块分类稀疏表示的超分辨率重构算法[J]. 电子学报, 2012, 40(5): 920-925.
	Lian Qiusheng, Zhang Wei. Super-resolution Reconstruction Algorithm Based on Sparse Representation of Image Block Classification[J]. Electronic Journal, 2012, 40(5): 920-925.
[8]	钟宝江, 陆志芳. 图像插值技术综述[J]. 数据采集与处理, 2016, 31(6): 1083-1096.
	Zhong Baojiang, Lu Zhifang. Overview of Image Interpolation Technology[J]. Data Collection and Processing, 2016, 31(6): 1083-1096.
[9]	宋佳伟, 徐煜明. 基于小波变换和迭代反向投影的超分辨率算法[J]. 计算机技术与发展, 2015, 25(2): 74-77.
	Song Jiawei, Xu Yuming. Super-Resolution Algorithm Based on Wavelet Transform and Iterative Back Projection[J]. Computer Technology and Development, 2015, 25(2): 74-77.
[10]	张砚, 徐昆. 基于外存和凸集投影法的遥感图像超分辨率方法[J]. 清华大学学报(自然科学版), 2010, 50(10): 1743-1746, 1756.
	Zhang Yan, Xu Kun. Remote Sensing Image Super-Resolution Method Based on External Memory and Convex Set Projection Method[J]. Journal of Tsinghua University (Natural Science Edition), 2010, 50(10): 1743-1746, 1756.
[11]	Chang H, Yeung D Y, Xiong Y. Super-Resolution Through Neighbor Embedding[C]//IEEE Conference on Computer Vision and Pattem Recognition. USA: IEEE, 2004: 275-282.
[12]	Yang J, Wrigbt J, Huang T S, et al. Image Super-Resolution via Sparse Representation[J]. IEEE Transactions on Image Processing (S1057-7149). 2010, 19(11): 2861-2873.
[13]	Dong C, Loy C. Image Super-Resolution Using Deep Convolutional Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence (S0612-8828), 2016, 38(2): 295-307.
[14]	Dong C, Chen C L, Tang X. Accelerating the Superresolution Convolutional Neural Network[C]//14th European Conference on Computer Vision. Berlin: Springer International Publishing, 2016: 391-407.
[15]	Tong T, Li G, Liu X, et al. Image Super-Resolution Using Dense Skip Connections[C]//2017 IEEE International Conference on Computer Vision. New York: IEEE Computer Society, 2017: 4809-4817.
[16]	Ledig C, Theis L, Huszar F, et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network[C]//Internaltional Conference on Computer Vision and Pattern Recogintion(CVPR). New York: IEEE Computer Society, 356-379.
[17]	Hui D L, Gu X F. Wasserstein GAN With Quadratic Transport Cost[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul, ROK: IEEE, 2019: 4831-4840.
[18]	Wang X, Yu K, Wu S, et al. Esrgan: Enhanced Super-Resolution Generative Adversarial Networks[C]//European Conference on Computer Vision. Munich, Germany: ECCV, 2018: 63-79.
[19]	Jin Q M, Lin R H. Enhanced Generative Model of Signaling Data Based on WGAN-GP and ACGAN[J]. IEEE Systems Journal (S1932-8184), 2020, 14(3): 3289-3330.
[20]	袁功霖, 侯静, 尹奎英. 基于迁移学习与图像增强的夜间航拍车辆识别方法[J]. 计算机辅助设计与图形学学报, 2019, 31(3): 467-473.
	Yuan Gonglin, Hou Jing, Yin Kuiying. Night Aerial Vehicle Recognition Method Based on Transfer Learning and Image Enhancement[J]. Journal of Computer Aided Design and Graphics, 2019, 31(3): 467-473.

Name	Type	Kernel	Stride	Padding	Output
CONV0_0	CONV	3×3	1×1	1	64
CONV1_0	CONV	4×4	2×2	1	64
CONV1_1	CONV	3×3	1×1	1	64
CONV2_0	CONV	4×4	2×2	1	128
CONV2_1	CONV	3×3	1×1	1	128
CONV3_0	CONV	4×4	2×2	1	256
CONV3_1	CONV	3×3	1×1	1	256
CONV4_0	CONV	4×4	2×2	1	512
CONV4_1	CONV	3×3	1×1	1	512
FC0	FC	-	-	-	100
FC1	FC	-	-	-	1

图像	PSNR	SSIM
a	28.71	0.876
b	32.03	0.782
c	32.03	0.782
d	32.03	0.782

图像	BI	文献[4]	文献[5]	SRGAN	ESRGAN	ESERGAN
a	30.58/0.827	30.78/0.824	30.97/0.831	31.24/0.843	32.06/0.862	32.53/0.859
b	28.01/0.792	28.24/0.799	28.16/0.795	28.39/0.806	28.60/0.809	29.20/0.824
c	22.69/0.741	23.96/0.743	24.84/0.745	25.17/0.749	26.20/0.751	27.03/0.776
d	25.48/0.739	26.54/0.740	27.29/0.751	26.59/0.743	28.42/0.746	28.06/0.764
e	24.41/0.764	25.16/0.766	25.75/0.771	25.81/0.767	26.54/0.774	27.29/0.786
f	23.59/0.731	24.88/0.730	25.43/0.733	25.66/0.734	26.28/0.738	26.39/0.743
Avg	25.79/0.765	26.59/0.767	27.07/0.771	27.14/0.773	28.01/0.779	28.41/0.792

图像	含有BN层	含有SN层
a	7.31	9.54
b	6.47	8.25
c	7.06	7.03
d	5.36	6.49
e	5.45	7.24
f	6.17	8.49