Research on Infrared and Visible Light Fusion Method Based on ResNet-50 and Laplacian Filtering

doi:10.16182/j.issn1004731x.joss.25-FZ0696

Abstract

Abstract:

In order to solve the problem that existing infrared and visible light image fusion techniques often suffer from artifacts caused by insufficient contrast, spectral distortion, and high computational complexity, a fusion framework based on ResNet-50 and Laplacian filtering was proposed. ResNet-50 was used to extract shallow and deep features, followed by multi-scale feature fusion. Laplacian filtering was applied to optimize feature information, and an automatic discriminator was introduced to further improve the fusion effect. Simulation results show that, compared with comparison algorithms, the proposed method achieves an average increase of 2.71% and 2.16% in information entropy, 5.98% and 7.86% in visual information fidelity, and 12.57% and 14.63% in wavelet-based feature mutual information on the TNO and VIFB datasets, respectively, which proves its effectiveness in improving image fusion quality.

Key words: image fusion, ResNet-50, Laplacian filtering, deep learning, automatic discriminator

CLC Number:

TP571

Wang Xiao, Li Xiangyang, Liang Feng, Zhang Zhili. Research on Infrared and Visible Light Fusion Method Based on ResNet-50 and Laplacian Filtering[J]. Journal of System Simulation, 2025, 37(12): 3202-3211.

Figures/Tables 15

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Table 1

Fig. 5

Fig. 6

Table 2

Fig. 7

Fig. 8

Table 3

Fig. 9

Fig. 10

Table 4

Table 5

References 25

[1]	何洋. 多频段图像融合方法研究及其在无源成像中的应用[D]. 成都: 电子科技大学, 2021.
	He Yang. Research on Multi-band Image Fusion Method and Its Application in Passive Imaging System[D]. Chengdu: University of Electronic Science and Technology of China, 2021.
[2]	李文. 基于非下采样剪切波变换的红外与可见光图像融合研究[D]. 赣州: 江西理工大学, 2022.
	Li Wen. Research on Infrared and Visible Image Fusion Based on Non-subsampled Shearlet Transform[D]. Ganzhou: Jiangxi University of Science and Technology, 2022.
[3]	叶发杰. 基于卷积神经网络的遥感图像融合算法[D]. 长春: 吉林大学, 2019.
	Ye Fajie. Remote Sensing Image Fusion Algorithm Based on Convolutional Neural Network[D]. Changchun: Jilin University, 2019.
[4]	赵海霞, 常霞, 纪峰. 联合信息增强与生成对抗网络的红外与可见光图像融合[J]. 激光杂志, 2023, 44(10): 116-121.
	Zhao Haixia, Chang Xia, Ji Feng. Infrared and Visible Image Fusion of Joint Information Enhancement and Generative Adversarial Networks[J]. Laser Journal, 2023, 44(10): 116-121.
[5]	Rong Chuanzhen, Liu Gaohang, Ping Zhuolin, et al. Fusion of Infrared and Visible Images Based on Infrared Object Extraction[J]. Chinese Journal of Electronics, 2021, 30(2): 339-348.
[6]	Yang Dongsheng, Hu Shaohai, Liu Shuaiqi, et al. Multi-focus Image Fusion Based on Block Matching in 3D Transform Domain[J]. Journal of Systems Engineering and Electronics, 2018, 29(2): 415-428.
[7]	李恒. 基于深度学习的多聚焦图像融合算法研究[D]. 兰州: 兰州交通大学, 2020.
	Li Heng. The Research on Multi-focus Image Fusion Algorithm Based on Deep Learning[D]. Lanzhou: Lanzhou Jiaotong University, 2020.
[8]	李永萍, 杨艳春, 党建武, 等. 基于变换域VGGNet19的红外与可见光图像融合[J]. 红外技术, 2022, 44(12): 1293-1300.
	Li Yongping, Yang Yanchun, Dang Jianwu, et al. Infrared and Visible Image Fusion Based on Transform Domain VGGNet19[J]. Infrared Technology, 2022, 44(12): 1293-1300.
[9]	黄珍, 潘颖, 苑毅. 基于改进神经网络的图像融合技术[J]. 机电工程技术, 2021, 50(7): 161-163.
	Huang Zhen, Pan Ying, Yuan Yi. Image Fusion Technology Based on Improved Neural Network[J]. Mechanical & Electrical Engineering Technology, 2021, 50(7): 161-163.
[10]	郭梦婷. 易腐垃圾异常物检测与乱扔垃圾行为识别算法研究[D]. 杭州: 中国计量大学, 2021.
	Guo Mengting. Research on the Algorithm for Detection of Abnormal Objects of Perishable Garbage and Identification of Littering Behavior[D]. Hangzhou: China Jiliang University, 2021.
[11]	李国梁. 基于深度学习的红外与可见光图像融合算法研究[D]. 北京: 北京交通大学, 2021.
	Li Guoliang. Research on Infrared and Visible Image Fusion Algorithm Based on Deep Learning[D]. Beijing: Beijing Jiaotong University, 2021.
[12]	魏亚南. 基于多尺度统计建模和深度学习的红外与可见光图像融合方法研究[D]. 济南: 山东建筑大学, 2022.
[13]	Li Shutao, Renwei Dian, Liu Haibo. Learning the External and Internal Priors for Multispectral and Hyperspectral Image Fusion[J]. Science China Information Sciences, 2023, 66(4): 140303.
[14]	Wang Jian, Qin Chunxia, Zhang Xiufei, et al. A Multi-source Image Fusion Algorithm Based on Gradient Regularized Convolution Sparse Representation[J]. Journal of Systems Engineering and Electronics, 2020, 31(3): 447-459.
[15]	Cai Runlin, Liu Chenying, Li Jun. Efficient Phase-induced Gabor Cube Selection and Weighted Fusion for Hyperspectral Image Classification[J]. Science China Technological Sciences, 2022, 65(4): 778-792.
[16]	杨艳春, 李永萍, 党建武, 等. 基于快速交替引导滤波和CNN的红外与可见光图像融合[J]. 光学精密工程, 2023, 31(10): 1548-1562.
	Yang Yanchun, Li Yongping, Dang Jianwu, et al. Infrared and Visible Image Fusion Based on Fast Alternating Guided Filtering and CNN[J]. Optics and Precision Engineering, 2023, 31(10): 1548-1562.
[17]	陈雅琼. 基于深度学习的BGA焊球空洞缺陷检测研究[D]. 桂林: 桂林电子科技大学, 2021.
	Chen Yaqiong. Research on Void Defect Detection of BGA Solder Ball Based on Deep Learning[D]. Guilin: Guilin University of Electronic Technology, 2021.
[18]	王佳琪. 基于多尺度分析的红外与可见光图像融合的研究[D]. 秦皇岛: 燕山大学, 2012.
	Wang Jiaqi. Research for the Fusion of Infrared and Visible Image Based on Multiscale Analysis[D]. Qinhuangdao: Yanshan University, 2012.
[19]	岑悦亮. 任意分辨率红外与可见光图像融合算法研究[D]. 昆明: 昆明理工大学, 2021.
[20]	邸敬, 梁婵, 任莉, 等. 基于多尺度对比度增强和跨维度交互注意力机制的红外与可见光图像融合[J]. 红外技术, 2024, 46(7): 754-764.
	Di Jing, Liang Chan, Ren Li, et al. Infrared and Visible Image Fusion Based on Multi-scale Contrast Enhancement and Cross-dimensional Interactive Attention Mechanism[J]. Infrared Technology, 2024, 46(7): 754-764.
[21]	陈海秀, 房威志, 陆康, 等. 基于多层卷积的红外与可见光图像融合算法[J]. 电光与控制, 2024, 31(9): 12-17, 44.
	Chen Haixiu, Fang Weizhi, Lu Kang, et al. Infrared and Visible Image Fusion Based on Multi-layer Convolution[J]. Electronics Optics & Control, 2024, 31(9): 12-17, 44.
[22]	邵大光, 邵现振, 刘鹏, 等. 基于ResNet50与卷积稀疏表达的红外与可见光图像融合算法[J]. 计算机应用与软件, 2024, 41(5): 189-196.
	Shao Daguang, Shao Xianzhen, Liu Peng, et al. The Infrared and Visible Image Fusion Algorithm Based on Resnet50 and Convolution Sparse Representation[J]. Computer Applications and Software, 2024, 41(5): 189-196.
[23]	He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep Residual Learning for Image Recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2016: 770-778.
[24]	李晓玲, 陈后金, 李艳凤, 等. 多重关系感知的红外与可见光图像融合网络[J]. 电子与信息学报, 2024, 46(5): 2217-2227.
	Li Xiaoling, Chen Houjin, Li Yanfeng, et al. Infrared and Visible Image Fusion Network with Multi-relation Perception[J]. Journal of Electronics & Information Technology, 2024, 46(5): 2217-2227.
[25]	卢琳. 基于深度学习的红外与可见光图像融合算法研究[D]. 石家庄: 石家庄铁道大学, 2022.
	Lu Lin. Research on Infrared and Visible Image Fusion Algorithm Based on Deep Learning[D]. Shijiazhuang: Shijiazhuang Tiedao University, 2022.

是否采用初始权重图		信息熵	标准差	空间频率	小波特征互信息	视觉保真度	平均梯度
TNO	否	6.632 7	7.356 6	0.047 5	0.456 2	0.674 3	5.867 1
	是	6.727 4	8.473 5	0.052 3	0.502 9	0.702 8	6.173 9
	提升率/%	1.43	15.18	10.11	10.24	4.23	5.23
VIFB	否	6.925 3	7.537 1	0.046 4	0.453 4	0.685 3	5.429 4
	是	7.129 5	8.068 2	0.049 1	0.500 2	0.770 4	6.058 6
	提升率/%	2.95	7.05	5.82	10.32	12.42	11.59

有无采用Laplacian		信息熵	标准差	空间频率	小波特征互信息	视觉保真度	平均梯度
TNO	无	6.623 5	7.958 9	0.042 3	0.427 6	0.651 3	6.271 7
	有	6.913 4	8.214 4	0.063 4	0.573 3	0.758 3	6.578 9
	提升率/%	4.38	3.21	49.88	34.07	16.43	4.90
VIFB	无	6.237 5	9.221 1	0.029 6	0.461 8	0.550 7	5.006 6
	有	6.830 5	9.968 8	0.042 0	0.565 7	0.659 1	6.801 0
	提升率/%	9.51	8.11	41.89	22.50	19.68	35.84

有无自动判别器		信息熵	标准差	空间频率	小波特征互信息	视觉保真度	平均梯度
TNO	无	7.144 8	8.588 4	0.056 3	0.396 0	0.659 1	6.471 7
	有	8.917 6	9.968 8	0.072 0	0.591 4	0.655 7	6.809 0
	提升率/%	24.81	16.07	27.89	49.34	-0.52	5.21
VIFB	无	6.485 3	8.688 3	0.048 7	0.434 4	0.625 5	6.122 7
	有	7.186 3	9.805 6	0.094 1	0.459 5	1.088 6	7.191 4
	提升率/%	10.81	12.86	93.22	5.78	74.04	17.45

实验	方法	信息熵	标准差	空间频率	小波特征互信息	视觉保真度	平均梯度
第 1 组	CBF	5.905 1	8.482 3	0.030 0	0.387 3	0.607 7	3.175 2
	JSR	6.235 0	8.844 6	0.043 2	0.398 9	0.722 1	4.239 3
	GTF	7.235 3	8.747 8	0.033 5	0.428 7	0.753 3	4.522 1
	ConvSR	7.832 6	9.724 7	0.044 8	0.572 1	0.782 2	4.964 7
	DeepFuse	8.200 3	10.442 9	0.063 3	0.638 8	0.843 9	5.338 9
	Ours	8.422 6	9.852 6	0.088 2	0.718 3	0.894 4	6.329 5
第 2 组	CBF	6.167 5	8.366 2	0.057 7	0.345 3	0.623 5	3.355 2
	JSR	6.449 8	8.275 3	0.046 6	0.396 5	0.762 9	4.498 2
	GTF	7.238 7	8.028 3	0.059 2	0.387 2	0.773 4	4.298 5
	ConvSR	8.092 8	9.337 4	0.062 2	0.446 3	0.788 4	5.229 5
	DeepFuse	9.287 3	11.128 5	0.075 3	0.652 8	0.829 4	6.338 9
	Ours	9.488 2	11.448 2	0.062 9	0.748 3	0.894 6	7.025 3

实验	信息熵	标准差	空间频率	小波特征互信息	视觉保真度	平均梯度
平均提升	+2.44	-1.39	+22.87	+13.54	+6.92	+14.69
第1组	+2.71	-5.65	+39.34	+12.57	+5.98	+18.55
第2组	+2.16	+2.87	-16.47	+14.63	+7.86	+10.83