An Unsupervised Deep Neural Network for Image Fusion

doi:10.16182/j.issn1004731x.joss.20-1062

Abstract

Abstract:

Due to the low dynamic range of camera, can not be expressed in the different region of the high dynamic scene a single-exposure image. An unsupervised depth neural network is constructed to fuse the multi-exposure images into a high dynamic image. Based on the VGG-Net, encoding and decoding sub-networks are designed. Guided by the structural similarity of the images before and after fusion, a loss function suitable for image fusion is designed by introducing the weight factors based on the local image information, and the valid information of the different input images is given consideration. Compared with the other methods, the subjective visual experience and objective quantitative indicators of the fused images are improved significantly.

Key words: pattern recognition, high dynamic scene, image fusion, unsupervised deep network, loss function

CLC Number:

TP391.4

Peipei Zhou, Xinglin Hou. An Unsupervised Deep Neural Network for Image Fusion[J]. Journal of System Simulation, 2022, 34(6): 1267-1274.

Figures/Tables 5

Fig. 1

Fig. 2

Fig. 3

Table 1

Fig. 4

References 27

1	Ma K, Li H, Yong H, et al. Robust Multi-Exposure Image Fusion: a Structural Patch Decomposition Approach[J]. IEEE Transactions on Image Processing(S1057-7149), 2017, 26(5): 2519-2532.
2	魏利胜, 张平改. 基于分层模型的图像快速融合方法研究[J]. 系统仿真学报, 2016, 28(6): 1372-1379.
	Wei Lisheng, Zhang Pingai. Fast Fusion Method for Multi-focus Image Based on Hierarchical Model[J]. Journal of System Simulation, 2016, 28(6): 1372-1379.
3	马夏一, 范方晴, 卢陶然, 等. 基于图像块分解的多曝光图像融合去鬼影算法[J]. 光学学报, 2019, 39(9): 132-140.
	Ma Xiayi, Fan Fangqing, Lu Taoran, et al. Multi-Exposure Image Fusion De-Ghosting Algorithm Based on Image Block Decomposition[J]. Acta Optica Sinica, 2019, 39(9): 132-140.
4	Goshtasby A A. Fusion of Multi-Exposure Images[J]. Image and Vision Computing(S0262-8856), 2005, 23(6): 611-618.
5	Ma Keda, Zhou Wang. Multi-Exposure Image Fusion: A Patch-Wise Approach[C]//2015 IEEE International Conference on Image Processing (ICIP). Quebec: IEEE, 2015: 1717-1721.
6	Burt P J, Kolczynski R J. Enhanced Image Capture Through Fusion[C]//1993 (4th) International Conference on Computer Vision. IEEE, 1993: 173-182.
7	Raman S, Chaudhuri S. Bilateral Filter Based Compositing for Variable Exposure Photography[C]//Eurographics. The Eurographics Association, 2009: 1-4.
8	Shen R, Cheng I, Shi J, et al. Generalized Random Walks for Fusion of Multi-Exposure Images[J]. IEEE Transactions on Image Processing(S1057-7149), 2011, 20(12): 3634-3646.
9	Li S, Kang X, Hu J. Image Fusion with Guided Filtering[J]. IEEE Transactions on Image processing(S1057-7149), 2013, 22(7): 2864-2875.
10	Hou X, Luo H, Qi F, et al. Guided Filter-Based Fusion Method for Multiexposure Images[J]. Optical Engineering(S0091-3286), 2016, 55(11): 113101.
11	Shen J, Zhao Y, Yan S, et al. Exposure Fusion Using Boosting Laplacian Pyramid[J]. IEEE Transactions on Cybernetics(S2168-2267), 2014, 44(9): 1579-1590.
12	Wang L, Xiong Y, Wang Z, et al. Temporal Segment Networks for Action Recognition in Videos[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence(S0182-8828), 2018, 41(11): 2740-2755.
13	Lin T Y, Goyal P, Girshick R, et al. Focal Loss for Dense Object Detection[C]//IEEE International Conference on Computer Vision. IEEE, ICCV, 2017: 2980-2988.
14	Fu J, Liu J, Tian H, et al. Dual Attention Network for Scene Segmentation[C]// IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2019: 3146-3154.
15	Divakar N, Venkatesh Babu R. Image Denoising Via CNNs: an Adversarial Approach[C]// IEEE Conference on Computer Vision and Pattern Recognition Workshops. IEEE, 2017: 80-87.
16	Prabhakar K R, Babu R V. Ghosting-Free Multi-Exposure Image Fusion in Gradient Domain[C]// 2016 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2016: 1766-1770.
17	Eilertsen G, Kronander J, Denes G, et al. HDR Image Reconstruction from a Single Exposure Using Deep CNNs[J]. Acm Transactions on Graphics(S0730-0301), 2017, 36(6): 1-15.
18	Prabhakar K R, Srikar V S, Babu R V. DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs[C]// 2017 IEEE International Conference on Computer Vision. IEEE Computer Society, 2017: 4714-4722.
19	Li H, Wu X J. DenseFuse: A Fusion Approach to Infrared and Visible Images[J]. IEEE Transactions on Image Processing(S1057-7149), 2018, 28(5): 2614-2623.
20	Ma K, Duanmu Z, Zhu H, et al. Deep Guided Learning for Fast Multi-Exposure Image Fusion[J]. IEEE Transactions on Image Processing(S1057-7149), 2019, 29(1): 2808-2819.
21	Ioffe S, Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[C]//International Conference on Machine Learning. PMLR, 2015: 448-456.
22	Nair V, Hinton G E. Rectified Linear Units Improve Restricted Boltzmann Machines Vinod Nair[C]// International Conference on International Conference on Machine Learning. Omnipress, 2010: 807-814.
23	Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition[J/OL]. [2020-12-14]. .
24	Wang Z, Bovik A C, Sheikh H R, et al. Image Quality Assessment: from Error Visibility to Structural Similarity[J]. IEEE Transactions on Image Processing(S1057-7149), 2004, 13(4): 600-612.
25	Zhang J, Shao J, Chen J, et al. PFNet: an Unsupervised Deep Network for Polarization Image Fusion[J]. Optics Letters(S0146-9592), 2020, 45(6): 1507-1510.
26	周培培. 无监督深度多曝光图像融合网络[EB/OL]. [2020-12-14]. .
	Zhou Peipei. Unsupervised Deep Multi-Exposure Image Fusion Network [EB/OL]. [2020-12-14]. .
27	Ma K, Zeng K, Wang Z. Perceptual Quality Assessment for Multi-Exposure Image Fusion[J]. IEEE Transactions on Image Processing(S1057-7149), 2015, 24(11): 3345-3356.

HDR场景	Shen11	Li13	Ma15	Hou16	DF19-add	DF19-L1	MEF-Net20	本文方法
Balloons	0.932 3	0.908 2	0.987 9	0.932 7	0.947 8	0.963 2	0.978 3	0.989 4
Belgium house	0.926 6	0.884 9	0.978 6	0.918 5	0.937 9	0.958 4	0.974 3	0.989 0
Chinese garden	0.922 9	0.917 9	0.993 5	0.923 5	0.941 3	0.985 3	0.987 1	0.993 3
House	0.925 2	0.830 6	0.964 2	0.916 5	0.901 3	0.765 7	0.935 0	0.966 5
Landscape	0.899 8	0.905 2	0.994 8	0.921 9	0.984 2	0.990 8	0.981 6	0.989 4
Lighthouse	0.912 8	0.914 9	0.984 3	0.933 4	0.952 5	0.953 3	0.969 2	0.986 3
Madison capitol	0.893 5	0.800 5	0.981 2	0.871 2	0.914 0	0.906 6	0.958 1	0.984 1
Office	0.904 4	0.725 6	0.983 9	0.893 8	0.948 6	0.912 3	0.955 5	0.983 9
Room	0.920 6	0.853 3	0.978 2	0.915 1	0.938 4	0.939 6	0.967 8	0.987 1
Tower	0.912 8	0.858 6	0.986 0	0.910 1	0.915 1	0.921 9	0.959 1	0.988 5
Venice	0.903 9	0.834 9	0.987 9	0.904 6	0.935 9	0.961 4	0.965 7	0.982 7
Average	0.914 1	0.857 7	0.983 7	0.912 8	0.937 9	0.932 6	0.966 5	0.985 5

[1]	Wang Xiao, Li Xiangyang, Liang Feng, Zhang Zhili. Research on Infrared and Visible Light Fusion Method Based on ResNet-50 and Laplacian Filtering [J]. Journal of System Simulation, 2025, 37(12): 3202-3211.
[2]	Shi Lanxi, Yan Wenxu, Ni Hongyu, Zhao Feng. Research on Dynamic Scene SLAM Based on Improved Object Detection [J]. Journal of System Simulation, 2024, 36(4): 1028-1042.
[3]	Su Tong, Wang Ying, Deng Qiyang, Li Zhaobin. Improved Foggy Pedestrian and Vehicle Detection Algorithm Based on YOLOv5 [J]. Journal of System Simulation, 2024, 36(10): 2413-2422.
[4]	Wei Dong, Liu Huan, Zhang Xiaohan, Li Changkai, Sun Tianyi, Zhang Ziyou. Multi-view Depth Estimation Based on Adaptive Space Feature Enhancement [J]. Journal of System Simulation, 2024, 36(1): 110-119.
[5]	Tong Weiguo, Pang Xuechun, Zhu Genghong. Gas-liquid Two-phase Flow Pattern Recognition Method Based on Convolutional Neural Network [J]. Journal of System Simulation, 2021, 33(4): 883-891.
[6]	Cai Xingquan, Tu Yuxin, Ge Yakun, Yang Zhe. Real-time Leaf Recognition Method Based on CNN Network and Multi-task Loss Function [J]. Journal of System Simulation, 2020, 32(7): 1279-1286.
[7]	Ji Weixi, Du Meng, Peng Wei, Xu Jie. Research on Gear Appearance Defect Recognition Based on Improved Faster R-CNN [J]. Journal of System Simulation, 2019, 31(11): 2198-2205.
[8]	Yang Zhenshan, Yue Wenjiao. Elevator Traffic Pattern Recognition with FCM Clustering Based Fuzzy Neural Network [J]. Journal of System Simulation, 2018, 30(4): 1433-1439.
[9]	Jin Weidong, Hu Yanhua, Tang Peng, Li Wei. Detailed Enhancement of Forward Vehicle Video Images Based on Structured Forest [J]. Journal of System Simulation, 2018, 30(12): 4602-4610.
[10]	Li Haiyang, Cao Weiguo, Li Shirui, Tao Kelu, Li Hua. Progressive Image Denoising Algorithm [J]. Journal of System Simulation, 2017, 29(2): 282-294.
[11]	Xie Shiqin, Zhao Tianzhong, Wang Wei, Shi Jingjing. Study on Fusion Algorithms of GF-2 Satellite Image [J]. Journal of System Simulation, 2017, 29(11): 2742-2747.
[12]	Luo Xiao, Li Weimin. Novel Distance Measure Between Intuitionistic Fuzzy Sets [J]. Journal of System Simulation, 2017, 29(10): 2360-2372.
[13]	Shen Weihua, Wang Shengbo, Pan Zhigeng. New 2D Flow Visualization Method Based on Mutual Information and Image Fusion [J]. Journal of System Simulation, 2015, 27(8): 1796-1800.
[14]	Wen Kaifeng, Li Bingjian. Title Medical Image Fusion Based on Multi-scale Coefficient Decomposition Framework [J]. Journal of System Simulation, 2015, 27(10): 2615-2621.