Parallax-tolerant Image Stitching with Geometric Structure Protection for Unmanned Ship Visual Perception

doi:10.16182/j.issn1004731x.joss.24-0310

Abstract

Abstract:

In order to address the artifacts caused by misalignment in maritime image stitch with low texture and large parallax, a geometric structure parallax-tolerant image stitch algorithm based on point-line feature registration and optimal seam fusion is proposed. Line segment features are introduced into the traditional homography transformation based on point features, and potential coplanar local line segments are merged into global line segments to provide accurate alignment conditions for seam line fusion. In the image fusion stage, the energy function of seam cutting method is designed by using the color difference and gradient difference of tanh measure and introducing saliency detection weight, which guides the optimal seam to avoid prominent maritime structures in the image, thus ensuring the continuity of structural edges. The SIFT flow method is used to correct the misalignment pixels on the stitching seam, and the maritime image stitching with accurate geometric structure is realized. Experimental results, based on 20 pairs of diverse scene data, compaed with the baseline method, the proposed algorithm achieves an average 44.6% reduction in seam quality error based on structural similarity, with a maximum reduction of 66.7%. The average reduction in seam quality error based on zero mean normalized cross-correlation is 24.7%, with a maximum reduction of 51.6%. The algorithm can effectively avoid artifacts, thus achieving visually natural stitching results and satisfying the requirements for a wide field of view during unmanned ship navigation.

Key words: unmanned ship visual perception, image stitching, point-line consistence registration, geometric structure protection, optimal seam fusion

CLC Number:

TP391.9

Yang Zhilin, Yin Yong, Zhang Rukai, Jing Qianfeng, Jiang Sen, Zhu Wenfeng. Parallax-tolerant Image Stitching with Geometric Structure Protection for Unmanned Ship Visual Perception[J]. Journal of System Simulation, 2024, 36(8): 1749-1763.

Figures/Tables 16

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Table 1

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Table 2

Fig. 11

Fig. 12

Fig. 13

Fig. 14

References 33

1	Gu Kim Hak, Heoun Taek Lim, Yong Man Ro. Deep Virtual Reality Image Quality Assessment with Human Perception Guider for Omnidirectional Image[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(4): 917-928.
2	Li Jia, Zhao Yifan, Ye Weihua, et al. Attentive Deep Stitching and Quality Assessment for 360° Omnidirectional Images[J]. IEEE Journal of Selected Topics in Signal Processing, 2020, 14(1): 209-221.
3	Wang Lang, Yu Wen, Li Bao. Multi-scenes Image Stitching Based on Autonomous Driving[C]//2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). Piscataway: IEEE, 2020: 694-698.
4	Chen Yuanming, Hong Xiaobin, Chen Weiguo, et al. Experimental Research on Overwater and Underwater Visual Image Stitching and Fusion Technology of Offshore Operation and Maintenance of Unmanned Ship[J]. Journal of Marine Science and Engineering, 2022, 10(6): 747.
5	马跃龙, 赵勇, 曹雪峰. 一种基于单目SLAM的无人机序列图像拼接方法[J]. 系统仿真学报, 2016, 28(10): 2349-2355.
	Ma Yuelong, Zhao Yong, Cao Xuefeng. Mono-SLAM Based Method for Stitching Images of Sequence from UAV[J]. Journal of System Simulation, 2016, 28(10): 2349-2355.
6	Zaragoza J, Chin T J, Brown M S, et al. As-projective-as-possible Image Stitching with Moving DLT[C]//2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2013: 2339-2346.
7	Lin C C, Pankanti S U, Ramamurthy K N, et al. Adaptive As-natural-as-possible Image Stitching[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2015: 1155-1163.
8	Chang Chehan, Sato Yoichi, Chuang Y Y. Shape-preserving Half-projective Warps for Image Stitching[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2014: 3254-3261.
9	Liu Shiguang, Chai Qingpeng. Shape-optimizing and Illumination-smoothing Image Stitching[J]. IEEE Transactions on Multimedia, 2019, 21(3): 690-703.
10	Chen Yusheng, Chuang Y Y. Natural Image Stitching with the Global Similarity Prior[C]//Computer Vision-ECCV 2016. Cham: Springer International Publishing, 2016: 186-201.
11	刘强, 韩敏. 基于直线特征约束网格变形的航拍图像拼接[J]. 控制与决策, 2022, 37(3): 669-675.
	Liu Qiang, Han Min. Linear-preserve-mesh Warps in Aerial Image Stitching[J]. Control and Decision, 2022, 37(3): 669-675.
12	何川, 周军. 具有直线结构保护的网格化图像拼接[J]. 中国图象图形学报, 2018, 23(7): 973-983.
	He Chuan, Zhou Jun. Mesh-based Image Stitching Algorithm with Linear Structure Protection[J]. Journal of Image and Graphics, 2018, 23(7): 973-983.
13	石祥滨, 贾贺林. 基于控制点网格优化的航拍图像拼接算法[J]. 计算机应用与软件, 2020, 37(5): 302-308.
	Shi Xiangbin, Jia Helin. Aerial Image Stitching Algorithm Based on Control Point Grid Optimization[J]. Computer Applications and Software, 2020, 37(5): 302-308.
14	Gao Junhong, Li Yu, Chin T J, et al. Seam-driven Image Stitching[C]//Eurographics 2013. Eindhoven: The Eurographics Association, 2013: 45-48.
15	Zhang Fan, Liu Feng. Parallax-tolerant Image Stitching[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2014: 3262-3269.
16	Lin Kaimo, Jiang Nianjuan, Cheong L F, et al. SEAGULL: Seam-guided Local Alignment for Parallax-tolerant Image Stitching[C]//Computer Vision-ECCV 2016. Cham: Springer International Publishing, 2016: 370-385.
17	Li Nan, Liao Tianli, Wang Chao. Perception-based Seam Cutting for Image Stitching[J]. Signal Image and Video Processing, 2018, 12(5): 967-974.
18	Cao Qingjie, Shi Zaifeng, Wang Pumeng, et al. A Seamless Image-stitching Method Based on Human Visual Discrimination and Attention[J]. Applied Sciences, 2020, 10(4): 1462.
19	Liao Tianli, Chen Jing, Xu Yifang. Quality Evaluation-based Iterative Seam Estimation for Image Stitching[J]. Signal Image and Video Processing, 2019, 13(6): 1199-1206.
20	Liao Tianli, Zhao Chenyang, Li Lei, et al. Seam-guided Local Alignment and Stitching for Large Parallax Images[EB/OL]. (2023-11-30) [2024-01-10]. .
21	DeTone D, Malisiewicz T, Rabinovich A. SuperPoint: Self-supervised Interest Point Detection and Description[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway: IEEE, 2018: 337-33712.
22	Li Jing, Wang Zhengming, Lai Shiming, et al. Parallax-tolerant Image Stitching Based on Robust Elastic Warping[J]. IEEE Transactions on Multimedia, 2018, 20(7): 1672-1687.
23	Joo Kyungdon, Kim Namil, Tae Hyun Oh, et al. Line Meets As-projective-as-possible Image Stitching with Moving DLT[C]//2015 IEEE International Conference on Image Processing (ICIP). Piscataway: IEEE, 2015: 1175-1179.
24	Liao Tianli, Li Nan. Single-perspective Warps in Natural Image Stitching[J]. IEEE Transactions on Image Processing, 2020, 29: 724-735.
25	Jia Qi, Li Zhengjun, Fan Xin, et al. Leveraging Line-point Consistence to Preserve Structures for Wide Parallax Image Stitching[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 12181-12190.
26	Shi Zaifeng, Li Hui, Cao Qingjie, et al. An Image Mosaic Method Based on Convolutional Neural Network Semantic Features Extraction[J]. Journal of Signal Processing Systems, 2020, 92(4): 435-444.
27	A A Alzohairy Tamer, Emad El-Dein H A, Sayed Mahmoud S. Image Mosaicing Based on Neural Networks[J]. International Journal of Computer Applications, 2016, 136(1): 25-31.
28	Nie Lang, Lin Chunyu, Liao Kang, et al. Deep Rectangling for Image Stitching: A Learning Baseline[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 5730-5738.
29	Song Dae Young, Lee Geonsoo, Lee HeeKyung, et al. Weakly-supervised Stitching Network for Real-world Panoramic Image Generation[C]//Computer Vision-ECCV 2022. Cham: Springer Nature Switzerland, 2022: 54-71.
30	Nie Lang, Lin Chunyu, Liao Kang, et al. Unsupervised Deep Image Stitching: Reconstructing Stitched Features to Images[J]. IEEE Transactions on Image Processing, 2021, 30: 6184-6197.
31	Nie Lang, Lin Chunyu, Liao Kang, et al. Parallax-tolerant Unsupervised Deep Image Stitching[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2023: 7365-7374.
32	Liu Ce, Yuen J, Torralba A. SIFT Flow: Dense Correspondence Across Scenes and Its Applications[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(5): 978-994.
33	Bingham B, Agüero Carlos, McCarrin M, et al. Toward Maritime Robotic Simulation in Gazebo[C]//Oceans 2019 MTS/IEEE Seattle. Piscataway: IEEE, 2019: 1-10.

数据	文献[17]		文献[20]		本文算法
数据	e₁	e₂	e₁	e₂	e₁	e₂
01	0.103	0.212	0.084	0.173	0.059	0.161
02	0.195	0.300	0.231	0.221	0.122	0.154
03	0.098	0.213	0.079	0.281	0.063	0.163
04	0.139	0.240	0.086	0.267	0.093	0.229
05	0.112	0.379	0.056	0.331	0.037	0.292
06	0.036	0.277	0.036	0.253	0.024	0.226
07	0.031	0.241	0.025	0.233	0.022	0.225
08	0.077	0.135	0.066	0.117	0.066	0.112
09	0.140	0.272	0.153	0.316	0.096	0.270
10	0.067	0.284	0.036	0.290	0.023	0.308
11	0.263	0.376	0.080	0.281	0.066	0.172
12	0.170	0.282	0.140	0.217	0.121	0.238
13	0.102	0.349	0.109	0.356	0.064	0.292
14	0.168	0.109	0.185	0.161	0.132	0.090
15	0.121	0.142	0.160	0.241	0.127	0.099
16	0.102	0.274	0.081	0.219	0.056	0.200
17	0.134	0.215	0.115	0.320	0.108	0.204
18	0.192	0.213	0.220	0.273	0.109	0.201
19	0.128	0.169	0.064	0.180	0.077	0.107
20	0.068	0.197	0.043	0.113	0.021	0.098

数据	Baseline		方法Ⅰ		方法Ⅰ+Ⅱ		方法Ⅰ+Ⅱ+Ⅲ
数据	e₁	e₂	e₁	e₂	e₁	e₂	e₁	e₂
01	0.139	0.305	0.134	0.279	0.069	0.171	0.059	0.161
02	0.241	0.267	0.199	0.220	0.156	0.167	0.122	0.154
03	0.130	0.247	0.128	0.237	0.073	0.207	0.063	0.163
04	0.107	0.211	0.104	0.205	0.096	0.238	0.093	0.229
05	0.078	0.385	0.074	0.328	0.074	0.309	0.037	0.292
06	0.037	0.265	0.036	0.258	0.025	0.246	0.024	0.226
07	0.034	0.256	0.034	0.236	0.027	0.233	0.022	0.225
08	0.096	0.142	0.077	0.135	0.076	0.124	0.066	0.112
09	0.141	0.311	0.140	0.303	0.110	0.276	0.096	0.270
10	0.069	0.378	0.060	0.375	0.029	0.317	0.023	0.308
11	0.169	0.228	0.138	0.217	0.089	0.197	0.066	0.172
12	0.224	0.295	0.168	0.287	0.140	0.268	0.121	0.238
13	0.133	0.367	0.102	0.348	0.071	0.320	0.064	0.292
14	0.221	0.186	0.187	0.177	0.203	0.170	0.132	0.090
15	0.149	0.170	0.143	0.158	0.144	0.157	0.127	0.099
16	0.153	0.167	0.079	0.136	0.075	0.224	0.056	0.200
17	0.156	0.242	0.151	0.238	0.128	0.237	0.108	0.204
18	0.212	0.248	0.207	0.244	0.149	0.230	0.109	0.201
19	0.159	0.219	0.151	0.205	0.184	0.209	0.077	0.107
20	0.052	0.198	0.052	0.184	0.035	0.136	0.021	0.098

[1]	Ma Jialin, Zhang Jinming, Sun Weixin. Research on Panoramic Image Mosaic Method Based on Camera Calibration [J]. Journal of System Simulation, 2017, 29(5): 1112-1119.
[2]	Ma Yuelong, Zhao Yong, Cao Xuefeng. Mono-SLAM Based Method for Stitching Images of Sequence From UAV [J]. Journal of System Simulation, 2016, 28(10): 2349-2355.