面向无人船视觉感知的几何结构保护视差图像拼接

doi:10.16182/j.issn1004731x.joss.24-0310

摘要/Abstract

摘要：

针对海上低纹理、大视差图像拼接过程中由于误对齐产生的伪影问题，提出一种基于点线特征配准和最佳接缝线融合的几何结构保视差图像拼接算法。在传统的基于点特征求解单应变换的基础上引入线段特征，并将潜在的共面局部线段合并为全局线段，为接缝线融合提供精准对齐的条件；在图像融合过程中，利用tanh度量的颜色差异和梯度差异以及引入显著性检测权重来设计接缝切割方法的能量函数，引导最佳接缝线避开图像中显著海上结构，从而确保结构边缘的连续性；使用SIFT flow方法校正拼接缝上的错位像素，实现几何结构准确的海上图像拼接。在20对不同场景数据上的拼接实验表明，与基准方法相比，所提算法的基于SSIM的接缝质量误差平均降低了44.6%，最多降低66.7%，基于ZNCC(zero mean normalized cross-correlation)的接缝质量误差平均降低了24.7%，最多降低了51.6%，能够有效地避免伪影，从而得到观感自然的拼接结果，满足无人船航行过程中对宽视场的需求。

关键词: 无人船视觉感知, 图像拼接, 点线一致性配准, 几何结构保护, 最佳接缝线融合

Abstract:

In order to address the artifacts caused by misalignment in maritime image stitch with low texture and large parallax, a geometric structure parallax-tolerant image stitch algorithm based on point-line feature registration and optimal seam fusion is proposed. Line segment features are introduced into the traditional homography transformation based on point features, and potential coplanar local line segments are merged into global line segments to provide accurate alignment conditions for seam line fusion. In the image fusion stage, the energy function of seam cutting method is designed by using the color difference and gradient difference of tanh measure and introducing saliency detection weight, which guides the optimal seam to avoid prominent maritime structures in the image, thus ensuring the continuity of structural edges. The SIFT flow method is used to correct the misalignment pixels on the stitching seam, and the maritime image stitching with accurate geometric structure is realized. Experimental results, based on 20 pairs of diverse scene data, compaed with the baseline method, the proposed algorithm achieves an average 44.6% reduction in seam quality error based on structural similarity, with a maximum reduction of 66.7%. The average reduction in seam quality error based on zero mean normalized cross-correlation is 24.7%, with a maximum reduction of 51.6%. The algorithm can effectively avoid artifacts, thus achieving visually natural stitching results and satisfying the requirements for a wide field of view during unmanned ship navigation.

Key words: unmanned ship visual perception, image stitching, point-line consistence registration, geometric structure protection, optimal seam fusion

中图分类号:

TP391.9

杨志林,尹勇,张如凯等 . 面向无人船视觉感知的几何结构保护视差图像拼接[J]. 系统仿真学报, 2024, 36(8): 1749-1763.

Yang Zhilin,Yin Yong,Zhang Rukai,et al . Parallax-tolerant Image Stitching with Geometric Structure Protection for Unmanned Ship Visual Perception[J]. Journal of System Simulation, 2024, 36(8): 1749-1763.

图/表 16

图1

图2

图3

图4

图5

表1

图6

图7

图8

图9

图10

表2

图11

图12

图13

图14

参考文献 33

1	Gu Kim Hak, Heoun Taek Lim, Yong Man Ro. Deep Virtual Reality Image Quality Assessment with Human Perception Guider for Omnidirectional Image[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(4): 917-928.
2	Li Jia, Zhao Yifan, Ye Weihua, et al. Attentive Deep Stitching and Quality Assessment for 360° Omnidirectional Images[J]. IEEE Journal of Selected Topics in Signal Processing, 2020, 14(1): 209-221.
3	Wang Lang, Yu Wen, Li Bao. Multi-scenes Image Stitching Based on Autonomous Driving[C]//2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). Piscataway: IEEE, 2020: 694-698.
4	Chen Yuanming, Hong Xiaobin, Chen Weiguo, et al. Experimental Research on Overwater and Underwater Visual Image Stitching and Fusion Technology of Offshore Operation and Maintenance of Unmanned Ship[J]. Journal of Marine Science and Engineering, 2022, 10(6): 747.
5	马跃龙, 赵勇, 曹雪峰. 一种基于单目SLAM的无人机序列图像拼接方法[J]. 系统仿真学报, 2016, 28(10): 2349-2355.
	Ma Yuelong, Zhao Yong, Cao Xuefeng. Mono-SLAM Based Method for Stitching Images of Sequence from UAV[J]. Journal of System Simulation, 2016, 28(10): 2349-2355.
6	Zaragoza J, Chin T J, Brown M S, et al. As-projective-as-possible Image Stitching with Moving DLT[C]//2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2013: 2339-2346.
7	Lin C C, Pankanti S U, Ramamurthy K N, et al. Adaptive As-natural-as-possible Image Stitching[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2015: 1155-1163.
8	Chang Chehan, Sato Yoichi, Chuang Y Y. Shape-preserving Half-projective Warps for Image Stitching[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2014: 3254-3261.
9	Liu Shiguang, Chai Qingpeng. Shape-optimizing and Illumination-smoothing Image Stitching[J]. IEEE Transactions on Multimedia, 2019, 21(3): 690-703.
10	Chen Yusheng, Chuang Y Y. Natural Image Stitching with the Global Similarity Prior[C]//Computer Vision-ECCV 2016. Cham: Springer International Publishing, 2016: 186-201.
11	刘强, 韩敏. 基于直线特征约束网格变形的航拍图像拼接[J]. 控制与决策, 2022, 37(3): 669-675.
	Liu Qiang, Han Min. Linear-preserve-mesh Warps in Aerial Image Stitching[J]. Control and Decision, 2022, 37(3): 669-675.
12	何川, 周军. 具有直线结构保护的网格化图像拼接[J]. 中国图象图形学报, 2018, 23(7): 973-983.
	He Chuan, Zhou Jun. Mesh-based Image Stitching Algorithm with Linear Structure Protection[J]. Journal of Image and Graphics, 2018, 23(7): 973-983.
13	石祥滨, 贾贺林. 基于控制点网格优化的航拍图像拼接算法[J]. 计算机应用与软件, 2020, 37(5): 302-308.
	Shi Xiangbin, Jia Helin. Aerial Image Stitching Algorithm Based on Control Point Grid Optimization[J]. Computer Applications and Software, 2020, 37(5): 302-308.
14	Gao Junhong, Li Yu, Chin T J, et al. Seam-driven Image Stitching[C]//Eurographics 2013. Eindhoven: The Eurographics Association, 2013: 45-48.
15	Zhang Fan, Liu Feng. Parallax-tolerant Image Stitching[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2014: 3262-3269.
16	Lin Kaimo, Jiang Nianjuan, Cheong L F, et al. SEAGULL: Seam-guided Local Alignment for Parallax-tolerant Image Stitching[C]//Computer Vision-ECCV 2016. Cham: Springer International Publishing, 2016: 370-385.
17	Li Nan, Liao Tianli, Wang Chao. Perception-based Seam Cutting for Image Stitching[J]. Signal Image and Video Processing, 2018, 12(5): 967-974.
18	Cao Qingjie, Shi Zaifeng, Wang Pumeng, et al. A Seamless Image-stitching Method Based on Human Visual Discrimination and Attention[J]. Applied Sciences, 2020, 10(4): 1462.
19	Liao Tianli, Chen Jing, Xu Yifang. Quality Evaluation-based Iterative Seam Estimation for Image Stitching[J]. Signal Image and Video Processing, 2019, 13(6): 1199-1206.
20	Liao Tianli, Zhao Chenyang, Li Lei, et al. Seam-guided Local Alignment and Stitching for Large Parallax Images[EB/OL]. (2023-11-30) [2024-01-10]. .
21	DeTone D, Malisiewicz T, Rabinovich A. SuperPoint: Self-supervised Interest Point Detection and Description[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway: IEEE, 2018: 337-33712.
22	Li Jing, Wang Zhengming, Lai Shiming, et al. Parallax-tolerant Image Stitching Based on Robust Elastic Warping[J]. IEEE Transactions on Multimedia, 2018, 20(7): 1672-1687.
23	Joo Kyungdon, Kim Namil, Tae Hyun Oh, et al. Line Meets As-projective-as-possible Image Stitching with Moving DLT[C]//2015 IEEE International Conference on Image Processing (ICIP). Piscataway: IEEE, 2015: 1175-1179.
24	Liao Tianli, Li Nan. Single-perspective Warps in Natural Image Stitching[J]. IEEE Transactions on Image Processing, 2020, 29: 724-735.
25	Jia Qi, Li Zhengjun, Fan Xin, et al. Leveraging Line-point Consistence to Preserve Structures for Wide Parallax Image Stitching[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2021: 12181-12190.
26	Shi Zaifeng, Li Hui, Cao Qingjie, et al. An Image Mosaic Method Based on Convolutional Neural Network Semantic Features Extraction[J]. Journal of Signal Processing Systems, 2020, 92(4): 435-444.
27	A A Alzohairy Tamer, Emad El-Dein H A, Sayed Mahmoud S. Image Mosaicing Based on Neural Networks[J]. International Journal of Computer Applications, 2016, 136(1): 25-31.
28	Nie Lang, Lin Chunyu, Liao Kang, et al. Deep Rectangling for Image Stitching: A Learning Baseline[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 5730-5738.
29	Song Dae Young, Lee Geonsoo, Lee HeeKyung, et al. Weakly-supervised Stitching Network for Real-world Panoramic Image Generation[C]//Computer Vision-ECCV 2022. Cham: Springer Nature Switzerland, 2022: 54-71.
30	Nie Lang, Lin Chunyu, Liao Kang, et al. Unsupervised Deep Image Stitching: Reconstructing Stitched Features to Images[J]. IEEE Transactions on Image Processing, 2021, 30: 6184-6197.
31	Nie Lang, Lin Chunyu, Liao Kang, et al. Parallax-tolerant Unsupervised Deep Image Stitching[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2023: 7365-7374.
32	Liu Ce, Yuen J, Torralba A. SIFT Flow: Dense Correspondence Across Scenes and Its Applications[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(5): 978-994.
33	Bingham B, Agüero Carlos, McCarrin M, et al. Toward Maritime Robotic Simulation in Gazebo[C]//Oceans 2019 MTS/IEEE Seattle. Piscataway: IEEE, 2019: 1-10.

数据	文献[17]		文献[20]		本文算法
数据	e₁	e₂	e₁	e₂	e₁	e₂
01	0.103	0.212	0.084	0.173	0.059	0.161
02	0.195	0.300	0.231	0.221	0.122	0.154
03	0.098	0.213	0.079	0.281	0.063	0.163
04	0.139	0.240	0.086	0.267	0.093	0.229
05	0.112	0.379	0.056	0.331	0.037	0.292
06	0.036	0.277	0.036	0.253	0.024	0.226
07	0.031	0.241	0.025	0.233	0.022	0.225
08	0.077	0.135	0.066	0.117	0.066	0.112
09	0.140	0.272	0.153	0.316	0.096	0.270
10	0.067	0.284	0.036	0.290	0.023	0.308
11	0.263	0.376	0.080	0.281	0.066	0.172
12	0.170	0.282	0.140	0.217	0.121	0.238
13	0.102	0.349	0.109	0.356	0.064	0.292
14	0.168	0.109	0.185	0.161	0.132	0.090
15	0.121	0.142	0.160	0.241	0.127	0.099
16	0.102	0.274	0.081	0.219	0.056	0.200
17	0.134	0.215	0.115	0.320	0.108	0.204
18	0.192	0.213	0.220	0.273	0.109	0.201
19	0.128	0.169	0.064	0.180	0.077	0.107
20	0.068	0.197	0.043	0.113	0.021	0.098

数据	Baseline		方法Ⅰ		方法Ⅰ+Ⅱ		方法Ⅰ+Ⅱ+Ⅲ
数据	e₁	e₂	e₁	e₂	e₁	e₂	e₁	e₂
01	0.139	0.305	0.134	0.279	0.069	0.171	0.059	0.161
02	0.241	0.267	0.199	0.220	0.156	0.167	0.122	0.154
03	0.130	0.247	0.128	0.237	0.073	0.207	0.063	0.163
04	0.107	0.211	0.104	0.205	0.096	0.238	0.093	0.229
05	0.078	0.385	0.074	0.328	0.074	0.309	0.037	0.292
06	0.037	0.265	0.036	0.258	0.025	0.246	0.024	0.226
07	0.034	0.256	0.034	0.236	0.027	0.233	0.022	0.225
08	0.096	0.142	0.077	0.135	0.076	0.124	0.066	0.112
09	0.141	0.311	0.140	0.303	0.110	0.276	0.096	0.270
10	0.069	0.378	0.060	0.375	0.029	0.317	0.023	0.308
11	0.169	0.228	0.138	0.217	0.089	0.197	0.066	0.172
12	0.224	0.295	0.168	0.287	0.140	0.268	0.121	0.238
13	0.133	0.367	0.102	0.348	0.071	0.320	0.064	0.292
14	0.221	0.186	0.187	0.177	0.203	0.170	0.132	0.090
15	0.149	0.170	0.143	0.158	0.144	0.157	0.127	0.099
16	0.153	0.167	0.079	0.136	0.075	0.224	0.056	0.200
17	0.156	0.242	0.151	0.238	0.128	0.237	0.108	0.204
18	0.212	0.248	0.207	0.244	0.149	0.230	0.109	0.201
19	0.159	0.219	0.151	0.205	0.184	0.209	0.077	0.107
20	0.052	0.198	0.052	0.184	0.035	0.136	0.021	0.098

[1]	马嘉琳, 张锦明, 孙卫新. 基于相机标定的全景图拼接方法研究[J]. 系统仿真学报, 2017, 29(5): 1112-1119.
[2]	赵毅力, 徐丹, 钱文华, 张雁. 基于种子区域增长的快速图像融合[J]. 系统仿真学报, 2016, 28(9): 1985-1989.
[3]	马跃龙, 赵勇, 曹雪峰. 一种基于单目SLAM的无人机序列图像拼接方法[J]. 系统仿真学报, 2016, 28(10): 2349-2355.