A Graph Neural Network Visual SLAM Algorithm for Large-angle View Motion

doi:10.16182/j.issn1004731x.joss.23-0053

Abstract

Abstract:

Aimed at the difficulty of feature point extraction in mobile robots with drastic changes in illumination or sparse texture scenes under large-angle view motion, difficulty in matching features at extreme angles leads to large errors in Epipolar Geometry calculations,a fusion of an improved graph neural network based visual SLAM algorithm (GNN-SLAM)is proposed.The priori location estimation feature extraction network is proposed to achieve fast and uniform detection and description of image feature points by a priori location estimation and to construct real and accurate feature point information.The graph attention mechanism feature matching network is proposed to aggregate feature point information through message passing graph neural network, and then use self and joint attention mechanism for before and after image frame weighted feature matching.The feature extraction and matching map neural network is fused with the back-end nonlinear optimization, closed-loop correction, and local mapping modules of the ORB-SLAM2 system to propose a complete monocular vision GNN-SLAM system. The experimental results show that:compared with the ORB-SLAM2 algorithm on the KITTI dataset, the absolute trajectory error of this algorithm is reduced by 37.59%, and the relative pose error is reduced by 19.67%.

Key words: SLAM, large-angle view motion, graph neural network, graph attention mechanism, mobile robot

CLC Number:

TP242

Liu Jinhui, Chen Mengyuan, Han Pengpeng, Chen Hebao, Zhang Yukun. A Graph Neural Network Visual SLAM Algorithm for Large-angle View Motion[J]. Journal of System Simulation, 2024, 36(5): 1043-1060.

Figures/Tables 16

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Table 1

Fig. 7

Fig. 8

Table 2

Fig. 9

Table 3

Fig. 10

Fig. 11

Fig. 12

Table 4

References 33

1	Smith R C, Cheeseman P. On the Representation and Estimation of Spatial Uncertainty[J]. The International Journal of Robotics Research, 1986, 5(4): 56-68.
2	Dissanayake M W M G, Newman P, Clark S, et al. A Solution to the Simultaneous Localization and Map Building (SLAM) Problem[J]. IEEE Transactions on Robotics and Automation, 2001, 17(3): 229-241.
3	王晨捷, 罗斌, 李成源, 等. 无人机视觉SLAM协同建图与导航[J]. 测绘学报, 2020, 49(6): 767-776.
	Wang Chenjie, Luo Bin, Li Chengyuan, et al. The Collaborative Mapping and Navigation Based on Visual SLAM in UAV Platform[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(6): 767-776.
4	Mur-Artal Raúl, Tardós Juan D. ORB-SLAM2: An Open-source SLAM System for Monocular, Stereo, and RGB-D Cameras[J]. IEEE Transactions on Robotics, 2017, 33(5): 1255-1262.
5	Rublee E, Rabaud V, Konolige K, et al. ORB: An Efficient Alternative to SIFT or SURF[C]//2011 International Conference on Computer Vision. Piscataway, NJ, USA: IEEE, 2011: 2564-2571.
6	聂伟, 文怀志, 谢良波, 等. 一种基于单目视觉的无人机室内定位方法[J]. 电子与信息学报, 2022, 44(3): 906-914.
	Nie Wei, Wen Huaizhi, Xie Liangbo, et al. Indoor Localization of UAV Using Monocular Vision[J]. Journal of Electronics & Information Technology, 2022, 44(3): 906-914.
7	丁文东, 徐德, 刘希龙, 等. 移动机器人视觉里程计综述[J]. 自动化学报, 2018, 44(3): 385-400.
	Ding Wendong, Xu De, Liu Xilong, et al. Review on Visual Odometry for Mobile Robots[J]. Acta Automatica Sinica, 2018, 44(3): 385-400.
8	陈孟元, 张玉坤, 田德红, 等. 基于兴趣倾向机制的仿生SLAM算法[J]. 电子与信息学报, 2022, 44(5): 1743-1753.
	Chen Mengyuan, Zhang Yukun, Tian Dehong, et al. Bionic SLAM Algorithm Based on Interest Tendency Mechanism[J]. Journal of Electronics & Information Technology, 2022, 44(5): 1743-1753.
9	邸凯昌, 万文辉, 赵红颖, 等. 视觉SLAM技术的进展与应用[J]. 测绘学报, 2018, 47(6): 770-779.
	Di Kaichang, Wan Wenhui, Zhao Hongying, et al. Progress and Applications of Visual SLAM[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6): 770-779.
10	牛珉玉, 黄宜庆. 基于动态耦合与空间数据关联的RGB-D SLAM算法[J]. 机器人, 2022, 44(3): 333-342.
	Niu Minyu, Huang Yiqing. An RGB-D SLAM Algorithm Based on Dynamic Coupling and Spatial Data Association[J]. Robot, 2022, 44(3): 333-342.
11	林景栋, 吴欣怡, 柴毅, 等. 卷积神经网络结构优化综述[J]. 自动化学报, 2020, 46(1): 24-37.
	Lin Jingdong, Wu Xinyi, Chai Yi, et al. Structure Optimization of Convolutional Neural Networks: A Survey[J]. Acta Automatica Sinica, 2020, 46(1): 24-37.
12	Tateno Keisuke, Tombari Federico, Laina Iro, et al. CNN-SLAM: Real-time Dense Monocular SLAM with Learned Depth Prediction[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2017: 6565-6574.
13	Hou Yi, Zhang Hong, Zhou Shilin. Convolutional Neural Network-based Image Representation for Visual Loop Closure Detection[C]//2015 IEEE International Conference on Information and Automation. Piscataway, NJ, USA: IEEE, 2015: 2238-2245.
14	李维鹏, 张国良, 姚二亮, 等. 基于场景显著区域的改进闭环检测算法[J]. 机器人, 2017, 39(1): 23-35.
	Li Weipeng, Zhang Guoliang, Yao Erliang, et al. An Improved Loop Closure Detection Algorithm Based on Scene Salient Regions[J]. Robot, 2017, 39(1): 23-35.
15	Ganti Pranav, Waslander Steven L. Network Uncertainty Informed Semantic Feature Selection for Visual SLAM[C]//2019 16th Conference on Computer and Robot Vision (CRV). Piscataway, NJ, USA: IEEE, 2019: 121-128.
16	Tang Jiexiong, Ericson Ludvig, Folkesson John, et al. GCNv2: Efficient Correspondence Prediction for Real-time SLAM[J]. IEEE Robotics and Automation Letters, 2019, 4(4): 3505-3512.
17	Li Dongjiang, Shi Xuesong, Long Qiwei, et al. DXSLAM: A Robust and Efficient Visual SLAM System with Deep Features[C]//2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway, NJ, USA: IEEE, 2020: 4958-4965.
18	Lee J B, Rossi R A, Kim S, et al. Attention Models in Graphs: A Survey[J]. ACM Transactions on Knowledge Discovery from Data, 2019, 13(6): 62.
19	Zhuang Chenyi, Ma Qiang. Dual Graph Convolutional Networks for Graph-based Semi-supervised Classification[C]//Proceedings of the 2018 World Wide Web Conference. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee, 2018: 499-508.
20	Zhang Xiwu, Su Yan, Zhu Xinhua. Loop Closure Detection for Visual SLAM Systems Using Convolutional Neural Network[C]//2017 23rd International Conference on Automation and Computing (ICAC). Piscataway, NJ, USA: IEEE, 2017: 1-6.
21	Scarselli Franco, Gori Marco, Ah Chung Tsoi, et al. The Graph Neural Network Model[J]. IEEE Transactions on Neural Networks, 2009, 20(1): 61-80.
22	徐冰冰, 岑科廷, 黄俊杰, 等. 图卷积神经网络综述[J]. 计算机学报, 2020, 43(5): 755-780.
	Xu Bingbing, Cen Keting, Huang Junjie, et al. A Survey on Graph Convolutional Neural Network[J]. Chinese Journal of Computers, 2020, 43(5): 755-780.
23	吴博, 梁循, 张树森, 等. 图神经网络前沿进展与应用[J]. 计算机学报, 2022, 45(1): 35-68.
	Wu Bo, Liang Xun, Zhang Shusen, et al. Advances and Applications in Graph Neural Network[J]. Chinese Journal of Computers, 2022, 45(1): 35-68.
24	Jia Kui, Chan T H, Zeng Zinan, et al. ROML: A Robust Feature Correspondence Approach for Matching Objects in a Set of Images[J]. International Journal of Computer Vision, 2016, 117(2): 173-197.
25	Sarlin Paul-Edouard, DeTone D, Malisiewicz T, et al. SuperGlue: Learning Feature Matching with Graph Neural Networks[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2020: 4937-4946.
26	Elmoogy Ahmed, Dong Xiaodai, Lu Tao, et al. Pose-GNN: Camera Pose Estimation System Using Graph Neural Networks[EB/OL]. (2021-03-17) [2022-12-15]. .
27	Wang Limin, Guo Sheng, Huang Weilin, et al. Places205-VGGNet Models for Scene Recognition[EB/OL]. (2015-08-07) [2022-11-07]. .
28	Meng Zihang, Ravi S N, Singh V. Physarum Powered Differentiable Linear Programming Layers and Applications[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(10): 8939-8949.
29	Lin T Y, Maire M, Belongie S, et al. Microsoft COCO: Common Objects in Context[C]//Computer Vision-ECCV 2014. Cham: Springer International Publishing, 2014: 740-755.
30	Sturm Jürgen, Engelhard Nikolas, Endres Felix, et al. A Benchmark for the Evaluation of RGB-D SLAM Systems[C]//2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway, NJ, USA: IEEE, 2012: 573-580.
31	Geiger A, Lenz P, Stiller C, et al. Vision Meets Robotics: The KITTI Dataset[J]. The International Journal of Robotics Research, 2013, 32(11): 1231-1237.
32	Balntas V, Lenc K, Vedaldi A, et al. HPatches: A Benchmark and Evaluation of Handcrafted and Learned Local Descriptors[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2017: 3852-3861.
33	DeTone D, Malisiewicz T, Rabinovich A. SuperPoint: Self-supervised Interest Point Detection and Description[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway, NJ, USA: IEEE, 2018: 337-33712.

数据集	ORB	Superpoint	GCNv2	本文
i_books	0.572	0.591	0.627	0.693
i_castle	0.583	0.614	0.641	0.716
i_fruits	0.537	0.569	0.599	0.593
i_ski	0.564	0.576	0.598	0.645
i_whitebuilding	0.524	0.513	0.545	0.617

数据集	ORB+FLANN			Superpoint+FLANN			GCNv2+FLANN			本文
数据集	@30°	@60°	@90°	@30°	@60°	@90°	@30°	@60°	@90°	@30°	@60°	@90°
v_abstract	0.643	0.465	0.298	0.620	0.473	0.239	0.728	0.533	0.295	0.802	0.739	0.507
v_bees	0.651	0.482	0.262	0.702	0.536	0.388	0.648	0.499	0.386	0.857	0.686	0.569
v_beyus	0.682	0.503	0.312	0.608	0.449	0.256	0.612	0.428	0.324	0.829	0.737	0.571
v_home	0.605	0.396	0.274	0.633	0.405	0.293	0.730	0.475	0.252	0.796	0.706	0.535
v_woman	0.597	0.391	0.238	0.652	0.433	0.264	0.671	0.416	0.241	0.840	0.744	0.522

数据集	ORB-SLAM2		DX-SLAM		GCNv2-SLAM		本文
数据集	ATE	RPE	ATE	RPE	ATE	RPE	ATE	RPE
fr1/360	―	―	―	―	0.065	0.082	0.145	0.199
fr1/floor	0.390	0.469	―	―	0.138	0.167	0.019	0.038
fr1/desk	0.081	0.101	0.300	0.364	0.188	0.230	0.039	0.050
fr1/desk2	0.571	0.999	0.343	0.416	0.156	0.197	0.047	0.075
fr1/room	0.295	0.354	0.181	0.212	0.246	0.305	0.070	0.111
fr2/desk	0.931	1.149	0.581	0.712	0.160	0.199	0.109	0.133
fr3/long_office_household	1.222	1.553	0.791	1.007	0.363	0.455	0.169	0.176

算法	特征提取/ms	特征匹配/ms	总时间/ms	平均帧率/(帧/s)
ORB-SLAM2	12.88	15.06	27.94	30
DX-SLAM	32.42	39.24	71.66	15
GCNv2-SLAM	19.45	31.57	51.02	20
本文	16.32	30.79	47.11	20

[1]	Lü Jinhu, Jiang Hongyi, Liu Deyuan, Tan Shaolin. Modeling and Simulation of Complex Systems Based on Graph Neural Networks [J]. Journal of System Simulation, 2025, 37(7): 1624-1638.
[2]	Wang Ziyi, Zhang Kai, Qian Dianwei, Liu Yuzhen. A DRL⁃based Approach for Distributed Equipment Nodes Selection [J]. Journal of System Simulation, 2025, 37(6): 1565-1573.
[3]	Zhou Xiaohui, Li Yanqiang, Wang Yong, Zhao Decai, Yang Xiaoyao. Robot Path Planning Based on Ant Colony Algorithm with Dual Heuristic Information [J]. Journal of System Simulation, 2025, 37(5): 1280-1289.
[4]	Yu Die, Bao Baizhong, Si Yan, Duan Jian, Zhan Xiaobin, Shi Tielin. Mobile Robot Path Planning Based on Search-step Optimized A* Algorithm [J]. Journal of System Simulation, 2025, 37(4): 1041-1050.
[5]	Lin Guijuan, Li Zihan, Wang Yu. Research on Improved A* Algorithm Path Planning Based on Global Key Point Extraction [J]. Journal of System Simulation, 2025, 37(3): 667-678.
[6]	Li Ruoqing, Zhao Yaochi, Hu Zhuhua, Qi Wenlu, Liu Guangfeng. TOHF: A Feature Extractor for Resource-constrained Indoor VSLAM [J]. Journal of System Simulation, 2025, 37(3): 691-703.
[7]	Jiang Limei, Chen Xinwei. Visual SLAM Algorithm Based on Feature Point Selection in Dynamic Scenes [J]. Journal of System Simulation, 2025, 37(3): 753-762.
[8]	Li Jiongyi, Li Qiang, Zhang Xinwen, Htet Zin Myo, Cai Yongbin. Improved Bidirectional A* Quadratic Path Planning Algorithm for Mobile Robots [J]. Journal of System Simulation, 2025, 37(2): 498-507.
[9]	Li Xiaobin, Hu Bing, Yin Chao, Li Bo, Ma Jun. Spatiotemporal Graph Convolution-based Demand Forecasting and Simulation Analysis for Automotive Parts Supply Chain [J]. Journal of System Simulation, 2025, 37(12): 3060-3074.
[10]	Zhang Wenkang, Sun Xiaofeng, Zhong Yiping, Yin Yong. Numerical Simulations of Ship Liquid Tank Sloshing Based on Graph Neural Networks [J]. Journal of System Simulation, 2025, 37(12): 3087-3099.
[11]	Liu Jia, Zhang Zengwei, Chen Dapeng, Huang Nanxuan, Wang Bin, Song Hong. Improvement of SLAM Localization Accuracy in AR by Enhancing YOLOv8 [J]. Journal of System Simulation, 2025, 37(11): 2701-2713.
[12]	Zhang Chi, Wei Wei. Path Planning for Mobile Robots Based on Improved Artificial Potential Field Algorithm [J]. Journal of System Simulation, 2025, 37(11): 2918-2926.
[13]	Yang Lanying, Li Chao, Zou Haifeng, Wan Jiangtao, Zhang Renqiang, Liu Hui, Lu Hong. Robot Path Planning Optimization Based on Fusion of Improved Ant Colony Algorithm and A* Algorithm [J]. Journal of System Simulation, 2025, 37(11): 2956-2965.
[14]	Luo Yi, Deng Jia. Path Planning for Mobile Robots Based on Improved RRT-Connect and DWA Fusion [J]. Journal of System Simulation, 2025, 37(10): 2545-2556.
[15]	Xu Jianmin, Song Lei, Deng Dongdong, Chen Yaoruo, Yang Wei. Path Planning of Mobile Robot Based on the Integration of Multi-scale A* and Optimized DWA Algorithm [J]. Journal of System Simulation, 2025, 37(1): 257-270.