A Graph Neural Network Visual SLAM Algorithm for Large-angle View Motion

doi:10.16182/j.issn1004731x.joss.23-0053

Abstract

Abstract:

Aimed at the difficulty of feature point extraction in mobile robots with drastic changes in illumination or sparse texture scenes under large-angle view motion, difficulty in matching features at extreme angles leads to large errors in Epipolar Geometry calculations,a fusion of an improved graph neural network based visual SLAM algorithm (GNN-SLAM)is proposed.The priori location estimation feature extraction network is proposed to achieve fast and uniform detection and description of image feature points by a priori location estimation and to construct real and accurate feature point information.The graph attention mechanism feature matching network is proposed to aggregate feature point information through message passing graph neural network, and then use self and joint attention mechanism for before and after image frame weighted feature matching.The feature extraction and matching map neural network is fused with the back-end nonlinear optimization, closed-loop correction, and local mapping modules of the ORB-SLAM2 system to propose a complete monocular vision GNN-SLAM system. The experimental results show that:compared with the ORB-SLAM2 algorithm on the KITTI dataset, the absolute trajectory error of this algorithm is reduced by 37.59%, and the relative pose error is reduced by 19.67%.

Key words: SLAM, large-angle view motion, graph neural network, graph attention mechanism, mobile robot

CLC Number:

TP242

Liu Jinhui, Chen Mengyuan, Han Pengpeng, Chen Hebao, Zhang Yukun. A Graph Neural Network Visual SLAM Algorithm for Large-angle View Motion[J]. Journal of System Simulation, 2024, 36(5): 1043-1060.

Figures/Tables 16

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Table 1

Fig. 7

Fig. 8

Table 2

Fig. 9

Table 3

Fig. 10

Fig. 11

Fig. 12

Table 4

References 33

1	Smith R C, Cheeseman P. On the Representation and Estimation of Spatial Uncertainty[J]. The International Journal of Robotics Research, 1986, 5(4): 56-68.
2	Dissanayake M W M G, Newman P, Clark S, et al. A Solution to the Simultaneous Localization and Map Building (SLAM) Problem[J]. IEEE Transactions on Robotics and Automation, 2001, 17(3): 229-241.
3	王晨捷, 罗斌, 李成源, 等. 无人机视觉SLAM协同建图与导航[J]. 测绘学报, 2020, 49(6): 767-776.
	Wang Chenjie, Luo Bin, Li Chengyuan, et al. The Collaborative Mapping and Navigation Based on Visual SLAM in UAV Platform[J]. Acta Geodaetica et Cartographica Sinica, 2020, 49(6): 767-776.
4	Mur-Artal Raúl, Tardós Juan D. ORB-SLAM2: An Open-source SLAM System for Monocular, Stereo, and RGB-D Cameras[J]. IEEE Transactions on Robotics, 2017, 33(5): 1255-1262.
5	Rublee E, Rabaud V, Konolige K, et al. ORB: An Efficient Alternative to SIFT or SURF[C]//2011 International Conference on Computer Vision. Piscataway, NJ, USA: IEEE, 2011: 2564-2571.
6	聂伟, 文怀志, 谢良波, 等. 一种基于单目视觉的无人机室内定位方法[J]. 电子与信息学报, 2022, 44(3): 906-914.
	Nie Wei, Wen Huaizhi, Xie Liangbo, et al. Indoor Localization of UAV Using Monocular Vision[J]. Journal of Electronics & Information Technology, 2022, 44(3): 906-914.
7	丁文东, 徐德, 刘希龙, 等. 移动机器人视觉里程计综述[J]. 自动化学报, 2018, 44(3): 385-400.
	Ding Wendong, Xu De, Liu Xilong, et al. Review on Visual Odometry for Mobile Robots[J]. Acta Automatica Sinica, 2018, 44(3): 385-400.
8	陈孟元, 张玉坤, 田德红, 等. 基于兴趣倾向机制的仿生SLAM算法[J]. 电子与信息学报, 2022, 44(5): 1743-1753.
	Chen Mengyuan, Zhang Yukun, Tian Dehong, et al. Bionic SLAM Algorithm Based on Interest Tendency Mechanism[J]. Journal of Electronics & Information Technology, 2022, 44(5): 1743-1753.
9	邸凯昌, 万文辉, 赵红颖, 等. 视觉SLAM技术的进展与应用[J]. 测绘学报, 2018, 47(6): 770-779.
	Di Kaichang, Wan Wenhui, Zhao Hongying, et al. Progress and Applications of Visual SLAM[J]. Acta Geodaetica et Cartographica Sinica, 2018, 47(6): 770-779.
10	牛珉玉, 黄宜庆. 基于动态耦合与空间数据关联的RGB-D SLAM算法[J]. 机器人, 2022, 44(3): 333-342.
	Niu Minyu, Huang Yiqing. An RGB-D SLAM Algorithm Based on Dynamic Coupling and Spatial Data Association[J]. Robot, 2022, 44(3): 333-342.
11	林景栋, 吴欣怡, 柴毅, 等. 卷积神经网络结构优化综述[J]. 自动化学报, 2020, 46(1): 24-37.
	Lin Jingdong, Wu Xinyi, Chai Yi, et al. Structure Optimization of Convolutional Neural Networks: A Survey[J]. Acta Automatica Sinica, 2020, 46(1): 24-37.
12	Tateno Keisuke, Tombari Federico, Laina Iro, et al. CNN-SLAM: Real-time Dense Monocular SLAM with Learned Depth Prediction[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2017: 6565-6574.
13	Hou Yi, Zhang Hong, Zhou Shilin. Convolutional Neural Network-based Image Representation for Visual Loop Closure Detection[C]//2015 IEEE International Conference on Information and Automation. Piscataway, NJ, USA: IEEE, 2015: 2238-2245.
14	李维鹏, 张国良, 姚二亮, 等. 基于场景显著区域的改进闭环检测算法[J]. 机器人, 2017, 39(1): 23-35.
	Li Weipeng, Zhang Guoliang, Yao Erliang, et al. An Improved Loop Closure Detection Algorithm Based on Scene Salient Regions[J]. Robot, 2017, 39(1): 23-35.
15	Ganti Pranav, Waslander Steven L. Network Uncertainty Informed Semantic Feature Selection for Visual SLAM[C]//2019 16th Conference on Computer and Robot Vision (CRV). Piscataway, NJ, USA: IEEE, 2019: 121-128.
16	Tang Jiexiong, Ericson Ludvig, Folkesson John, et al. GCNv2: Efficient Correspondence Prediction for Real-time SLAM[J]. IEEE Robotics and Automation Letters, 2019, 4(4): 3505-3512.
17	Li Dongjiang, Shi Xuesong, Long Qiwei, et al. DXSLAM: A Robust and Efficient Visual SLAM System with Deep Features[C]//2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway, NJ, USA: IEEE, 2020: 4958-4965.
18	Lee J B, Rossi R A, Kim S, et al. Attention Models in Graphs: A Survey[J]. ACM Transactions on Knowledge Discovery from Data, 2019, 13(6): 62.
19	Zhuang Chenyi, Ma Qiang. Dual Graph Convolutional Networks for Graph-based Semi-supervised Classification[C]//Proceedings of the 2018 World Wide Web Conference. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee, 2018: 499-508.
20	Zhang Xiwu, Su Yan, Zhu Xinhua. Loop Closure Detection for Visual SLAM Systems Using Convolutional Neural Network[C]//2017 23rd International Conference on Automation and Computing (ICAC). Piscataway, NJ, USA: IEEE, 2017: 1-6.
21	Scarselli Franco, Gori Marco, Ah Chung Tsoi, et al. The Graph Neural Network Model[J]. IEEE Transactions on Neural Networks, 2009, 20(1): 61-80.
22	徐冰冰, 岑科廷, 黄俊杰, 等. 图卷积神经网络综述[J]. 计算机学报, 2020, 43(5): 755-780.
	Xu Bingbing, Cen Keting, Huang Junjie, et al. A Survey on Graph Convolutional Neural Network[J]. Chinese Journal of Computers, 2020, 43(5): 755-780.
23	吴博, 梁循, 张树森, 等. 图神经网络前沿进展与应用[J]. 计算机学报, 2022, 45(1): 35-68.
	Wu Bo, Liang Xun, Zhang Shusen, et al. Advances and Applications in Graph Neural Network[J]. Chinese Journal of Computers, 2022, 45(1): 35-68.
24	Jia Kui, Chan T H, Zeng Zinan, et al. ROML: A Robust Feature Correspondence Approach for Matching Objects in a Set of Images[J]. International Journal of Computer Vision, 2016, 117(2): 173-197.
25	Sarlin Paul-Edouard, DeTone D, Malisiewicz T, et al. SuperGlue: Learning Feature Matching with Graph Neural Networks[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2020: 4937-4946.
26	Elmoogy Ahmed, Dong Xiaodai, Lu Tao, et al. Pose-GNN: Camera Pose Estimation System Using Graph Neural Networks[EB/OL]. (2021-03-17) [2022-12-15]. .
27	Wang Limin, Guo Sheng, Huang Weilin, et al. Places205-VGGNet Models for Scene Recognition[EB/OL]. (2015-08-07) [2022-11-07]. .
28	Meng Zihang, Ravi S N, Singh V. Physarum Powered Differentiable Linear Programming Layers and Applications[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(10): 8939-8949.
29	Lin T Y, Maire M, Belongie S, et al. Microsoft COCO: Common Objects in Context[C]//Computer Vision-ECCV 2014. Cham: Springer International Publishing, 2014: 740-755.
30	Sturm Jürgen, Engelhard Nikolas, Endres Felix, et al. A Benchmark for the Evaluation of RGB-D SLAM Systems[C]//2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway, NJ, USA: IEEE, 2012: 573-580.
31	Geiger A, Lenz P, Stiller C, et al. Vision Meets Robotics: The KITTI Dataset[J]. The International Journal of Robotics Research, 2013, 32(11): 1231-1237.
32	Balntas V, Lenc K, Vedaldi A, et al. HPatches: A Benchmark and Evaluation of Handcrafted and Learned Local Descriptors[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2017: 3852-3861.
33	DeTone D, Malisiewicz T, Rabinovich A. SuperPoint: Self-supervised Interest Point Detection and Description[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway, NJ, USA: IEEE, 2018: 337-33712.

数据集	ORB	Superpoint	GCNv2	本文
i_books	0.572	0.591	0.627	0.693
i_castle	0.583	0.614	0.641	0.716
i_fruits	0.537	0.569	0.599	0.593
i_ski	0.564	0.576	0.598	0.645
i_whitebuilding	0.524	0.513	0.545	0.617

数据集	ORB+FLANN			Superpoint+FLANN			GCNv2+FLANN			本文
数据集	@30°	@60°	@90°	@30°	@60°	@90°	@30°	@60°	@90°	@30°	@60°	@90°
v_abstract	0.643	0.465	0.298	0.620	0.473	0.239	0.728	0.533	0.295	0.802	0.739	0.507
v_bees	0.651	0.482	0.262	0.702	0.536	0.388	0.648	0.499	0.386	0.857	0.686	0.569
v_beyus	0.682	0.503	0.312	0.608	0.449	0.256	0.612	0.428	0.324	0.829	0.737	0.571
v_home	0.605	0.396	0.274	0.633	0.405	0.293	0.730	0.475	0.252	0.796	0.706	0.535
v_woman	0.597	0.391	0.238	0.652	0.433	0.264	0.671	0.416	0.241	0.840	0.744	0.522

数据集	ORB-SLAM2		DX-SLAM		GCNv2-SLAM		本文
数据集	ATE	RPE	ATE	RPE	ATE	RPE	ATE	RPE
fr1/360	―	―	―	―	0.065	0.082	0.145	0.199
fr1/floor	0.390	0.469	―	―	0.138	0.167	0.019	0.038
fr1/desk	0.081	0.101	0.300	0.364	0.188	0.230	0.039	0.050
fr1/desk2	0.571	0.999	0.343	0.416	0.156	0.197	0.047	0.075
fr1/room	0.295	0.354	0.181	0.212	0.246	0.305	0.070	0.111
fr2/desk	0.931	1.149	0.581	0.712	0.160	0.199	0.109	0.133
fr3/long_office_household	1.222	1.553	0.791	1.007	0.363	0.455	0.169	0.176

算法	特征提取/ms	特征匹配/ms	总时间/ms	平均帧率/(帧/s)
ORB-SLAM2	12.88	15.06	27.94	30
DX-SLAM	32.42	39.24	71.66	15
GCNv2-SLAM	19.45	31.57	51.02	20
本文	16.32	30.79	47.11	20

[1]	Lei Xu, Chen Jingyi, Chen Xiaoyang. Research on Path Planning of Warehouse Robot with Improved Harris Hawks Algorithm [J]. Journal of System Simulation, 2024, 36(5): 1081-1092.
[2]	Shi Lanxi, Yan Wenxu, Ni Hongyu, Zhao Feng. Research on Dynamic Scene SLAM Based on Improved Object Detection [J]. Journal of System Simulation, 2024, 36(4): 1028-1042.
[3]	Zhang Rui, Zhou Li, Liu Zhengyang. Dynamic Path Planning for Mobile Robot Based on RRT* and Dynamic Window Approach [J]. Journal of System Simulation, 2024, 36(4): 957-968.
[4]	Liu Fulin, Li Qingxin. Formation Strategy of Hybrid Obstacle Avoidance Algorithm for Multiple Mobile Robots [J]. Journal of System Simulation, 2024, 36(3): 726-734.
[5]	Tang Yunchao, Qi Shaojun, Zhu Lixue, Zhuo Xianrong, Zhang Yunqi, Meng Fan. Obstacle Avoidance Motion in Mobile Robotics [J]. Journal of System Simulation, 2024, 36(1): 1-26.
[6]	Laiyi Yang, Jing Bi, Haitao Yuan. Intelligent Path Planning for Mobile Robots Based on SAC Algorithm [J]. Journal of System Simulation, 2023, 35(8): 1726-1736.
[7]	Li Qian, Tao Ye, Li Hui. Application of Improved Path Tracking Algorithm in Robot SLAM [J]. Journal of System Simulation, 2023, 35(12): 2602-2613.
[8]	Wang Haoyu, Gong Guanghong, Cai Jihong, Ye Bipeng, Zhou Zhaofang, Mei Zheng, Li Ni. Dynamic 3D Scene Perception Based on Battlefield Metaverse [J]. Journal of System Simulation, 2023, 35(10): 2262-2278.
[9]	Tong Wang, Guangtao Shang, Shan Gao. Research on Multi-Robot SLAM Map FusionMethod Based on Heuristics [J]. Journal of System Simulation, 2022, 34(7): 1449-1458.
[10]	Shiqi Lin, Jikai Wang, Haoyuan Pei, Hao Zhao, Zonghai Chen. Monocular Semantic SLAM Method Based on Object Relation Description [J]. Journal of System Simulation, 2022, 34(2): 278-284.
[11]	Yu Yanan, Shi Dunhuang, Hua Chunjie. Adaptive Optimization in Feature-based SLAM Visual Odometry [J]. Journal of System Simulation, 2022, 34(1): 104-112.
[12]	Wang Xiaohan, Zhang Lin, Ren Lei, Xie Kunyu, Wang Kunyu, Ye Fei, Chen Zhen. Brief Review on Applying Reinforcement Learning to Job Shop Scheduling Problems [J]. Journal of System Simulation, 2021, 33(12): 2782-2791.
[13]	You Bo, Wang Mingrui, Li Zhi, Ding Liang. Trajectory Planning of Wheeled Mobile Robot Based on Model Predictive Control [J]. Journal of System Simulation, 2020, 32(4): 591-600.
[14]	Zhao Ping, Lei Xinyu, Chen Bozhi, Yang Jiaoyun. TI-A* Based Multi-robot Dynamic Planning and Coordination Method [J]. Journal of System Simulation, 2019, 31(5): 925-935.
[15]	Wang Gang, Sun Tairen, Ding Shengpei. Locally Weighted Learning Control for Dynamic Restricted Manipulators [J]. Journal of System Simulation, 2019, 31(4): 733-739.