Research on Dynamic Scene SLAM Based on Improved Object Detection

doi:10.16182/j.issn1004731x.joss.22-1332

Abstract

Abstract:

Aiming at the epipolar constraint matching problem of monocular SLAM in dynamic scenes adynamic feature point selection method based on object detection is proposed, in which the dynamic feature points in the front-end image frame of SLAM system is eliminated during feature extraction to improve the localization accuracy of SLAM. An improved target detection network is proposed to construct a loss function to describe the bounding box by using the overlap area, distance similarity and cosine similarity, which can achieve the accurate localization of target objects and obtain the range of object feature points in the current image frame. The object category is judged in SLAM, and the dynamic feature points in the front-end image frame are rejected according to the target detection result for the objects marked as dynamic. Based on the static feature point results, the epipolar geometry is used for the feature matching between two frames to estimate pose the to carry out the tracking, map building and closed-loop detection of monocular camera motion. The speed of the inference process is improved by the structural reparameterization of the backbone of target detection network to ensure the real-time operation of the overall system. Experimental results on KITTI dataset show that the improved system improves the localization accuracy by 23.4% over ORB-SLAM3 system, and the frame rate can reach more than 30fps. The algorithm can effectively improve the localization accuracy of monocular SLAM system in dynamic scenes under the condition of ensuring the real-time operation.

Key words: visual SLAM, epipolar geometry, feature matching, object detection, IoU loss function, structural reparameterization

CLC Number:

TP391.9

Shi Lanxi, Yan Wenxu, Ni Hongyu, Zhao Feng. Research on Dynamic Scene SLAM Based on Improved Object Detection[J]. Journal of System Simulation, 2024, 36(4): 1028-1042.

Figures/Tables 29

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Table 1

Fig. 12

Fig. 13

Fig. 14

Fig. 15

Fig. 16

Fig. 17

Table 2

Fig. 18

Fig. 19

Fig. 20

Fig. 21

Table 3

Fig. 22

Fig. 23

Table 5

Table 6

References 24

1	Davison A J, Reid I D, Molton N D, et al. MonoSLAM: Real-time Single Camera SLAM[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 1052-1067.
2	张良桥, 陈国良, 许晓东, 等. 一种用于图像特征提取的改进ORB-SLAM算法[J]. 测绘通报, 2019(3): 16-20.
	Zhang Liangqiao, Chen Guoliang, Xu Xiaodong, et al. An Improved ORB-SLAM Algorithm for Feature Extraction[J]. Bulletin of Surveying and Mapping, 2019(3): 16-20.
3	Zhou Lipu, Wang Shengze, Kaess M. DPLVO: Direct Point-line Monocular Visual Odometry[J]. IEEE Robotics and Automation Letters, 2021, 6(4): 7113-7120.
4	Ban Xicheng, Wang Hongjian, Chen Tao, et al. Monocular Visual Odometry Based on Depth and Optical Flow Using Deep Learning[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-19.
5	Li Jinquan, Pei Ling, Zou Danping, et al. Attention-SLAM: A Visual Monocular SLAM Learning from Human Gaze[J]. IEEE Sensors Journal, 2021, 21(5): 6408-6420.
6	Lee Sehyung, Lim Jongwoo, Il Hong Suh. Progressive Feature Matching: Incremental Graph Construction and Optimization[J]. IEEE Transactions on Image Processing, 2020, 29: 6992-7005.
7	胡立华, 左威健, 张继福. 采用逆近邻与影响空间的图像特征误匹配剔除方法[J]. 计算机辅助设计与图形学学报, 2022, 34(3): 449-458.
	Hu Lihua, Zuo Weijian, Zhang Jifu. A Mismatch Elimination Method Based on Reverse Nearest Neighborhood and Influence Space[J]. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(3): 449-458.
8	任彬, 宋海丽, 赵增旭, 等. 基于RANSAC的视觉里程计优化方法研究[J]. 仪器仪表学报, 2022, 43(6): 205-212.
	Ren Bin, Song Haili, Zhao Zengxu, et al. Study on Optimization Method of Visual Odometry Based on RANSAC[J]. Chinese Journal of Scientific Instrument, 2022, 43(6): 205-212.
9	盛超, 潘树国, 赵涛, 等. 基于图像语义分割的动态场景下的单目SLAM算法[J]. 测绘通报, 2020(1): 40-44.
	Sheng Chao, Pan Shuguo, Zhao Tao, et al. Monocular SLAM System in Dynamic Scenes Based on Image Semantic Segmentation[J]. Bulletin of Surveying and Mapping, 2020(1): 40-44.
10	Chang Jianfang, Dong Na, Li Donghui. A Real-time Dynamic Object Segmentation Framework for SLAM System in Dynamic Scenes[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-9.
11	刘瑞军, 王向上, 张晨, 等. 基于深度学习的视觉SLAM综述[J]. 系统仿真学报, 2020, 32(7): 1244-1256.
	Liu Ruijun, Wang Xiangshang, Zhang Chen, et al. A Survey on Visual SLAM Based on Deep Learning[J]. Journal of System Simulation, 2020, 32(7): 1244-1256.
12	张翠文, 张长伦, 何强, 等. 目标检测中框回归损失函数的研究[J]. 计算机工程与应用, 2021, 57(20): 97-103.
	Zhang Cuiwen, Zhang Changlun, He Qiang, et al. Research on Loss Function of Box Regression in Object Detection[J]. Computer Engineering and Applications, 2021, 57(20): 97-103.
13	Rezatofighi H, Tsoi N, Gwak J Y, et al. Generalized Intersection over Union: A Metric and a Loss for Bounding Box Regression[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2019: 658-666.
14	Zheng Zhaohui, Wang Ping, Ren Dongwei, et al. Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation[J]. IEEE Transactions on Cybernetics, 2022, 52(8): 8574-8586.
15	Xue Hongtao, Wu Meng, Zhang Ziming, et al. Intelligent Diagnosis of Mechanical Faults of In-wheel Motor Based on Improved Artificial Hydrocarbon Networks[J]. ISA Transactions, 2022, 120: 360-371.
16	Zoph B, Vasudevan V, Shlens J, et al. Learning Transferable Architectures for Scalable Image Recognition[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2018: 8697-8710.
17	Ding Xiaohan, Zhang Xiangyu, Ma Ningning, et al. RepVGG: Making VGG-style ConvNets Great Again[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2021: 13728-13737.
18	Campos Carlos, Elvira Richard, J Gómez Rodríguez Juan, et al. ORB-SLAM3: An Accurate Open-source Library for Visual, Visual-inertial, and Multimap SLAM[J]. IEEE Transactions on Robotics, 2021, 37(6): 1874-1890.
19	艾青林, 刘刚江, 徐巧宁. 动态环境下基于改进几何与运动约束的机器人RGB-D SLAM算法[J]. 机器人, 2021, 43(2): 167-176.
	Ai Qinglin, Liu Gangjiang, Xu Qiaoning. An RGB-D SLAM Algorithm for Robot Based on the Improved Geometric and Motion Constraints in Dynamic Environment[J]. Robot, 2021, 43(2): 167-176.
20	Shao Chunyan, Zhang Chi, Fang Zaojun, et al. A Deep Learning-based Semantic Filter for RANSAC-based Fundamental Matrix Calculation and the ORB-SLAM System[J]. IEEE Access, 2020, 8: 3212-3223.
21	João Carlos Virgolino Soares, Gattass Marcelo, Marco Antonio Meggiolaro. Visual SLAM in Human Populated Environments: Exploring the Trade-off between Accuracy and Speed of YOLO and Mask R-CNN[C]//2019 19th International Conference on Advanced Robotics (ICAR). Piscataway, NJ, USA: IEEE, 2019: 135-140.
22	Bescos Berta, Fácil José M, Civera Javier, et al. DynaSLAM: Tracking, Mapping, and Inpainting in Dynamic Scenes[J]. IEEE Robotics and Automation Letters, 2018, 3(4): 4076-4083.
23	Geiger A, Lenz P, Stiller C, et al. Vision Meets Robotics: The KITTI Dataset[J]. The International Journal of Robotics Research, 2013, 32(11): 1231-1237.
24	Wang Ke, Cao Chuan, Ma Sai, et al. An Optimization-Based Multi-sensor Fusion Approach Towards Global Drift-free Motion Estimation[J]. IEEE Sensors Journal, 2021, 21(10): 12228-12235.

损失函数	准确率	召回率	mAP@.5	mAP@.5:.95
L_CIoU	0.952	0.893	0.957	0.837
L_New-IoU	0.956	0.980	0.974	0.856

序列	平均值			标准差			均方根误差
序列	ORB-SLAM3	文献[24]	本文	ORB-SLAM3	文献[24]	本文	ORB-SLAM3	文献[24]	本文
00	0.513	13.238	0.396	0.286	5.222	0.245	0.587	15.165	0.466
01	―	6.505	2.021	―	2.631	1.407	―	7.017	2.462
02	0.735	9.155	0.727	0.535	6.015	0.618	0.907	10.594	0.954
03	0.042	0.865	0.025	0.042	0.267	0.019	0.059	0.906	0.032
04	0.043	0.247	0.006	0.022	0.126	0.003	0.048	0.277	0.007
05	0.164	5.997	0.185	0.056	3.093	0.075	0.173	6.748	0.200
06	0.513	3.157	0.434	0.286	1.786	0.259	0.587	3.627	0.505
07	0.229	1.353	0.175	0.114	1.515	0.101	0.256	2.484	0.212
08	4.770	4.913	3.758	3.743	3.432	2.723	6.063	5.993	4.641
09	0.608	4.866	0.287	0.570	3.953	0.126	0.834	6.269	0.314
10	0.293	3.158	0.379	0.295	1.778	0.331	0.474	3.753	0.503

序列	均方根误差/%			旋转误差/((˚)/m)
序列	ORB-SLAM3	文献[24]	本文	ORB-SLAM3	文献[24]	本文
00	1.334	1.791	1.468	0.767	0.216	0.474
01	―	3.632	0.958	―	0.064	2.352
02	0.692	1.207	0.166	0.272	0.158	0.302
03	0.591	1.174	0.403	0.107	0.058	0.169
04	0.669	0.617	0.552	0.147	0.046	0.14
05	1.022	0.988	1.347	0.328	0.077	0.497
06	2.601	0.797	1.607	0.229	0.057	0.193
07	1.122	0.812	1.126	0.191	0.075	0.182
08	10.855	1.479	6.392	0.184	0.072	0.19
09	4.113	1.612	1.041	0.282	0.067	0.246
10	2.022	1.318	1.504	0.183	0.083	0.186

序列	平均值			标准差			均方根误差
序列	ORB-SLAM3	O+Y	本文方法	ORB-SLAM3	O+Y	本文方法	ORB-SLAM3	O+Y	本文方法
04	0.043	0.034	0.006	0.022	0.02	0.003	0.048	0.039	0.007
07	0.229	0.377	0.175	0.114	0.237	0.101	0.256	0.446	0.212

序列	ORB-SLAM3		本文算法
序列	中位数	平均数	中位数	平均数
平均值	24.5	26.6	29.8	30.9
03	24.7	26.6	28.9	30.2
06	24.4	26.5	29.9	31.4
07	24.5	26.7	30.5	31.1