Dynamic 3D Scene Perception Based on Battlefield Metaverse

doi:10.16182/j.issn1004731x.joss.23-FZ0815

Abstract

Abstract:

Informatization combat needs higher requirements for battlefield situational awareness, and the use of unmanned intelligences to conduct battlefield reconnaissance and perceive target information is particularly important. Facing the needs of complex dynamic environment localization and target recognition, a dynamic 3D scene perception system is proposed and constructed based on battlefield meta-universe target data and operational environment,which uses vision and IMU fusion sensor simulation data as inputs, extracts battlefield target information through instance segmentation and dense optical flow estimation network and uses it as a scene prior, and synchronizes the position estimation of unmanned intelligences in the battlefield with the tracking of battlefield targets during SLAM and optimization process. The experimental results show that the system is able to provide the sparse reconstruction of the scene and accurate localization of the unmanned intelligent body under the dynamic and complex environment of the battlefield, and continuously outputs the information of each battlefield target such as type, position, and speed, which shows the good accuracy and robustness, and provides technical support for the battlefield situational fusion and other modules.

Key words: situation awareness, dynamic scenes, SLAM, instance segmentation, battlefield metaverse

CLC Number:

TP391.9

Wang Haoyu, Gong Guanghong, Cai Jihong, Ye Bipeng, Zhou Zhaofang, Mei Zheng, Li Ni. Dynamic 3D Scene Perception Based on Battlefield Metaverse[J]. Journal of System Simulation, 2023, 35(10): 2262-2278.

Figures/Tables 22

Fig. 1

Fig. 2

Table 1

Fig. 3

Fig. 4

Table 2

Table 3

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Table 4

Fig. 14

Fig. 15

Fig. 16

Table 5

Table 6

References 26

1	贺玲, 贺照辉. 大数据技术在战场态势感知中的应用[J]. 科技与创新, 2023(7): 178-180.
2	郑锦伟. 复杂三维场景感知及可视化处理技术[D]. 西安: 西安电子科技大学, 2022.
	Zheng Jinwei. Complex 3D Scene Perception and Visualization Processing Technology[D]. Xi'an: Xidian University, 2022.
3	牛斌, 徐伟伟. 陆军智能化有人/无人协同战场侦察[J]. 国防科技, 2019, 40(6): 24-27.
	Niu Bin, Xu Weiwei. Army Intelligent Manned/Unmanned Joint Battlefield Reconnaissance Analysis[J]. National Defense Technology, 2019, 40(6): 24-27.
4	刘书雷, 徐海洋. 无人机在军事行动中的作用评析[J]. 国际社会科学杂志(中文版), 2023, 40(1): 183-186, 7, 12.
	Liu Shulei, Xu Haiyang. An Analysis of the Function of Drones in Military Actions[J]. International Journal of Social Sciences(Chinese Edition), 2023, 40(1): 183-186, 7, 12.
5	刘康. 复杂场景下地面小样本军事目标识别算法研究[D]. 西安: 西安工业大学, 2023.
	Liu Kang. Research on Small Sample Military Target Recognition Algorithm in Complex Scene[D]. Xi'an: Xi'an Technological University, 2023.
6	张志铭, 张军, 刘元盛, 等. 动态环境下的实时定位与建图系统综述[C]//中国计算机用户协会网络应用分会2022年第二十六届网络新技术与应用年会论文集. 北京: 中国计算机用户协会, 2022: 43-47.
7	Mur-Artal Raúl, Tardós Juan D. ORB-SLAM2: An Open-source SLAM System for Monocular, Stereo, and RGB-D Cameras[J]. IEEE Transactions on Robotics, 2017, 33(5): 1255-1262.
8	杭程. 面向动态场景的视觉SLAM系统研究[D]. 南京: 南京邮电大学, 2022.
	Hang Cheng. Research on Visual SLAM Systems for Dynamic Scenes[D]. Nanjing: Nanjing University of Posts and Telecommunications, 2022.
9	Girshick R. Fast R-CNN[C]//2015 IEEE International Conference on Computer Vision (ICCV). Piscataway, NJ, USA: IEEE, 2015: 1440-1448.
10	李一. 基于Unity3D的虚拟战场环境研究与实现[D]. 石家庄: 河北科技大学, 2018.
	Li Yi. Research and Implementation of Virtual Battlefield Environment Based on Unity3D[D]. Shijiazhuang: Hebei University of Science and Technology, 2018.
11	Wu Lingda, Hao Hongxing, Yang Chao, et al. Research on Visualization Techniques in Large Scale Virtual Battlefield[C]//2016 IEEE First International Conference on Data Science in Cyberspace (DSC). Piscataway, NJ, USA: IEEE, 2016: 452-455.
12	张昭, 郭玉杰, 赵晓宁, 等. 军事元宇宙刍议与展望[J]. 系统仿真学报, 2023, 35(7): 1421-1437.
	Zhang Zhao, Guo Yujie, Zhao Xiaoning, et al. Military Metaverse: Key Technologies, Potential Applications and Future Directions[J]. Journal of System Simulation, 2023, 35(7): 1421-1437.
13	杨新颖, 龚光红, 王行仁. 飞行模拟器中的虚拟声音建模与实时渲染技术[J]. 航空学报, 2009, 30(7): 1305-1311.
	Yang Xinying, Gong Guanghong, Wang Xingren. Virtual Sound Modeling and Real-time Rendering in Aircraft Simulator[J]. Acta Aeronautica et Astronautica Sinica, 2009, 30(7): 1305-1311.
14	赵坦, 吴琳, 陶九阳, 等. 元宇宙概念及其军事运用[J]. 系统仿真学报, 2023, 35(7): 1405-1420.
	Zhao Tan, Wu Lin, Tao Jiuyang, et al. Metaverse Concept and Its Military Application[J]. Journal of System Simulation, 2023, 35(7): 1405-1420.
15	He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep Residual Learning for Image Recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2016: 770-778.
16	Lin T Y, Dollár Piotr, Girshick R, et al. Feature Pyramid Networks for Object Detection[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2017: 936-944.
17	He Kaiming, Gkioxari G, Dollár Piotr, et al. Mask R-CNN[C]//2017 IEEE International Conference on Computer Vision (ICCV). Piscataway, NJ, USA: IEEE, 2017: 2980-2988.
18	Sun Deqing, Yang Xiaodong, Liu Mingyu, et al. PWC-net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2018: 8934-8943.
19	Lin T Y, Maire M, Belongie S, et al. Microsoft COCO: Common Objects in Context[C]//Computer Vision-ECCV 2014. Cham: Springer International Publishing, 2014: 740-755.
20	李月. 基于光流法和语义分割的视觉SLAM系统研究[D]. 沈阳: 沈阳工业大学, 2022.
	Li Yue. Research on Visual SLAM System Based on Optical Flow Method and Semantic Segmentation[D]. Shenyang: Shenyang University of Technology, 2022.
21	Sun Deqing, Yang Xiaodong, Liu Mingyu, et al. Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(6): 1408-1423.
22	Zhang Jun, Henein M, Mahony R, et al. VDO-SLAM: A Visual Dynamic Object-aware SLAM System[EB/OL]. (2021-12-14) [2022-12-23]. .
23	Viswanathan D G. Features From Accelerated Segment Test (FAST)[EB/OL]. [2022-12-20]. .
24	Calonder M, Lepetit V, Strecha C, et al. BRIEF: Binary Robust Independent Elementary Features[C]//Computer Vision-ECCV 2010. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010: 778-792.
25	Fischler M A, Bolles R C. Random Sample Consensus: a Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography[J]. Communications of the ACM, 1981, 24(6): 381-395.
26	刘春, 马颖. 改进卡尔曼滤波在北斗伪距定位中的研究[J]. 电子测量与仪器学报, 2016, 30(5): 779-785.
	Liu Chun, Ma Ying. Research on Improved Kalman Filter in Beidou Pseudo Ranges Positioning[J]. Journal of Electronic Measurement and Instrumentation, 2016, 30(5): 779-785.

实例命名	类型	是否伪装
APC_Amb	装甲运输车	否
APC	装甲运输车	是
Tank_White	坦克	否
Tank_Green	坦克	是
DF	导弹发射车	是
Helicopter	直升机	否
UAV	作战无人机	否
Army_Jeep	吉普车	是
MRAP	防雷多功能车	是
MTruck	军用卡车	是

文件夹	内容
Mask	json_to_dataset生成的.png格式label文件
Json	labelme生成的json文件
Labelme	json_to_dataset生成的文件夹
Pic	尺寸标准化之后的原图

参数	参数含义	设定值
RPN_ANCHOR_SCALES	特征层锚框大小	(16, 32, 64, 128, 256)
BACKBONE_STRIDES	特征层下采样倍数	[4, 8, 16, 32, 64]
WEIGHT_DECAY	权值衰减	0.000 1
LEARNING_RATE	学习率	0.000 05
LEARNING_MOMENTUM	学习动量	0.9
EPOCHS	训练轮数	500

相机参数	标定值	相机参数	标定值
相机水平焦距/px	533.333 3	基线距离/mm	120
相机垂直焦距/px	533.333 3	帧率/(帧/s)	10
相机光心坐标	(320, 240)

指标	智能体	目标1	目标2	目标3	目标4
RPE_t/m	0.235 5	0.199 3	0.303 8	0.225 7	0.184 9
跟踪帧	350	87	110	52	23