Overview of Research and Application on Autonomous Vehicle Oriented Perception System Simulation

doi:10.16182/j.issn1004731x.joss.22-FZ0921

Abstract

Abstract:

Following the rapid progress of science and technology, vehicles with autonomous driving or auxiliary driving function enter into vehicle market. However, in the past decade, traffic accidents still occurred frequently, and the safety of these functions become the focus. Simulation technology provides a good platform to test the perception system of autonomous vehicle. Focus on the sensor simulation modeling of autonomous vehicle perception system, from the perspective of single sensor simulation, multi-sensor simulation and classic simulation platform including millimeter wave radar, lidar and camera, the existing research are reviewed, and the shortcomings and development trends of simulation modeling of perception system are summarized. The perception efficiency, travel safety, and the development of autonomous vehicle is promoted.

Key words: perception system, autonomous vehicle, sensors, radar

CLC Number:

TP391.9

Ruoxuan Wang, Jianping Wu, Hui Xu. Overview of Research and Application on Autonomous Vehicle Oriented Perception System Simulation[J]. Journal of System Simulation, 2022, 34(12): 2507-2521.

Figures/Tables 3

Table 1

Some autonomous vehicle accidents in 2013—2021

时间	国家	地点	事故描述
2021-07	美国	高速公路	特斯拉因未能识别出高速上静止的车辆和行人，导致事故车辆撞向高速公路左侧修理爆胎的人，被撞人当场死亡
2021-04	美国	高速公路	启用了AutoPilot自动驾驶系统行驶的特斯拉，由于在转弯处速度过快，导致事故车辆失控并撞向路旁树木引发车体自燃，车内2名乘客死亡
2019-12	美国	高速公路	一辆特斯拉未能在远处检测到静止的消防车并提前作出刹车动作，最终撞上静止的消防车车尾，导致司机死亡
2019-08	美国	高速公路	一辆福特皮卡向右变道，后方的特斯拉高速撞上了福特，导致福特副驾被甩出窗外
2019-03	美国	十字路口	汽车在驶入十字路口时撞上正在左转的货车，卡车车身相对较高，轿车车身相对较矮，毫米波雷达由于高度原因未能检测到卡车，导致轿车司机死亡
2018-04	日本	城市道路	位于特斯拉前方车辆因躲避人群变道避让，特斯拉系统检测到前方车辆离开之后判定需要加速(未能考虑到前方车辆变道的原因)，最终撞上人群导致一人死亡(司机瞌睡)
2018-03	美国	高速公路	特斯拉的Model X在高速公路上行驶时，由于未能判定出公路上的隔离墩，导致Model X直接撞上，司机当场死亡
2018-03	美国	城市道路	优步自动驾驶汽车因未能及时识别前方推着自行车的行人并做出反应，导致行人死亡
2018-01	美国	高速公路	消防车停在道路左肩，一辆特斯拉Model S跟在一辆轿车后在最左侧车道行驶，前方车辆变道之后特斯拉保持在原车道随即自动加速行驶，未预测到前方静止车辆，导致未能及时躲避前方停着的消防车
2017-11	美国	小巷	一辆卡车在倒车进入小巷子时不慎撞到自动驾驶的小型旅游巴士，小型巴士在即将发生碰撞时未有躲避动作
2016-05	美国	十字路口	特斯拉无人驾驶汽车因在强光照射下未能分辨出前方车辆底盘较高的白色货车，导致司机死亡
2016-01	中国	高速公路	中国京港澳高速河北邯郸段，一辆特斯拉无人驾驶汽车因未能分辨出前方与背景颜色相近的车辆，全速撞上了正在工作的道路清洁车辆，导致司机死亡

Table 1

Table 2

Table 3

Simulation platform and its sensor simulation capability

软件/平台	发布时间	功能/优势	传感器仿真能力
CarSim	1996	属于整车动力学仿真软件，主要从整车角度进行仿真。内置相当数量的车辆数学模型，具有丰富的经验参数	与Matlab/Simulink联合仿真，对车辆配置感知传感器
PreScan	2011	可用于从基于模型的控制器设计(MIL)到利用软件在环(SIL)和硬件在环(HIL)系统进行的实时测试等应用。支持和Simulink，ROS，Autoware，python，FMI，C++的联合仿真	支持种类丰富的传感器，例如，V2X传感器，激光雷达、毫米波雷达、超声波雷达，单目和双目相机、鱼眼相机等
PanoSim	2014	具有复杂车辆动力学模型、汽车三维行驶环境模型、汽车行驶交通模型、车载环境传感模型、无线通信模型、GPS和数字地图模型。功能强大，包含Matlab/Simulink接口	基于几何模型与物理建模相结合理念建立了高精度的像机、雷达和无线通信模型
RightHook	2017	拥有一整套的工具链，包括RightWorld、RightWorldHD、RightWorldHIL。RightWorld提供了包含车辆、行人和自行车的确定性的智能交通仿真模型。RightWorldHD可实现对动力学、天气、时间变化的模拟，同时包括丰富的接口	包含摄像头、激光雷达毫米波雷达、IMU和GPS的模拟
CARLA	2017	开源模拟器，具有python接口。提供了用于创建场景的开源数字资源(包括城市布局、建筑以及车辆)以及部分搭建好的自动驾驶测试训练的场景	支持传感器和环境的灵活配置，可调节光照和天气。支持多摄像头、激光雷达、GPS等传感器的仿真
51Sim-One	2018	一体化的自动驾驶仿真与测试平台。该平台基于物理特性进行机理建模，具有高精度和实时仿真的特点	相比其他软件仿真能力更加强大。对于摄像头仿真提供语义分割图、深度图、2D/3等带注释的图像数据集，单目、广角、鱼眼等摄像头的仿真。可提供激光雷达点云原始数据、带标注点云数据等数据。同时也提供目标级毫米波雷达检测物数据

Table 3

References 56

1	Bernsteiner S, Magosi Z, Lindvai-Soos D, et al. Radar Sensor Model for the Virtual Development Process[J]. ATZelektronik Worldwide(S2192-9092), 2015, 10(2): 46-52.
2	Cao P, Wachenfeld W, Winner H. Perception Sensor Modeling for Virtual Validation of Automated Driving[J]. IT-Information Technology(S2196-7032), 2015, 57(4): 243-251.
3	Stolz M, Nestlinger G. Fast Generic Sensor Models for Testing Highly Automated Vehicles in Simulation[J]. Elektrotechnik und Informationstechnik (S0932-383X), 2018, 135(4): 365-369.
4	Muckenhuber S, Holzer H, Rübsam J, et al. Object-Based Sensor Model for Virtual Testing of ADAS/Ad Functions[C]//2019 IEEE International Conference on Connected Vehicles and Expo (ICCVE). Graz, Austria: IEEE, 2019: 1-6.
5	Hanke T, Hirsenkorn N, Dehlink B, et al. Generic Architecture for Simulation of ADAS Sensors[C]// 2015 16th International Radar Symposium (IRS). Dresden: IEEE, 2015: 125-130.
6	Mesow L. Multisensorielle Datensimulation Im Fahrzeugumfeld FüR Die Bewertung Von Sensorfusionsalgorithmen[M]. Germany: Shaker, 2007.
7	Schuler K, Becker D, Wiesbeck W. Extraction of Virtual Scattering Centers of Vehicles by Ray-Tracing Simulations[J]. IEEE Transactions on Antennas and Propagation(S0018-926X), 2008, 56(11): 3543-3551.
8	Hammarstrand L, Svensson L, Sandblom F, et al. Extended Object Tracking Using a Radar Resolution Model[J]. IEEE Transactions on Aerospace and Electronic Systems(S0018-9251), 2012, 48(3): 2371-2386.
9	Hanke T, Schaermann A, Geiger M, et al. Generation and Validation of Virtual Point Cloud Data for Automated Driving Systems[C]//2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). Yokohama: IEEE, 2017: 1-6.
10	Li X, Deng W, Zhang S, et al. Research on Millimeter Wave Radar Simulation Model for Intelligent Vehicle[J]. International Journal of Automotive Technology (S1229-9138), 2020, 21(2): 275-284.
11	Guo J, Deng W, Zhang S, et al. A Novel Method of Radar Modeling for Vehicle Intelligence[J]. SAE International Journal of Passenger Cars-Electronic and Electrical Systems(S1946-4614), 2016, 10(1): 50-56.
12	Zhu B, Sun Y, Zhao J, et al. Millimeter-Wave Radar in-the-Loop Testing for Intelligent Vehicles[J]. IEEE Transactions on Intelligent Transportation Systems (S1524-9050), 2022, 23(8): 11126-11136.
13	Dudek M, Wahl R, Kissinger D, et al. Millimeter Wave FMCW Radar System Simulations Including a 3D Ray Tracing Channel Simulator[C]//2010 Asia-Pacific Microwave Conference Proceedings(APMC). Yokohama: IEEE, 2010: 1665-1668.
14	Schlager B, Muckenhuber S, Schmidt S, et al. State-of-the-Art Sensor Models for Virtual Testing of Advanced Driver Assistance Systems/Autonomous Driving Functions[J]. SAE International Journal of Connected and Automated Vehicles(S2574-075X), 2020, 3(3): 233-261. DOI:10.4271/12-03-03-0018 .
15	Hirsenkorn N, Subkowski P, Hanke T, et al. A Ray Launching Approach for Modeling an FMCW Radar System[C]//2017 18th International Radar Symposium (IRS). Prague: IEEE, 2017: 1-10.
16	Maier F, Makkapati V, Horn M. Environment Perception Simulation for Radar Stimulation in Automated Driving Function Testing[J]. Elektrotechnik und Informationstechnik(S0932-383X), 2018, 135(4): 309-315.
17	Suzuki Y, Sugiura Y, Shimamura T, et al. Model-Based Vehicle Position Estimation Using Millimeter Wave Radar[J]. International Journal of Future Computer and Communication(S2010-3751), 2019, 8(3): 94-98.
18	Song Y, Wang Y, Li Y. Radar Data Simulation Using Deep Generative Networks[J]. The Journal of Engineering(S2051-3305), 2019(20): 6699-6702.
19	Zhao J, Li Y, Zhu B, et al. Method and Applications of Lidar Modeling for Virtual Testing of Intelligent Vehicles[J]. IEEE Transactions on Intelligent Transportation Systems(S1524-9050), 2021, 22(5): 2990-3000.
20	Khatry R, Thompson A. Camera and Lidar Sensor Models for Autonomous Vehicles[R]. Teddington: National Physical Laboratory, 2021.
21	Byeon M, Yoon S W. Analysis of Automotive Lidar Sensor Model Considering Scattering Effects in Regional Rain Environments[J]. IEEE Access(S2169-3536), 2020, 8: 102669-102679.
22	Peinecke N, Lueken T, Korn B R. Lidar Simulation Using Graphics Hardware Acceleration[C]//2008 IEEE/AIAA 27th Digital Avionics Systems Conference. St. Paul, MN: IEEE, 2008.
23	Wang S, Heinrich S, Wang M, et al. Shader-Based Sensor Simulation for Autonomous Car Testing[C]//2012 15th International IEEE Conference on Intelligent Transportation Systems. Anchorage, AK: IEEE, 2012: 224-229.
24	OBrien M E, Fouche D G. Simulation of 3D Laser Radar Systems[J]. Lincoln Laboratory Journal, 2005, 15(1): 37-60.
25	Manivasagam S, Wang S, Wong K, et al. LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Seattle, WA: IEEE, 2020: 11164-11173.
26	Fang J, Zhou D, Yan F, et al. Augmented LiDAR Simulator for Autonomous Driving[J]. IEEE Robotics and Automation Letters(S2377-3766), 2020, 5(2): 1931-1938.
27	Goodin C, Kala R, Carrrillo A, et al. Sensor Modeling for the Virtual Autonomous Navigation Environment[C]//2009 IEEE Sensors Conference. Christchurch: IEEE, 2009: 1588-1592. DOI:10.1109/ICSENS.2009.5398491 .
28	Dosovitskiy A, Ros G, Codevilla F, et al. CARLA: An Open Urban Driving Simulator[C]// Conference on robot learning. California: Proceedings of Machine Learning Research, 2017: 1-16.
29	Shah S, Dey D, Lovett C, et al. AirSim: High-fidelity Visual and Physical Simulation for Autonomous Vehicles[C]//Field and Service Robotics. Zurich: Springer, Cham, 2018: 621-635.
30	Jiang H, Tian Q, Farrell J, et al. Learning the Image Processing Pipeline[J]. IEEE Transaction on Image Processing(S1057-7149), 2017, 26(10): 5032-5042.
31	Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative Adversarial Networks[J]. Communications of the ACM(S1557-7317), 2020, 63(11): 139-144.
32	Sturm P, Ramalingam S, Tardif J P, et al. Camera Models and Fundamental Concepts Used in Geometric Computer Vision[J]. Foundations and Trends in Computer Graphics and Vision(S1572-2759), 2011, 6(1/2): 1-183.
33	Santana-Cedrés D, Gomez L, Alemán Flores M, et al. Estimation of the Lens Distortion Model by Minimizing a Line Reprojection Error[J]. IEEE Sensors Journal(S1530-437X), 2017, 17(9): 2848-2855.
34	薛艺舟. 面向自动驾驶的激光雷达与相机联合标定方法研究[D]. 北京: 北京建筑大学, 2021. DOI:10.26943/d.cnki.gbjzc.2021.000083 .
	Xue Yizhou. Research on the Joint Calibration Method of LiDAR and Camera for Autonomous Driving[D]. Beijing: Beijing University of Civil Engineering and Architecture, 2021. DOI:10.26943/d.cnki.gbjzc.2021.000083 .
35	张姗. 智能驾驶车辆环境感知系统建模与仿真[D]. 长春: 长春大学, 2021. DOI:10.27912/d.cnki.gcdcx.2021.000106 .
	Zhang Shan. Modeling and Simulation of Environment Awareness System for Intelligent Driving Vehicles[D]. Changchun: Changchun University, 2021. DOI:10.27912/d.cnki.gcdcx.2021.000106 .
36	王瑞. 基于车载的多相机联合标定[D]. 西安: 西安电子科技大学, 2017.
	Wang Rui. Multi-Camer a Joint Calibration Based on Vehicle[D]. Xiʹan: Xidian University, 2017.
37	舒娜. 摄像机标定方法的研究[D]. 南京: 南京理工大学, 2014.
	Shu Na. Research on Camera Calibration Method[D]. Nanjing: Nanjing University of Science and Technology, 2014.
38	Elmquist A, Negrut D. Modeling Cameras for Autonomous Vehicle and Robot Simulation: An Overview[J]. IEEE Sensors Journal(S1530-437X), 2021, 21(22): 25547-25560.
39	Walch A, Luksch C, Szabo A, et al. Lens Flare Prediction Based on Measurements with Real-time Visualization[J] The Visual Computer(S0178-2789), 2018, 34(9): 1155-1164.
40	Zheng Q, Zheng C. NeuroLens: Data-driven Camera Lens Simulation Using Neural Networks[J]. Computer Graphics Forum(S1467-8659), 2017, 36(8): 390-401.
41	Bisagno N, Conci N. Virtual Camera Nodeling for Multi-view Simulation of Surveillance Scenes[C]//2018 26th European Signal Processing Conference (EUSIPCO). Rome: IEEE, 2018: 2170-2174.
42	Pharr M, Jakob W, Humphreys G. Physically Based Rendering: From Theory to Implementation[M]. San Mateo, CA, USA: Morgan Kaufmann, 2016.
43	Afifi M, Brown M S. Deep White-balance Editing[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA: IEEE, 2020: 1397-1406.
44	Liu C, Szeliski R, Kang S B, et al. Automatic Estimation and Removal of Noise from a Single Image[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence(S0162-8828), 2008, 30(2): 299-314.
45	Rosenberger P, Holder M, Huch S, et al. Benchmarking and Functional Decomposition of Automotive Lidar Sensor Models[C]//2019 IEEE Intelligent Vehicles Symposium (IV). Paris: IEEE, 2019: 632-639.
46	齐翔. 基于多传感器信息融合技术的汽车行驶状态估计[D]. 镇江: 江苏大学, 2019.
	Qi Xiang. State Estimation of Vehicle Based on Multi-sensors Information Fusion[D]. Zhenjiang: Jiangsu University, 2019.
47	刘敏. 基于自动驾驶车载毫米波雷达间相互干扰的研究[D]. 重庆: 重庆理工大学, 2021.
	Liu Min. Research on Mutual Interference between Millimeter Wave Radars for Autonomous Vehicles[D]. Chongqing: Chongqing University of Technology, 2021.
48	毛雪松, 雷闰龙, 黄少伟, 等. 道路环境中激光雷达互干扰的仿真研究[J]. 系统仿真学报, 2022, 34(4): 891-900.
	Mao Xuesong, Lei Runlong, Huang Shaowei, et al. Simulation on Mutual Interference of Laser Radar in Road Environments[J]. Journal of System Simulation, 2022, 34(4): 891-900.
49	刘晶郁, 马辉, 张学文, 等. 自动驾驶虚拟仿真测试技术研究进展[J]. 中国科技论文, 2021, 16(6): 571-577, 584.
	Liu Jingyu, Ma Hui, Zhang Xuewen, et al. Research and Development of Autonomous Driving Virtual Simulation Technology[J]. China Science Paper, 2021, 16(6): 571-577, 584.
50	清华大学－剑桥大学－麻省理工学院低碳能源大学联盟未来交通研究中心. 交通仿真软件FLOWSIM[EB/OL]. (2021-04-03)[2021-06-01]. .
	Tsinghua University-University of Cambridge-Massachusetts Institute of Technology Low Carbon Energy University Alliance Future Transport Research Center. Traffic Simulation Software: FLOWSIM[EB/OL]. (2021-04-03)[2021-06-01]. .
51	毕锟, 范英, 候峙朴, 等. 基于Carsim和Simulink联合仿真的汽车操纵稳定性评价[J]. 河北工程大学学报(自然科学版), 2022, 39(2): 106-112.
	Bi Kun, Fan Ying, Hou Zhipu, et al. Vehicle Handling Stability Evaluation Based on CarSim and Simulink Co-Simulation[J]. Journal of Hebei University of Engineering (Natural Science Edition), 2022, 39(2): 106-112.
52	朱鹏飞, 梁威, 丁文政, 等. 基于PreScan的自动泊车仿真研究[J]. 计算机与数字工程, 2021, 49(1): 122-125.
	Zhu Pengfei, Liang Wei, Ding Wenzheng, et al. Research on Automatic Parking Simulation Based on PreScan[J]. Computer & Digital Engineering, 2021, 49(1): 122-125.
53	郭路兵, 梅志千, 贺勇. 基于PreScan软件的交叉路口防碰撞控制系统仿真[J]. 机械与电子, 2014(2): 22-26.
	Guo Lubing, Mei Zhiqian, He Yong. Simulation of Crossroads Anti‑collision Control System Based on PreScan[J]. Machinery and Electronics, 2014(2): 22-26.
54	王成康, 黄李波. 基于CARLA的驾驶仿真平台搭建[J]. 佳木斯大学学报(自然科学版), 2021, 39(3): 27-29.
	Wang Chengkang, Huang Libo. Construction of Driving Simulation Platform Based on CARLA[J]. Journal of Jiamusi University(Natural Science Edition), 2021, 39(3): 27-29.
55	Brekke Å, Vatsendvik F, Lindseth F. Multimodal 3D Object Detection from Simulated Pretraining[C]//Symposium of the Norwegian AI Society. Trondheim: Springer, Cham, 2019: 102-113.
56	Zapridou E, Bartocci E, Katsaros P. Runtime Verification of Autonomous Driving System in CARLA[C]//International Conference on Runtime Verification. Los Angeles, CA: Springer, Cham, 2020: 172-183.

畸变种类	产生原因
径向畸变	相机组装过程中，透镜与芯片不平行引起光心与几何中心不一致造成的畸变
切向畸变	由于制作工艺缺陷引起透镜形状改变造成的畸变
色差	波长相关折射引起的颜色位置偏移
渐晕	透镜自身原因，光线离轴越远，光强度越弱
光斑	强光的多径反射产生的鬼影
景深模糊	超出镜头纵深清晰范围的部分出现模糊，受光圈、焦距、物距影响

[1]	Fan Yu, Dongqi Luo, Binqiang Si, Jihong Zhu. Phased Array Radar DOA Estimation Simulation System Designed with Interactive Control [J]. Journal of System Simulation, 2022, 34(5): 964-977.
[2]	Xu Kang, Xiaofeng Zhang. Radar Remote Sensing Data Augmentation Method Based on Generative Adversarial Network [J]. Journal of System Simulation, 2022, 34(4): 920-927.
[3]	Jing Wu, Zhiming Xu, Xiaofeng Ai, Feng Zhao, Shunping Xiao. Component Design and Simulation of Netted Radar Fusion Processing [J]. Journal of System Simulation, 2022, 34(2): 409-418.
[4]	Jiaji Lou, Jing Ma, Xiaowei Li, Yue Zhao, Youbin Song. Research on Modeling and Simulation Technology of Microwave Radar High Precision Tracking Loop [J]. Journal of System Simulation, 2022, 34(12): 2619-2628.
[5]	Yuan Fei, Li Jianhong, Yin Hao, Wang Yuhao, Hong Sheng. A Fast Simulation Method for Ship Target SAR Signal Echo [J]. Journal of System Simulation, 2022, 34(1): 11-19.
[6]	Wu Xiaolong, Xia Fugen, Chen Jing, Xu Jia. Autonomous Vehicle Path Tracking Control System Based on Energy Optimization [J]. Journal of System Simulation, 2022, 34(1): 163-169.
[7]	Huang Xingyuan, Li Yanyi. The Allocation of Jamming Resources Based on Double Q-learning Algorithm [J]. Journal of System Simulation, 2021, 33(8): 1801-1808.
[8]	Feng Xinyang, Shao Chao. SAR Image Target Recognition Based on Across Convolution Network Feature Fusion [J]. Journal of System Simulation, 2021, 33(3): 554-561.
[9]	Diao Guijie, Ni Hong, Yang Liang, Gong Ningbo, Guo Jiao. Study on the Scattering Echo Signal Simulation Method of Sea Scene for Wide-Band Phased Array Radar [J]. Journal of System Simulation, 2021, 33(2): 401-408.
[10]	Yin Zhishuai, He Jiaxiong, Nie Linzhen, Guan Jiayi. Longitudinal Adaptive Control of Autonomous Vehicles Base on Optimization Algorithm [J]. Journal of System Simulation, 2021, 33(2): 409-420.
[11]	Huang Yingkun, Jin Weidong, Yan Kang, Zhu Jiehao. Radar Emitter Signal Identification Via Distance Features [J]. Journal of System Simulation, 2021, 33(12): 2959-2966.
[12]	Wang Rui, Li Xiangyang, Zhang Zhili, Ma Hongguang. Track Deception Modeling Method Based on Multi-jammer Cooperation [J]. Journal of System Simulation, 2021, 33(12): 2992-3000.
[13]	Liu Shengli, Wang Gang. Threat Assessment for the Defense Penetration Paths of Air Strike Aircrafts Based on Radar Chart [J]. Journal of System Simulation, 2021, 33(1): 196-204.
[14]	Suo Yongfeng, Yang Shenhua, Chen Guoquan, Xiang Zhuqiang. Maritime Radar Terrain Echo Simulation Based on DEM and SAR [J]. Journal of System Simulation, 2020, 32(3): 518-524.
[15]	Zhou Chunhui, Zhou Ling, Gan Langxiong, Cheng Xiaodong, Gao Junjie. Shielding Modeling and Simulation of Coastal Building to VTS Radar [J]. Journal of System Simulation, 2020, 32(2): 306-313.