Flipper Control Method for Tracked Robot Based on Deep Reinforcement Learning

doi:10.16182/j.issn1004731x.joss.22-1105

Abstract

Abstract:

Tracked robots with flippers have certain terrain adaptation capabilities. To improve the intelligent operation level of robots in complex environments, it is significant to realize the flipper autonomously control. Combining the expert experience in obstacle crossing and optimization indicators, Markov decision process(MDP) modeling of the robot's flipper control problem is carried out and a simulation training environment based on physics simulation engine Pymunk is built. A deep reinforcement learning control algorithm based on dueling double DQN(D3QN) network is proposed for controlling the flippers. With terrain information and robot state as the input and the four flippers' angle as the output, the algorithm can achieve the self-learning control of the flippers in challenging terrain. The learned flipper control policy is compared with the manual operation in Gazebo 3D simulation environment. The results show that the proposed algorithm can enable the flippers of robot to obtain adaptive adjustment ability, which helps the robot pass complex terrain more efficiently.

Key words: tracked robot, flipper autonomous control, autonomous traversal, DRL, robot operation

CLC Number:

TP242.6

Pan Hainan, Chen Bailiang, Huang Kaihong, Ren Junkai, Cheng Chuang, Lu Huimin, Zhang Hui. Flipper Control Method for Tracked Robot Based on Deep Reinforcement Learning[J]. Journal of System Simulation, 2024, 36(2): 405-414.

Figures/Tables 15

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Table 1

Table 2

Parameters of flipper control algorithm

$ξ 1$	$ξ 2$	$ξ 3$	$ξ 4$	$t m a x$	γ	l_r
0.04	0.1	10	3	400	0.96	0.000 5

Table 2

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Table 3

Fig. 11

Table 4

Indicator comparison between manual and our work

指标	人工操作		本文算法
指标	0.4 m 单台阶	陡峭楼梯	0.4 m 单台阶	陡峭楼梯
t_cost/s	46.17	76.53	28.07	41.21
$θ ̂ R$ /rad	4.02	6.30	3.43	4.53
$θ ̂ R, m a x - θ ̂ R, m i n$ /rad	2.92	3.88	1.11	2.60

Table 4

References 20

1	Liu Jinguo, Wang Yuechao, Li Bin, et al. Current Research, Key Performances and Future Development of Search and Rescue Robots[J]. Frontiers of Mechanical Engineering in China, 2007, 2(4): 404-416.
2	Kruijff G J M, Pirri F, Gianni M, et al. Rescue Robots at Earthquake-hit Mirandola, Italy: A Field Report[C]//2012 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). Piscataway, NJ, USA: IEEE, 2012: 1-8.
3	Kruijff G J M, Janíček M, Keshavdas S, et al. Experience in System Design for Human-robot Teaming in Urban Search and Rescue[M]//Yoshida K, Tadokoro S. Field and Service Robotics: Results of the 8th International Conference. Berlin, Heidelberg: Springer Berlin Heidelberg, 2014: 111-125.
4	商德勇. 薄煤层综采工作面巡检机器人运动分析及试验研究[D]. 北京: 中国矿业大学(北京), 2016.
	Shang Deyong. Study on Motion Analysis and Experiment of the Inspection Robot for Fully-mechanized Workface in Thin Coal Seam[D]. Beijing: China University of Mining & Technology(Beijing), 2016.
5	Ohno K, Morimura S, Tadokoro S, et al. Semi-autonomous Control System of Rescue Crawler Robot Having Flippers for Getting Over Unknown-steps[C]//2007 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway, NJ, USA: IEEE, 2007: 3012-3018.
6	Nagatani K, Yamasaki A, Yoshida K, et al. Semi-autonomous Traversal on Uneven Terrain for a Tracked Vehicle Using Autonomous Control of Active Flippers[C]//2008 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway, NJ, USA: IEEE, 2008: 2667-2672.
7	Okada Y, Nagatani K, Yoshida K, et al. Shared Autonomy System for Tracked Vehicles on Rough Terrain Based on Continuous Three-dimensional Terrain Scanning[J]. Journal of Field Robotics, 2011, 28(6): 875-893.
8	Kober J, Bagnell J A, Peters J, et al. Reinforcement Learning in Robotics: A Survey[J]. The International Journal of Robotics Research, 2013, 32(11): 1238-1274.
9	Zimmermann K, Zuzanek P, Reinstein M, et al. Adaptive Traversability of Unknown Complex Terrain with Obstacles for Mobile Robots[C]//2014 IEEE International Conference on Robotics and Automation (ICRA). Piscataway, NJ, USA: IEEE, 2014: 5177-5182.
10	Paolo G, Tai Lei, Liu Ming. Towards Continuous Control of Flippers for a Multi-terrain Robot Using Deep Reinforcement Learning[J]. (2017-09-25) [2022-09-19]. .
11	Mitriakov A, Papadakis P, Mai Nguyen S, et al. Staircase Traversal via Reinforcement Learning for Active Reconfiguration of Assistive Robots[C]//2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE). Piscataway, NJ, USA: IEEE, 2020: 1-8.
12	Mitriakov A, Papadakis P, Mai Nguyen S, et al. Staircase Negotiation Learning for Articulated Tracked Robots with Varying Degrees of Freedom[C]//2020 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). Piscataway, NJ, USA: IEEE, 2020: 394-400.
13	李允旺, 葛世荣, 朱华, 等. 四履带双摆臂机器人越障机理及越障能力[J]. 机器人, 2010, 32(2): 157-165.
	Li Yunwang, Ge Shirong, Zhu Hua, et al. Obstacle-surmounting Mechanism and Capability of Four-track Robot with Two Swing Arms[J]. Robot, 2010, 32(2): 157-165.
14	Suzuki S, Hasegawa S, Okugawa M. Remote Control System of Disaster Response Robot with Passive Sub-crawlers Considering Falling Down Avoidance[J]. ROBOMECH Journal, 2014, 1(1): 20.
15	Silver D, Singh S, Precup D, et al. Reward Is Enough[J]. Artificial Intelligence, 2021, 299: 103535.
16	Hasselt H. Double Q-learning[C]//Advances in Neural Information Processing Systems. San Francisco, CA, USA: Curran Associates Inc., 2010: 2613-2621.
17	Wang Ziyu, Schaul T, Hessel M, et al. Dueling Network Architectures for Deep Reinforcement Learning[C]//Proceedings of the 33rd International Conference on International Conference on Machine Learning. Cambridge: JMLR, 2016: 1995-2003.
18	Pecka M, Zimmermann K, Svoboda Tomáš. Fast Simulation of Vehicles with Non-deformable Tracks[C]//2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway, NJ, USA: IEEE, 2017: 6414-6419.
19	Zhang Ji, Singh S. LOAM: Lidar Odometry and Mapping in Real-time[C]//Robotics: Science and Systems Conference(RSS). [S.l.]: [s.n.], 2014, 10(7): 1-9.
20	Pecka M, Šalanský Vojtěch, Zimmermann K, et al. Autonomous Flipper Control with Safety Constraints[C]//2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Piscataway, NJ, USA: IEEE, 2016: 2889-2894.

地形场景	越障类型	台阶高度/m	台阶长度/m	台阶数量	楼梯坡度/(°)
0.4 m单台阶	向上/向下	0.4	3.0	1
陡峭楼梯	向上	0.2	0.3	6	33.7
陡峭楼梯	向下	0.2	0.2	6	45.0

[1]	Wang Ziyi, Zhang Kai, Qian Dianwei, Liu Yuzhen. A DRL⁃based Approach for Distributed Equipment Nodes Selection [J]. Journal of System Simulation, 2025, 37(6): 1565-1573.
[2]	Li Min, Zhang Sen, Zeng Xiangguang, Wang Gang, Zhang Tongwei, Xie Dijie, Ren Wenzhe, Zhang Tao. Trajectory Planning of Quadruped Robot Over Obstacle with Single Leg Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(4): 895-909.
[3]	Zhang Sen, Dai Qiangqiang. UAV Path Planning Based on Improved Deep Deterministic Policy Gradients [J]. Journal of System Simulation, 2025, 37(4): 875-881.
[4]	Zhang Bin, Lei Yonglin, Li Qun, Gao Yuan, Chen Yong, Zhu Jiajun, Bao Chenlong. Reinforcement Learning Modeling of Missile Penetration Decision Based on Combat Simulation [J]. Journal of System Simulation, 2025, 37(3): 763-774.
[5]	Wang He, Xu Jianing, Yan Guangyu. Research on Pedestrian Avoidance Strategy for AGV Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(3): 595-606.
[6]	Fei Shuaidi, Cai Changlong, Liu Fei, Chen Minghui, Liu Xiaoming. Research on the Target Allocation Method for Air Defense and Anti-missile Defense of Naval Ships [J]. Journal of System Simulation, 2025, 37(2): 508-516.
[7]	Huang Sijin, Wen Jia, Chen Zheyi. Intelligent Service Migration towards MEC-based IoV Systems [J]. Journal of System Simulation, 2025, 37(2): 379-391.
[8]	Li Chao, Li Jiabao, Ding Caichang, Ye Zhiwei, Zuo Fangwei. Edge Surveillance Task Offloading and Resource Allocation Algorithm Based on DRL [J]. Journal of System Simulation, 2024, 36(9): 2113-2126.
[9]	Wang Hongjun, Lin Junqiang, Zou Xiangjun, Zhang Po, Zhou Mingxuan, Zou Weirui, Tang Yunchao, Luo Lufeng. Construction of a Virtual Interactive System for Orchards Based on Digital Twin [J]. Journal of System Simulation, 2024, 36(6): 1493-1508.
[10]	Zhu Zilu, Liu Yongkui, Zhang Lin, Wang Lihui, Lin Tingyu. Simulation of Robotic Peg-in-hole Assembly Strategy Based on DRL [J]. Journal of System Simulation, 2024, 36(6): 1414-1424.
[11]	Wang Yuan, Xu Lin, Gong Xiaoze, Zhang Yongliang, Wang Yongli. Gradient-based Deep Reinforcement Learning Interpretation Methods [J]. Journal of System Simulation, 2024, 36(5): 1130-1140.
[12]	Wang Xinpeng, Fu Huiqiao, Deng Guizhou, Tang Kaiqiang, Chen Chunlin, Liu Canghai. Research on Motion Planning of Hexapod Robot Based on DRL and Free Gait [J]. Journal of System Simulation, 2024, 36(2): 373-384.
[13]	Zhao Yingying, Dong Pusen, Zhu Tianchen, Li Fan, Su Yun, Tai Zhenying, Sun Qingyun, Fan Hang. Efficiency Optimization Method for Data Sampling in Power Grid Topology Scheduling Simulation [J]. Journal of System Simulation, 2024, 36(2): 283-295.
[14]	An Jing, Si Guangya, Zhang Lei. Strategy Optimization Method of Multi-dimension Projection Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(1): 39-49.
[15]	Wang Yukun, Wang Ze, Dong Liwei, Li Ni. Research on Multi-aircraft Air Combat Behavior Modeling Based on Hierarchical Intelligent Modeling Methods [J]. Journal of System Simulation, 2023, 35(10): 2249-2261.