Research on Motion Planning of Hexapod Robot Based on DRL and Free Gait

doi:10.16182/j.issn1004731x.joss.22-1220

Abstract

Abstract:

To improve the passability and the motion performance of the hexapod robot in the unstructured environment, a multi-contact motion planning algorithm based on DRL and free gait planner is proposed. Firstly, the free gait planner obtains the reachable footholds under the target state and outputs the optimal gait sequence. The center of mass motion policy of the hexapod robot in the randomly generated plum blossom pile environment is obtained by using deep reinforcement learning training. To ensure the reachability between adjacent states of the robot in motion, the state transition feasibility model is used to judge the state transition feasibility. Finally, the foothold planning of the hexapod robot in the plum blossom pile environment with gullies of different widths is realized. Simulation and physical experiments show that the multi-contact motion planning algorithm can make the robot reach the target area quickly and smoothly from the starting point, and automatically adjust the gait pattern to deal with the randomly distributed plum blossom piles in different environments.

Key words: hexapod robot, free gait, DRL, multi-contact motion planning, unstructured environment

CLC Number:

TP242.6

Wang Xinpeng, Fu Huiqiao, Deng Guizhou, Tang Kaiqiang, Chen Chunlin, Liu Canghai. Research on Motion Planning of Hexapod Robot Based on DRL and Free Gait[J]. Journal of System Simulation, 2024, 36(2): 373-384.

Figures/Tables 18

Fig. 1

Table 1

Table 2

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Table 3

Fig. 8

Fig. 9

Table 4

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Fig.14

References 23

1	Wu Jianxu, Yang Hui, Li Ruiming, et al. Design and Analysis of a Novel Octopod Platform with a Reconfigurable Trunk[J]. Mechanism and Machine Theory, 2021, 156: 104134.
2	石琳, 罗庆生, 韩宝玲, 等. 仿生六足机器人稳定性的仿生实验研究[J]. 系统仿真学报, 2008, 20(19): 5384-5387.
	Shi Lin, Luo Qingsheng, Han Baoling, et al. Research on Biomimetic Experiment of Stability of Hexapod Robot[J]. Journal of System Simulation, 2008, 20(19): 5384-5387.
3	Hwangbo Jemin, Lee Joonho, Dosovitskiy Alexey, et al. Learning Agile and Dynamic Motor Skills for Legged Robots[J]. Science Robotics, 2019, 4(26): 5872.
4	Stelzer Annett, Hirschmüller Heiko, Görner Martin. Stereo-vision-based Navigation of a Six-legged Walking Robot in Unknown Rough Terrain[J]. The International Journal of Robotics Research, 2012, 31(4): 381-402.
5	李满宏, 张明路, 张建华, 等. 六足机器人关键技术综述[J]. 机械设计, 2015, 32(10): 1-8.
	Li Manhong, Zhang Minglu, Zhang Jianhua, et al. Review on Key Technology of the Hexapod Robot[J]. Journal of Machine Design, 2015, 32(10): 1-8.
6	Belter Dominik, Wietrzykowski Jan, Skrzypczyński Piotr. Employing Natural Terrain Semantics in Motion Planning for a Multi-legged Robot[J]. Journal of Intelligent & Robotic Systems, 2019, 93(3): 723-743.
7	Estremera J, Cobano J A, Gonzalez de Santos P. Continuous Free-crab Gaits for Hexapod Robots on a Natural Terrain with Forbidden Zones: An Application to Humanitarian Demining[J]. Robotics and Autonomous Systems, 2010, 58(5): 700-711.
8	Estremera Joaquin, Pablo Gonzalez de Santos. Free Gaits for Quadruped Robots over Irregular Terrain[J]. The International Journal of Robotics Research, 2002, 21(2): 115-130.
9	Chen Zhijun, Liu Jimu, Gao Feng. Real-time Gait Planning Method for Six-legged Robots to Optimize the Performances of Terrain Adaptability and Walking Speed[J]. Mechanism and Machine Theory, 2022, 168: 104545.
10	Chen Xi, Wang Liquan, Ye Xiufen, et al. Prototype Development and Gait Planning of Biologically Inspired Multi-legged Crablike Robot[J]. Mechatronics, 2013, 23(4): 429-444.
11	Xu Peng, Ding Liang, Wang Zhikai, et al. Contact Sequence Planning for Hexapod Robots in Sparse Foothold Environment Based on Monte-carlo Tree[J]. IEEE Robotics and Automation Letters, 2022, 7(2): 826-833.
12	Mustafa Suphi Erden, Leblebicioğlu Kemal. Free Gait Generation with Reinforcement Learning for a Six-legged Robot[J]. Robotics and Autonomous Systems, 2008, 56(3): 199-212.
13	Chen Zhihua, Li Jiehao, Wang Shoukun, et al. Flexible Gait Transition for Six Wheel-legged Robot with Unstructured Terrains[J]. Robotics and Autonomous Systems, 2022, 150: 103989.
14	Coelho Joana, Ribeiro Fernando, Dias Bruno, et al. Trends in the Control of Hexapod Robots: A Survey[J]. Robotics, 2021, 10(3): 100.
15	Mnih V, Kavukcuoglu K, Silver D, et al. Human-level Control Through Deep Reinforcement Learning[J]. Nature, 2015, 518(7540): 529-533.
16	傅汇乔, 唐开强, 邓归洲, 等. 基于深度强化学习的六足机器人运动规划[J]. 智能科学与技术学报, 2020, 2(4): 361-371.
	Fu Huiqiao, Tang Kaiqiang, Deng Guizhou, et al. Motion Planning for Hexapod Robot Using Deep Reinforcement Learning[J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4): 361-371.
17	董星宇, 唐开强, 傅汇乔, 等. 基于强化学习的六足机器人动态避障研究[J]. 传感器与微系统, 2022, 41(1): 19-23.
	Dong Xingyu, Tang Kaiqiang, Fu Huiqiao, et al. Study on Dynamic Obstacle Avoidance for Hexapod Robot Based on Reinforcement Learning[J]. Transducer and Microsystem Technologies, 2022, 41(1): 19-23.
18	Fu Huiqiao, Tang Kaiqiang, Li Peng, et al. Deep Reinforcement Learning for Multi-contact Motion Planning of Hexapod Robots[C]//Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. California: IJCAI, 2021: 2381-2388.
19	Tsounis Vassilios, Alge Mitja, Lee Joonho, et al. DeepGait: Planning and Control of Quadrupedal Gaits Using Deep Reinforcement Learning[J]. IEEE Robotics and Automation Letters, 2020, 5(2): 3699-3706.
20	Winkler A W. Optimization-based Motion Planning for Legged Robots[D]. Zurich: ETH Zurich, 2018.
21	Haarnoja T, Zhou A, Abbeel P, et al. Soft Actor-critic: Off-policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[C]//Proceedings of the 35th International Conference on Machine Learning. Chia Laguna Resort, Sardinia, Italy: PMLR, 2018: 1861-1870.
22	In Kim Jae, Hong Mineui, Lee Kyungjae, et al. Learning to Walk a Tripod Mobile Robot Using Nonlinear Soft Vibration Actuators with Entropy Adaptive Reinforcement Learning[J]. IEEE Robotics and Automation Letters, 2020, 5(2): 2317-2324.
23	孙辉辉, 胡春鹤, 张军国. 移动机器人运动规划中的深度强化学习方法[J]. 控制与决策, 2021, 36(6): 1281-1292.
	Sun Huihui, Hu Chunhe, Zhang Junguo. Deep Reinforcement Learning for Motion Planning of Mobile Robots[J]. Control and Decision, 2021, 36(6): 1281-1292.

i	α_i /(°)	a_i	θ_i
1	90	l₁	θ₁
2	0	l₂	θ₂
3	0	l₃	θ₃

参数	基节	髋关节	膝关节
长度/mm	60	120	145.8
转角范围/(°)	[-45, 45]	[0, 45]	[-135, -90]

环境	沟壑宽度
I	0
II	100
III	150

环境	平均回合步数	平均回合成功率/%
I	25	100
II	31	98
III	64	52

[1]	Wang Ziyi, Zhang Kai, Qian Dianwei, Liu Yuzhen. A DRL⁃based Approach for Distributed Equipment Nodes Selection [J]. Journal of System Simulation, 2025, 37(6): 1565-1573.
[2]	Li Min, Zhang Sen, Zeng Xiangguang, Wang Gang, Zhang Tongwei, Xie Dijie, Ren Wenzhe, Zhang Tao. Trajectory Planning of Quadruped Robot Over Obstacle with Single Leg Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(4): 895-909.
[3]	Zhang Sen, Dai Qiangqiang. UAV Path Planning Based on Improved Deep Deterministic Policy Gradients [J]. Journal of System Simulation, 2025, 37(4): 875-881.
[4]	Zhang Bin, Lei Yonglin, Li Qun, Gao Yuan, Chen Yong, Zhu Jiajun, Bao Chenlong. Reinforcement Learning Modeling of Missile Penetration Decision Based on Combat Simulation [J]. Journal of System Simulation, 2025, 37(3): 763-774.
[5]	Wang He, Xu Jianing, Yan Guangyu. Research on Pedestrian Avoidance Strategy for AGV Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(3): 595-606.
[6]	Fei Shuaidi, Cai Changlong, Liu Fei, Chen Minghui, Liu Xiaoming. Research on the Target Allocation Method for Air Defense and Anti-missile Defense of Naval Ships [J]. Journal of System Simulation, 2025, 37(2): 508-516.
[7]	Huang Sijin, Wen Jia, Chen Zheyi. Intelligent Service Migration towards MEC-based IoV Systems [J]. Journal of System Simulation, 2025, 37(2): 379-391.
[8]	Li Chao, Li Jiabao, Ding Caichang, Ye Zhiwei, Zuo Fangwei. Edge Surveillance Task Offloading and Resource Allocation Algorithm Based on DRL [J]. Journal of System Simulation, 2024, 36(9): 2113-2126.
[9]	Wang Hongjun, Lin Junqiang, Zou Xiangjun, Zhang Po, Zhou Mingxuan, Zou Weirui, Tang Yunchao, Luo Lufeng. Construction of a Virtual Interactive System for Orchards Based on Digital Twin [J]. Journal of System Simulation, 2024, 36(6): 1493-1508.
[10]	Zhu Zilu, Liu Yongkui, Zhang Lin, Wang Lihui, Lin Tingyu. Simulation of Robotic Peg-in-hole Assembly Strategy Based on DRL [J]. Journal of System Simulation, 2024, 36(6): 1414-1424.
[11]	Wang Yuan, Xu Lin, Gong Xiaoze, Zhang Yongliang, Wang Yongli. Gradient-based Deep Reinforcement Learning Interpretation Methods [J]. Journal of System Simulation, 2024, 36(5): 1130-1140.
[12]	Pan Hainan, Chen Bailiang, Huang Kaihong, Ren Junkai, Cheng Chuang, Lu Huimin, Zhang Hui. Flipper Control Method for Tracked Robot Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(2): 405-414.
[13]	Zhao Yingying, Dong Pusen, Zhu Tianchen, Li Fan, Su Yun, Tai Zhenying, Sun Qingyun, Fan Hang. Efficiency Optimization Method for Data Sampling in Power Grid Topology Scheduling Simulation [J]. Journal of System Simulation, 2024, 36(2): 283-295.
[14]	An Jing, Si Guangya, Zhang Lei. Strategy Optimization Method of Multi-dimension Projection Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(1): 39-49.
[15]	Wang Yukun, Wang Ze, Dong Liwei, Li Ni. Research on Multi-aircraft Air Combat Behavior Modeling Based on Hierarchical Intelligent Modeling Methods [J]. Journal of System Simulation, 2023, 35(10): 2249-2261.