Trajectory Planning of Quadruped Robot Over Obstacle with Single Leg Based on Deep Reinforcement Learning

doi:10.16182/j.issn1004731x.joss.23-1535

Abstract

Abstract:

Aiming at the problems of joint vibration and high energy consumption of quadruped robot in the process of walking over obstacles, a foot trajectory planning method of quadruped robot based on deep reinforcement learning SAC algorithm is proposed. Based on robot kinematics and Monte Carlo method, the motion space of the single-legged foot of quadruped robot is analyzed. A compound seventh-degree polynomial trajectory of the quadruped robot is planned. The SAC algorithm is used to train and obtain the low energy consumption obstacle crossing strategy of four-legged robot under different obstacle environment. The simulation results show that the compound seventh-degree polynomial trajectory planning can effectively reduce the joint vibration and foot contact force generated by the legs of the four-legged robot during obstacle crossing, and the robot can obtain the ideal trajectory planning parameters after the SAC algorithm training, and realize the stable walking over the obstacle with low energy consumption.

Key words: quadruped robot, trajectory planning, DRL, walking over obstacles, joint energy consumption

CLC Number:

TP242

Li Min, Zhang Sen, Zeng Xiangguang, Wang Gang, Zhang Tongwei, Xie Dijie, Ren Wenzhe, Zhang Tao. Trajectory Planning of Quadruped Robot Over Obstacle with Single Leg Based on Deep Reinforcement Learning[J]. Journal of System Simulation, 2025, 37(4): 895-909.

Figures/Tables 31

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Table 1

Acceleration in x and z directionsat foot of single leg

规划方法	x方向	z方向
复合七次多项式	$2.149 × 10 - 3$	0
复合五次多项式	$- 2.814 × 10 - 3$	$- 2.45 × 10 - 3$

Table 1

Fig. 7

Fig. 8

Table 2

SAC algorithm hyperparameters

参数名称	参数值
Q网络学习率	0.001
策略网络学习率	0.001
温度熵系数 $α$	3
折扣因子 $γ$	0.99
每回合采样数量	256
神经元个数	256, 128
训练最大回合数	$1 × 104$
经验池容量	$1 × 1012$

Table 2

Fig. 9

Fig. 10

Table 3

Table 4

Training time and average energy consumption results

环境

训练

时间 $/ h$

平均奖励

最高奖励

训练前期

平均能耗 $/ J$

训练后期

平均能耗 $/ J$

训练前期平均后撤距离

S 2 / m

训练后期平均后撤距离

S 2 / m

训练前期平均抬腿高度

H / m

训练后期平均抬腿高度

H / m

环境Ⅰ

1.5

0.073

0.035

0.030

0.005 7

0.041

0.034

环境Ⅱ

1.9

0.084

0.033

0.004 2

0.037

0.034

Table 4

Fig. 11

Fig. 12

Fig. 13

Fig. 14

Fig. 15

Fig. 16

Fig. 17

Fig. 18

Fig. 19

Fig. 20

Fig. 21

Fig. 22

Fig. 23

Fig. 24

Fig. 25

Fig. 26

Table 5

Algorithm comparison test set results

深度强化学习算法	环境	越障成功率/%	低能耗越障成功率/%	平均能耗 $/ J$	测试集平均后撤距离 $S 2 / m$	测试集平均抬腿高度 $H / m$
SAC	环境Ⅰ	98	98	0.038	0.006 5	0.035
SAC	环境Ⅱ	97	97	0.036	0.005 2	0.034
DDPG	环境Ⅰ	100	0	0.085	0.036 0	0.037
DDPG	环境Ⅱ	70	0	0.140	0.043 0	0.068

Table 5

References 22

1	Li Xiangfei, Zhao Huan, He Xianming, et al. A Novel Cartesian Trajectory Planning Method by Using Triple NURBS Curves for Industrial Robots[J]. Robotics and Computer-Integrated Manufacturing, 2023, 83: 102576.
2	Huang Zhenzhen, Wu Hong. Overview of Trajectory Planning Methods for Robot Systems[C]//2021 IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS). Piscataway: IEEE, 2021: 375-381.
3	Taheri Hamid, Mozayani Nasser. A Study on Quadruped Mobile Robots[J]. Mechanism and Machine Theory, 2023, 190: 105448.
4	董理, 杨东, 鹿建森. 工业机器人轨迹规划方法综述[J]. 控制工程, 2022, 29(12): 2365-2374.
	Dong Li, Yang Dong, Lu Jiansen. Trajectory Planning Methods for Industrial Robots: A Review[J]. Control Engineering of China, 2022, 29(12): 2365-2374.
5	董豪, 杨静, 李少波, 等. 基于深度强化学习的机器人运动控制研究进展[J]. 控制与决策, 2022, 37(2): 278-292.
	Dong Hao, Yang Jing, Li Shaobo, et al. Research Progress of Robot Motion Control Based on Deep Reinforcement Learning[J]. Control and Decision, 2022, 37(2): 278-292.
6	Zhong Jun, Luo Minzhou, Fan Jizhuang, et al. Trajectory Planning of an Intermittent Jumping Quadruped Robot with Variable Redundant and Underactuated Joints[J]. Complexity, 2018, 2018(1): 5369427.
7	Zeng Xuanqi, Zhang Songyuan, Zhang Hongji, et al. Leg Trajectory Planning for Quadruped Robots with High-speed Trot Gait[J]. Applied Sciences, 2019, 9(7): 1508.
8	Chen Mingfang, Zhang Kaixiang, Wang Sen, et al. Analysis and Optimization of Interpolation Points for Quadruped Robots Joint Trajectory[J]. Complexity, 2020, 2020(1): 3507679.
9	Wang Kang, Zhao Haoyu, Meng Fei, et al. Research on the Jumping Control Methods of a Quadruped Robot That Imitates Animals[J]. Biomimetics, 2023, 8(1): 36.
10	Tao Bo, Yang Dongchao, Huang Geng, et al. Omnidirectional Motion Control Method of Quadruped Robot Based on 3D-CPG Oscillator Group[C]//Robotics in Natural Settings. Cham: Springer International Publishing, 2023: 301-312.
11	Liu Qingyu, Xu Duo, Yuan Bing, et al. Distance-controllable Long Jump of Quadruped Robot Based on Parameter Optimization Using Deep Reinforcement Learning[J]. IEEE Access, 2023, 11: 98566-98577.
12	Sharma Bibhu, Pillai Branesh M, Suthakorn Jackrit. Biomechanical Trajectory Optimization of Human Sit-to-stand Motion with Stochastic Motion Planning Framework[J]. IEEE Transactions on Medical Robotics and Bionics, 2022, 4(4): 1022-1033.
13	Tan Wenhao, Fang Xing, Zhang Wei, et al. A Hierarchical Framework for Quadruped Omnidirectional Locomotion Based on Reinforcement Learning[J]. IEEE Transactions on Automation Science and Engineering, 2024, 21(4): 5367-5378.
14	Wu Jiaxi, Wang Chen'an, Zhang Dianmin, et al. Learning Smooth and Omnidirectional Locomotion for Quadruped Robots[C]//2021 6th IEEE International Conference on Advanced Robotics and Mechatronics (ICARM). Piscataway: IEEE, 2021: 633-638.
15	Bogdanovic Miroslav, Khadiv Majid, Righetti L. Model-free Reinforcement Learning for Robust Locomotion Using Demonstrations from Trajectory Optimization[J]. Frontiers in Robotics and AI, 2022, 9: 854212.
16	Zhu Wangshu, Rosendo A. PSTO: Learning Energy-efficient Locomotion for Quadruped Robots[J]. Machines, 2022, 10(3): 185.
17	Yang Laiyi, Bi Jing, Yuan Haitao. Dynamic Path Planning for Mobile Robots with Deep Reinforcement Learning[J]. IFAC-PapersOnLine, 2022, 55(11): 19-24.
18	Nippun Kumaar A A, Kochuvila Sreeja. Mobile Service Robot Path Planning Using Deep Reinforcement Learning[J]. IEEE Access, 2023, 11: 100083-100096.
19	Dong Lu, He Zichen, Song Chunwei, et al. A Review of Mobile Robot Motion Planning Methods: From Classical Motion Planning Workflows to Reinforcement Learning-based Architectures[J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 439-459.
20	Tian Hao, Xu Xiaolong, Qi Lianyong, et al. CoPace: Edge Computation Offloading and Caching for Self-driving with Deep Reinforcement Learning[J]. IEEE Transactions on Vehicular Technology, 2021, 70(12): 13281-13293.
21	Chihara Takanori, Sakamoto Jiro. Generating Deceleration Behavior of Automatic Driving by Reinforcement Learning That Reflects Passenger Discomfort[J]. International Journal of Industrial Ergonomics, 2022, 91: 103343.
22	陈善言, 关永, 施智平, 等. 机器人碰撞检测方法形式化[J]. 软件学报, 2022, 33(6): 2246-2263.
	Chen Shanyan, Guan Yong, Shi Zhiping, et al. Formalization of Collision Detection Method for Robots[J]. Journal of Software, 2022, 33(6): 2246-2263.

[1]	Zhang Sen, Dai Qiangqiang. UAV Path Planning Based on Improved Deep Deterministic Policy Gradients [J]. Journal of System Simulation, 2025, 37(4): 875-881.
[2]	Wang He, Xu Jianing, Yan Guangyu. Research on Pedestrian Avoidance Strategy for AGV Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(3): 595-606.
[3]	Zhang Bin, Lei Yonglin, Li Qun, Gao Yuan, Chen Yong, Zhu Jiajun, Bao Chenlong. Reinforcement Learning Modeling of Missile Penetration Decision Based on Combat Simulation [J]. Journal of System Simulation, 2025, 37(3): 763-774.
[4]	Huang Sijin, Wen Jia, Chen Zheyi. Intelligent Service Migration towards MEC-based IoV Systems [J]. Journal of System Simulation, 2025, 37(2): 379-391.
[5]	Fei Shuaidi, Cai Changlong, Liu Fei, Chen Minghui, Liu Xiaoming. Research on the Target Allocation Method for Air Defense and Anti-missile Defense of Naval Ships [J]. Journal of System Simulation, 2025, 37(2): 508-516.
[6]	Zhang Daoxun, Chen Xieyuanli, Zhong Zhengyu, Xu Ming, Zheng Zhiqiang, Lu Huimin. Novel Multi-gait Strategy for Stable and Efficient Quadruped Robot Locomotion [J]. Journal of System Simulation, 2025, 37(1): 13-24.
[7]	Wang Zhu, Zhang Zhenpeng, Zhang Mengtong, Xu Guangtong. Sequential Convex Programming Using Safe Flight Corridor for Trajectory Planning of UAVs [J]. Journal of System Simulation, 2025, 37(1): 134-144.
[8]	Li Chao, Li Jiabao, Ding Caichang, Ye Zhiwei, Zuo Fangwei. Edge Surveillance Task Offloading and Resource Allocation Algorithm Based on DRL [J]. Journal of System Simulation, 2024, 36(9): 2113-2126.
[9]	Zhu Zilu, Liu Yongkui, Zhang Lin, Wang Lihui, Lin Tingyu. Simulation of Robotic Peg-in-hole Assembly Strategy Based on DRL [J]. Journal of System Simulation, 2024, 36(6): 1414-1424.
[10]	Wang Hongjun, Lin Junqiang, Zou Xiangjun, Zhang Po, Zhou Mingxuan, Zou Weirui, Tang Yunchao, Luo Lufeng. Construction of a Virtual Interactive System for Orchards Based on Digital Twin [J]. Journal of System Simulation, 2024, 36(6): 1493-1508.
[11]	Wang Yuan, Xu Lin, Gong Xiaoze, Zhang Yongliang, Wang Yongli. Gradient-based Deep Reinforcement Learning Interpretation Methods [J]. Journal of System Simulation, 2024, 36(5): 1130-1140.
[12]	Zhao Yingying, Dong Pusen, Zhu Tianchen, Li Fan, Su Yun, Tai Zhenying, Sun Qingyun, Fan Hang. Efficiency Optimization Method for Data Sampling in Power Grid Topology Scheduling Simulation [J]. Journal of System Simulation, 2024, 36(2): 283-295.
[13]	Wang Xinpeng, Fu Huiqiao, Deng Guizhou, Tang Kaiqiang, Chen Chunlin, Liu Canghai. Research on Motion Planning of Hexapod Robot Based on DRL and Free Gait [J]. Journal of System Simulation, 2024, 36(2): 373-384.
[14]	Pan Hainan, Chen Bailiang, Huang Kaihong, Ren Junkai, Cheng Chuang, Lu Huimin, Zhang Hui. Flipper Control Method for Tracked Robot Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(2): 405-414.
[15]	An Jing, Si Guangya, Zhang Lei. Strategy Optimization Method of Multi-dimension Projection Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(1): 39-49.