基于深度强化学习的四足机器人单腿越障轨迹规划

doi:10.16182/j.issn1004731x.joss.23-1535

摘要/Abstract

摘要：

针对四足机器人在越障行走过程中存在关节振动、能耗高等问题，提出了一种基于深度强化学习SAC算法的四足机器人越障轨迹规划方法。基于机器人运动学和蒙特卡罗法分析了四足机器人单腿足端的运动空间；规划了一种复合七次多项式四足机器人越障运动轨迹；利用SAC算法训练并得到了四足机器人单腿在不同障碍物环境下的低能耗越障策略。仿真结果表明：复合七次多项式轨迹规划能有效减少四足机器人腿部在越障时产生的关节振动及足端接触力，在经过SAC算法训练后，机器人能够得到理想的轨迹规划参数，实现低能耗越障稳定行走。

关键词: 四足机器人, 轨迹规划, 深度强化学习, 越障行走, 关节能耗

Abstract:

Aiming at the problems of joint vibration and high energy consumption of quadruped robot in the process of walking over obstacles, a foot trajectory planning method of quadruped robot based on deep reinforcement learning SAC algorithm is proposed. Based on robot kinematics and Monte Carlo method, the motion space of the single-legged foot of quadruped robot is analyzed. A compound seventh-degree polynomial trajectory of the quadruped robot is planned. The SAC algorithm is used to train and obtain the low energy consumption obstacle crossing strategy of four-legged robot under different obstacle environment. The simulation results show that the compound seventh-degree polynomial trajectory planning can effectively reduce the joint vibration and foot contact force generated by the legs of the four-legged robot during obstacle crossing, and the robot can obtain the ideal trajectory planning parameters after the SAC algorithm training, and realize the stable walking over the obstacle with low energy consumption.

Key words: quadruped robot, trajectory planning, DRL, walking over obstacles, joint energy consumption

中图分类号:

TP242

李敏,张森,曾祥光等 . 基于深度强化学习的四足机器人单腿越障轨迹规划[J]. 系统仿真学报, 2025, 37(4): 895-909.

Li Min,Zhang Sen,Zeng Xiangguang,et al . Trajectory Planning of Quadruped Robot Over Obstacle with Single Leg Based on Deep Reinforcement Learning[J]. Journal of System Simulation, 2025, 37(4): 895-909.

图/表 31

图 1

图 2

图3

图 4

图 5

图6

表1

单腿足端x、z方向加速度 (m/s2)

规划方法	x方向	z方向
复合七次多项式	$2.149 × 10 - 3$	0
复合五次多项式	$- 2.814 × 10 - 3$	$- 2.45 × 10 - 3$

表1

图7

图 8

表2

SAC算法超参数

参数名称	参数值
Q网络学习率	0.001
策略网络学习率	0.001
温度熵系数 $α$	3
折扣因子 $γ$	0.99
每回合采样数量	256
神经元个数	256, 128
训练最大回合数	$1 × 104$
经验池容量	$1 × 1012$

表2

图9

图 10

表3

表4

训练集结果

环境

训练

时间 $/ h$

平均奖励

最高奖励

训练前期

平均能耗 $/ J$

训练后期

平均能耗 $/ J$

训练前期平均后撤距离

S 2 / m

训练后期平均后撤距离

S 2 / m

训练前期平均抬腿高度

H / m

训练后期平均抬腿高度

H / m

环境Ⅰ

1.5

0.073

0.035

0.030

0.005 7

0.041

0.034

环境Ⅱ

1.9

0.084

0.033

0.004 2

0.037

0.034

表4

图11

图12

图13

图14

图15

图16

图17

图18

图19

图20

图21

图22

图 23

图24

图25

图26

表 5

算法对比测试集结果

深度强化学习算法	环境	越障成功率/%	低能耗越障成功率/%	平均能耗 $/ J$	测试集平均后撤距离 $S 2 / m$	测试集平均抬腿高度 $H / m$
SAC	环境Ⅰ	98	98	0.038	0.006 5	0.035
SAC	环境Ⅱ	97	97	0.036	0.005 2	0.034
DDPG	环境Ⅰ	100	0	0.085	0.036 0	0.037
DDPG	环境Ⅱ	70	0	0.140	0.043 0	0.068

表 5

参考文献 22

1	Li Xiangfei, Zhao Huan, He Xianming, et al. A Novel Cartesian Trajectory Planning Method by Using Triple NURBS Curves for Industrial Robots[J]. Robotics and Computer-Integrated Manufacturing, 2023, 83: 102576.
2	Huang Zhenzhen, Wu Hong. Overview of Trajectory Planning Methods for Robot Systems[C]//2021 IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS). Piscataway: IEEE, 2021: 375-381.
3	Taheri Hamid, Mozayani Nasser. A Study on Quadruped Mobile Robots[J]. Mechanism and Machine Theory, 2023, 190: 105448.
4	董理, 杨东, 鹿建森. 工业机器人轨迹规划方法综述[J]. 控制工程, 2022, 29(12): 2365-2374.
	Dong Li, Yang Dong, Lu Jiansen. Trajectory Planning Methods for Industrial Robots: A Review[J]. Control Engineering of China, 2022, 29(12): 2365-2374.
5	董豪, 杨静, 李少波, 等. 基于深度强化学习的机器人运动控制研究进展[J]. 控制与决策, 2022, 37(2): 278-292.
	Dong Hao, Yang Jing, Li Shaobo, et al. Research Progress of Robot Motion Control Based on Deep Reinforcement Learning[J]. Control and Decision, 2022, 37(2): 278-292.
6	Zhong Jun, Luo Minzhou, Fan Jizhuang, et al. Trajectory Planning of an Intermittent Jumping Quadruped Robot with Variable Redundant and Underactuated Joints[J]. Complexity, 2018, 2018(1): 5369427.
7	Zeng Xuanqi, Zhang Songyuan, Zhang Hongji, et al. Leg Trajectory Planning for Quadruped Robots with High-speed Trot Gait[J]. Applied Sciences, 2019, 9(7): 1508.
8	Chen Mingfang, Zhang Kaixiang, Wang Sen, et al. Analysis and Optimization of Interpolation Points for Quadruped Robots Joint Trajectory[J]. Complexity, 2020, 2020(1): 3507679.
9	Wang Kang, Zhao Haoyu, Meng Fei, et al. Research on the Jumping Control Methods of a Quadruped Robot That Imitates Animals[J]. Biomimetics, 2023, 8(1): 36.
10	Tao Bo, Yang Dongchao, Huang Geng, et al. Omnidirectional Motion Control Method of Quadruped Robot Based on 3D-CPG Oscillator Group[C]//Robotics in Natural Settings. Cham: Springer International Publishing, 2023: 301-312.
11	Liu Qingyu, Xu Duo, Yuan Bing, et al. Distance-controllable Long Jump of Quadruped Robot Based on Parameter Optimization Using Deep Reinforcement Learning[J]. IEEE Access, 2023, 11: 98566-98577.
12	Sharma Bibhu, Pillai Branesh M, Suthakorn Jackrit. Biomechanical Trajectory Optimization of Human Sit-to-stand Motion with Stochastic Motion Planning Framework[J]. IEEE Transactions on Medical Robotics and Bionics, 2022, 4(4): 1022-1033.
13	Tan Wenhao, Fang Xing, Zhang Wei, et al. A Hierarchical Framework for Quadruped Omnidirectional Locomotion Based on Reinforcement Learning[J]. IEEE Transactions on Automation Science and Engineering, 2024, 21(4): 5367-5378.
14	Wu Jiaxi, Wang Chen'an, Zhang Dianmin, et al. Learning Smooth and Omnidirectional Locomotion for Quadruped Robots[C]//2021 6th IEEE International Conference on Advanced Robotics and Mechatronics (ICARM). Piscataway: IEEE, 2021: 633-638.
15	Bogdanovic Miroslav, Khadiv Majid, Righetti L. Model-free Reinforcement Learning for Robust Locomotion Using Demonstrations from Trajectory Optimization[J]. Frontiers in Robotics and AI, 2022, 9: 854212.
16	Zhu Wangshu, Rosendo A. PSTO: Learning Energy-efficient Locomotion for Quadruped Robots[J]. Machines, 2022, 10(3): 185.
17	Yang Laiyi, Bi Jing, Yuan Haitao. Dynamic Path Planning for Mobile Robots with Deep Reinforcement Learning[J]. IFAC-PapersOnLine, 2022, 55(11): 19-24.
18	Nippun Kumaar A A, Kochuvila Sreeja. Mobile Service Robot Path Planning Using Deep Reinforcement Learning[J]. IEEE Access, 2023, 11: 100083-100096.
19	Dong Lu, He Zichen, Song Chunwei, et al. A Review of Mobile Robot Motion Planning Methods: From Classical Motion Planning Workflows to Reinforcement Learning-based Architectures[J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 439-459.
20	Tian Hao, Xu Xiaolong, Qi Lianyong, et al. CoPace: Edge Computation Offloading and Caching for Self-driving with Deep Reinforcement Learning[J]. IEEE Transactions on Vehicular Technology, 2021, 70(12): 13281-13293.
21	Chihara Takanori, Sakamoto Jiro. Generating Deceleration Behavior of Automatic Driving by Reinforcement Learning That Reflects Passenger Discomfort[J]. International Journal of Industrial Ergonomics, 2022, 91: 103343.
22	陈善言, 关永, 施智平, 等. 机器人碰撞检测方法形式化[J]. 软件学报, 2022, 33(6): 2246-2263.
	Chen Shanyan, Guan Yong, Shi Zhiping, et al. Formalization of Collision Detection Method for Robots[J]. Journal of Software, 2022, 33(6): 2246-2263.

[1]	张森, 代强强. 改进型深度确定性策略梯度的无人机路径规划[J]. 系统仿真学报, 2025, 37(4): 875-881.
[2]	王贺, 许佳宁, 闫广宇. 基于深度强化学习的AGV行人避让策略研究[J]. 系统仿真学报, 2025, 37(3): 595-606.
[3]	张斌, 雷永林, 李群, 高远, 陈永, 朱佳俊, 鲍琛龙. 基于强化学习的导弹突防决策建模研究[J]. 系统仿真学报, 2025, 37(3): 763-774.
[4]	黄思进, 文佳, 陈哲毅. 面向边缘车联网系统的智能服务迁移方法[J]. 系统仿真学报, 2025, 37(2): 379-391.
[5]	费帅迪, 蔡长龙, 刘飞, 陈明晖, 刘晓明. 舰船防空反导的目标分配方法研究[J]. 系统仿真学报, 2025, 37(2): 508-516.
[6]	张道勋, 陈谢沅澧, 钟铮语, 徐明, 郑志强, 卢惠民. 四足机器人稳定高效行进的多步态运动策略[J]. 系统仿真学报, 2025, 37(1): 13-24.
[7]	王祝, 张振鹏, 张梦通, 徐广通. 安全走廊约束的无人机轨迹序列凸优化方法[J]. 系统仿真学报, 2025, 37(1): 134-144.
[8]	姜嘉成, 贾政轩, 徐钊, 林廷宇, 赵芃芃, 欧一鸣. 基于博弈对抗复杂系统的决策建模与求解[J]. 系统仿真学报, 2025, 37(1): 66-78.
[9]	李超, 李贾宝, 丁才昌, 叶志伟, 左方威. 基于DRL的边缘监控任务卸载与资源分配算法[J]. 系统仿真学报, 2024, 36(9): 2113-2126.
[10]	王红军, 林俊强, 邹湘军, 张坡, 周铭轩, 邹伟锐, 唐昀超, 罗陆锋. 基于数字孪生的果园虚拟交互系统构建[J]. 系统仿真学报, 2024, 36(6): 1493-1508.
[11]	王远, 徐琳, 宫小泽, 张永亮, 王永利. 基于梯度的深度强化学习解释方法[J]. 系统仿真学报, 2024, 36(5): 1130-1140.
[12]	秦保新, 张羽霄, 吴思锐, 曹卫冲, 李湛. 基于改进D3QN的煤炭码头卸车排产智能优化方法[J]. 系统仿真学报, 2024, 36(3): 770-781.
[13]	赵莹莹, 董普森, 朱天晨, 李凡, 苏运, 邰振赢, 孙庆赟, 凡航. 面向电网拓扑调度仿真的采样效率优化方法研究[J]. 系统仿真学报, 2024, 36(2): 283-295.
[14]	王鑫鹏, 傅汇乔, 邓归洲, 唐开强, 陈春林, 留沧海. 基于DRL和自由步态的六足机器人运动规划研究[J]. 系统仿真学报, 2024, 36(2): 373-384.
[15]	潘海南, 陈柏良, 黄开宏, 任君凯, 程创, 卢惠民, 张辉. 基于深度强化学习的履带机器人摆臂控制方法[J]. 系统仿真学报, 2024, 36(2): 405-414.