基于强化学习的连续型机械臂自适应跟踪控制

doi:10.16182/j.issn1004731x.joss.21-0632

系统仿真学报 ›› 2022, Vol. 34 ›› Issue (10): 2264-2271.doi: 10.16182/j.issn1004731x.joss.21-0632

• 仿真支撑平台/系统技术 • 上一篇下一篇

基于强化学习的连续型机械臂自适应跟踪控制

江达¹(), 蔡志勤¹(), 刘忠振¹, 彭海军¹^,², 吴志刚²

^1.大连理工大学，辽宁大连 116024
^2.工业装备结构分析国家重点实验室，辽宁大连 116024

收稿日期:2021-07-07 修回日期:2021-09-12 出版日期:2022-10-30 发布日期:2022-10-18
通讯作者: 蔡志勤 E-mail:ziangdar@sina.com;zhqcai@dlut.edu.cn
作者简介:江达(1992-)，男，博士生，研究方向为空间机器人动力学与控制。E-mail：ziangdar@sina.com
基金资助:
国家自然科学基金重大研究计划重点项目(91748203);国家自然科学基金优秀青年项目(11922203)

Reinforcement-learning-based Adaptive Tracking Control for a Space Continuum Robot Based on Reinforcement Learning

Da Jiang¹(), Zhiqin Cai¹(), Zhongzhen Liu¹, Haijun Peng¹^,², Zhigang Wu²

^1.Dalian University of Technology, Dalian 116024, China
^2.State Key Laboratory of Structural Analysis for Industrial Equipment, Dalian 116024, China

Received:2021-07-07 Revised:2021-09-12 Online:2022-10-30 Published:2022-10-18
Contact: Zhiqin Cai E-mail:ziangdar@sina.com;zhqcai@dlut.edu.cn

摘要/Abstract

摘要：

针对空间主动碎片清除操作中连续型三臂节机器人系统跟踪问题，提出一种基于强化学习的自适应滑模控制算法。基于数据驱动的建模方法，采用BP神经网络对三臂节连续型机械臂进行建模，并作为预测模型指导强化学习实时调节所提出滑模控制器的控制参数，从而实现连续型机器人运动的实时跟踪控制。仿真结果表明：提出的数据驱动的预测模型对随机轨迹预测的相对误差保持在 $± 1 %$ 以内，能够高精度地反映系统动态特性。对比固定参数的滑模控制器，提出的自适应控制器在保证系统达到控制目标的同时具有更低的超调量和更短的调节时间，表现出更好的控制效果。

关键词: 空间连续型机器人, 强化学习, 预测控制, 滑模控制, 轨迹跟踪

Abstract:

Aiming at the tracking control for three-arm space continuum robot in space active debris removal manipulation, an adaptive sliding mode control algorithm based on deep reinforcement learning is proposed. Through BP network, a data-driven dynamic model is developed as the predictive model to guide the reinforcement learning to adjust the sliding mode controller's parameters online, and finally realize a real-time tracking control. Simulation results show that the proposed data-driven predictive model can accurately predict the robot's dynamic characteristics with the relative error within $± 1 %$ to random trajectories. Compared with the fixed-parameter sliding mode controller, the proposed adaptive controller has a lower overshoot and shorter settling time and can achieve a better tracking performance.

Key words: space continuum robot, reinforcement learning, predictive control, sliding mode control, trajectory tracking

中图分类号:

TP273.2

江达, 蔡志勤, 刘忠振, 彭海军, 吴志刚. 基于强化学习的连续型机械臂自适应跟踪控制[J]. 系统仿真学报, 2022, 34(10): 2264-2271.

Da Jiang, Zhiqin Cai, Zhongzhen Liu, Haijun Peng, Zhigang Wu. Reinforcement-learning-based Adaptive Tracking Control for a Space Continuum Robot Based on Reinforcement Learning[J]. Journal of System Simulation, 2022, 34(10): 2264-2271.

图/表 8

图1

图2

图3

图4

图5

图6

图7

图8

参考文献 15

1	Grassmann R, Modes V, Burgner-Kahrs J. Learning the Forward and Inverse Kinematics of a 6-DOF Concentric Tube Continuum Robot in SE(3)[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). New York: IEEE, 2018: 5125-5132.
2	Lai J, Huang K, Chu H K. A Learning-based Inverse Kinematics Solver for a Multi-Segment Continuum Robot in Robot-Independent Mapping[C]//2019 IEEE International Conference on Robotics and Biomimetics (ROBIO). New York: IEEE, 2019: 576-582.
3	Thuruthel T G, Falotico E, Manti M, et al. Stable Open-Loop Control of Soft Robotic Manipulators[J]. IEEE Robotics and Automation Letters(S2377-3766), 2018, 3(2): 1292-1298.
4	Thuruthel T G, Falotico E, Renda F, et al. Learning Dynamic Models for Open Loop Predictive Control of Soft Robotic Manipulators[J]. Bioinspiration & Biomimetics (S1748-3182), 2017, 12(6): 066003.
5	Li L, Miao Y, Qureshi A H, et al. MPC-MPNet: Model-Predictive Motion Planning Networks for Fast, Near-Optimal Planning Under Kinodynamic Constraints [J]. IEEE Robotics and Automation Letters (S2377-3766), 2021, 6(3): 4496-4503.
6	Ouyang B, Mo H, Chen H, et al. Robust Model Predictive Deformation Control of a Soft Object by Using a Flexible Continuum Robot[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS). New York: IEEE, 2018: 613-618.
7	Tang Z Q, Heung H L, Tong K Y, et al. A Novel Iterative Learning Model Predictive Control Method for Soft Bending Actuators [C]//2019 International Conference on Robotics and Automation (ICRA). New York: IEEE, 2019: 4004-4010.
8	Frazelle C, Rogers J, Karamouzas I, et al. Optimizing a Continuum Manipulator's Search Policy Through Model-Free Reinforcement Learning [C]//IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). New York: IEEE, 2020: 5564-5571.
9	Shin C, Ferguson P W, Pedram S A, et al. Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Control [C]//2019 International Conference on Robotics and Automation (ICRA). New York: IEEE, 2019: 3875-3881.
10	Thuruthel T G, Falotico E, Renda F, et al. Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators [J]. IEEE Transactions on Robotics (S1552-3098), 2019, 35(1): 124-134.
11	邱小璐, 蔡志勤, 刘忠振, 等. 空间连续型机器人自适应鲁棒容错控制[J]. 计算力学学报, 2021, 38(1): 46-50.
	Qiu Xiaolu, Cai Zhiqin, Liu Zhongzhen, et al. Adaptive Robust Fault Tolerant Control of A Space Continuum Robot[J]. Chinese Journal of Computational Mechanics, 2021, 38(1): 46-50.
12	Fujimoto S, Van H, Meger D. Addressing Function Approximation Error in Actor-Critic Methods [C]//2018 International Conference on Machine Learning(ICML). New York: PMLR, 2018: 1587-1596.
13	Gu S, Holly E, Lillicrap T, et al. Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-policy Updates [C]// 2017 IEEE International Conference on Robotics and Automation (ICRA). New York: IEEE, 2017: 3389-3396.
14	Braganza D, Dawson D M, Walker I D, et al. A Neural Network Controller for Continuum Robots [J]. IEEE Transactions on Robotics (S1552-3098), 2007, 23(6): 1270-1277.
15	Wang J, Zhou Y, Bao Y, et al. Trajectory Tracking Control Using Fractional-Order Terminal Sliding Mode Control With Sliding Perturbation Observer for a 7-DOF Robot Manipulator [J]. IEEE/ASME Transactions on Mechatronics (S1083-4435), 2020, 25(4): 1886-1893.

[1]	赵也践, 王艳红, 张俊, 于洪霞, 田中大. 改进Q学习算法在作业车间调度问题中的应用[J]. 系统仿真学报, 2022, 34(6): 1247-1258.
[2]	张森, 张孟炎, 邵敬平, 普杰信. 基于随机策略搜索的多机三维路径规划方法[J]. 系统仿真学报, 2022, 34(6): 1286-1295.
[3]	杜宝林, 朱大昌, 盘意华. 机械臂模糊超螺旋二阶滑模轨迹跟踪控制[J]. 系统仿真学报, 2022, 34(6): 1343-1352.
[4]	倪凌佳, 黄晓霞, 李红旮, 张子博. 基于协作式深度强化学习的火灾应急疏散仿真研究[J]. 系统仿真学报, 2022, 34(6): 1353-1366.
[5]	王红微, 杨鹏. 基于深度强化学习的机场货运业务优化研究[J]. 系统仿真学报, 2022, 34(3): 651-660.
[6]	王霄汉, 张霖, 赖李媛君, 谢堃钰, 胡听春. 基于DEVS原子模型的智能体离散仿真构建方法[J]. 系统仿真学报, 2022, 34(2): 191-200.
[7]	李启锐, 彭心怡. 基于深度强化学习的云作业调度及仿真研究[J]. 系统仿真学报, 2022, 34(2): 258-268.
[8]	赵静, 王鹏, 丁筱茜, 蒋国平, 徐丰羽, 孙雁飞. 基于观测器的四旋翼容错控制及仿真研究[J]. 系统仿真学报, 2022, 34(1): 1-10.
[9]	路宏广, 赵树恩. 基于鲁棒模型预测的智能汽车轨迹跟踪控制研究[J]. 系统仿真学报, 2022, 34(1): 153-162.
[10]	王启明, 蒋江月, 吕志超, 张汉祖. 基于改进MPC的协同自适应巡航控制策略研究[J]. 系统仿真学报, 2022, 34(09): 2087-2097.
[11]	吴曦, 孟祥林, 杨镜宇. 下一代战略博弈推演系统研究[J]. 系统仿真学报, 2021, 33(9): 2017-2024.
[12]	赵毓, 郭继峰, 颜鹏, 白成超. 稀疏奖励下多航天器规避决策自学习仿真[J]. 系统仿真学报, 2021, 33(8): 1766-1774.
[13]	潘红光, 高磊, 米文毓. 基于改进宏观交通流模型的MPC算法设计[J]. 系统仿真学报, 2021, 33(8): 1875-1881.
[14]	张瑞民, 陈巧玉. 基于光滑二阶滑模的机械臂轨迹跟踪控制[J]. 系统仿真学报, 2021, 33(6): 1315-1322.
[15]	冷姚, 赵树恩. 智能车辆横向轨迹跟踪的显式模型预测控制方法[J]. 系统仿真学报, 2021, 33(5): 1177-1187.

基于强化学习的连续型机械臂自适应跟踪控制

Reinforcement-learning-based Adaptive Tracking Control for a Space Continuum Robot Based on Reinforcement Learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 15

相关文章 15

编辑推荐

Metrics

本文评价