Reinforcement-learning-based Adaptive Tracking Control for a Space Continuum Robot Based on Reinforcement Learning

doi:10.16182/j.issn1004731x.joss.21-0632

Abstract

Abstract:

Aiming at the tracking control for three-arm space continuum robot in space active debris removal manipulation, an adaptive sliding mode control algorithm based on deep reinforcement learning is proposed. Through BP network, a data-driven dynamic model is developed as the predictive model to guide the reinforcement learning to adjust the sliding mode controller's parameters online, and finally realize a real-time tracking control. Simulation results show that the proposed data-driven predictive model can accurately predict the robot's dynamic characteristics with the relative error within $± 1 %$ to random trajectories. Compared with the fixed-parameter sliding mode controller, the proposed adaptive controller has a lower overshoot and shorter settling time and can achieve a better tracking performance.

Key words: space continuum robot, reinforcement learning, predictive control, sliding mode control, trajectory tracking

CLC Number:

TP273.2

Da Jiang, Zhiqin Cai, Zhongzhen Liu, Haijun Peng, Zhigang Wu. Reinforcement-learning-based Adaptive Tracking Control for a Space Continuum Robot Based on Reinforcement Learning[J]. Journal of System Simulation, 2022, 34(10): 2264-2271.

Figures/Tables 8

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

References 15

1	Grassmann R, Modes V, Burgner-Kahrs J. Learning the Forward and Inverse Kinematics of a 6-DOF Concentric Tube Continuum Robot in SE(3)[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). New York: IEEE, 2018: 5125-5132.
2	Lai J, Huang K, Chu H K. A Learning-based Inverse Kinematics Solver for a Multi-Segment Continuum Robot in Robot-Independent Mapping[C]//2019 IEEE International Conference on Robotics and Biomimetics (ROBIO). New York: IEEE, 2019: 576-582.
3	Thuruthel T G, Falotico E, Manti M, et al. Stable Open-Loop Control of Soft Robotic Manipulators[J]. IEEE Robotics and Automation Letters(S2377-3766), 2018, 3(2): 1292-1298.
4	Thuruthel T G, Falotico E, Renda F, et al. Learning Dynamic Models for Open Loop Predictive Control of Soft Robotic Manipulators[J]. Bioinspiration & Biomimetics (S1748-3182), 2017, 12(6): 066003.
5	Li L, Miao Y, Qureshi A H, et al. MPC-MPNet: Model-Predictive Motion Planning Networks for Fast, Near-Optimal Planning Under Kinodynamic Constraints [J]. IEEE Robotics and Automation Letters (S2377-3766), 2021, 6(3): 4496-4503.
6	Ouyang B, Mo H, Chen H, et al. Robust Model Predictive Deformation Control of a Soft Object by Using a Flexible Continuum Robot[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS). New York: IEEE, 2018: 613-618.
7	Tang Z Q, Heung H L, Tong K Y, et al. A Novel Iterative Learning Model Predictive Control Method for Soft Bending Actuators [C]//2019 International Conference on Robotics and Automation (ICRA). New York: IEEE, 2019: 4004-4010.
8	Frazelle C, Rogers J, Karamouzas I, et al. Optimizing a Continuum Manipulator's Search Policy Through Model-Free Reinforcement Learning [C]//IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). New York: IEEE, 2020: 5564-5571.
9	Shin C, Ferguson P W, Pedram S A, et al. Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Control [C]//2019 International Conference on Robotics and Automation (ICRA). New York: IEEE, 2019: 3875-3881.
10	Thuruthel T G, Falotico E, Renda F, et al. Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators [J]. IEEE Transactions on Robotics (S1552-3098), 2019, 35(1): 124-134.
11	邱小璐, 蔡志勤, 刘忠振, 等. 空间连续型机器人自适应鲁棒容错控制[J]. 计算力学学报, 2021, 38(1): 46-50.
	Qiu Xiaolu, Cai Zhiqin, Liu Zhongzhen, et al. Adaptive Robust Fault Tolerant Control of A Space Continuum Robot[J]. Chinese Journal of Computational Mechanics, 2021, 38(1): 46-50.
12	Fujimoto S, Van H, Meger D. Addressing Function Approximation Error in Actor-Critic Methods [C]//2018 International Conference on Machine Learning(ICML). New York: PMLR, 2018: 1587-1596.
13	Gu S, Holly E, Lillicrap T, et al. Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-policy Updates [C]// 2017 IEEE International Conference on Robotics and Automation (ICRA). New York: IEEE, 2017: 3389-3396.
14	Braganza D, Dawson D M, Walker I D, et al. A Neural Network Controller for Continuum Robots [J]. IEEE Transactions on Robotics (S1552-3098), 2007, 23(6): 1270-1277.
15	Wang J, Zhou Y, Bao Y, et al. Trajectory Tracking Control Using Fractional-Order Terminal Sliding Mode Control With Sliding Perturbation Observer for a 7-DOF Robot Manipulator [J]. IEEE/ASME Transactions on Mechatronics (S1083-4435), 2020, 25(4): 1886-1893.

[1]	Yejian Zhao, Yanhong Wang, Jun Zhang, Hongxia Yu, Zhongda Tian. Application of Improved Q Learning Algorithm in Job Shop Scheduling Problem [J]. Journal of System Simulation, 2022, 34(6): 1247-1258.
[2]	Sen Zhang, Mengyan Zhang, Jingping Shao, Jiexin Pu. Multi-UAVs 3D Path Planning Method Based on Random Strategy Search [J]. Journal of System Simulation, 2022, 34(6): 1286-1295.
[3]	Baolin Du, Dachang Zhu, Yihua Pan. Fuzzy Super-twisting Second Order Sliding Mode Trajectory Tracking Control for Robotic Manipulator [J]. Journal of System Simulation, 2022, 34(6): 1343-1352.
[4]	Lingjia Ni, Xiaoxia Huang, Hongga Li, Zibo Zhang. Research on Fire Emergency Evacuation Simulation Based on Cooperative Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(6): 1353-1366.
[5]	Hongwei Wang, Peng Yang. Research on Optimization of Airport Cargo Business Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(3): 651-660.
[6]	Xiaohan Wang, Lin Zhang, Yuanjun Laili, Kunyu Xie, Tingchun Hu. Constructing the Agent Discrete Simulation Based on DEVS Atomic Model [J]. Journal of System Simulation, 2022, 34(2): 191-200.
[7]	Qirui Li, Xinyi Peng. Job Scheduling and Simulation in Cloud Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(2): 258-268.
[8]	Zhao Jing, Wang Peng, Ding Xiaoqian, Jiang Guoping, Xu Fengyu, Sun Yanfei. Fault Tolerant Control and Simulation of Quadrotor Based on Adaptive Observer [J]. Journal of System Simulation, 2022, 34(1): 1-10.
[9]	Lu Hongguang, Zhao Shuen. Research on Intelligent Vehicle Trajectory Tracking Control Based on Robust Model Prediction [J]. Journal of System Simulation, 2022, 34(1): 153-162.
[10]	Qiming Wang, Jiangyue Jiang, Zhichao Lü, Hanzu Zhang. Research on Cooperative Adaptive Cruise Control Strategy Based on Improved MPC [J]. Journal of System Simulation, 2022, 34(09): 2087-2097.
[11]	Wu Xi, Meng Xianglin, Yang Jingyu. Study on Next-generation Strategic Wargame System [J]. Journal of System Simulation, 2021, 33(9): 2017-2024.
[12]	Zhao Yu, Guo Jifeng, Yan Peng, Bai Chengchao. Self-learning-based Multiple Spacecraft Evasion Decision Making Simulation Under Sparse Reward Condition [J]. Journal of System Simulation, 2021, 33(8): 1766-1774.
[13]	Pan Hongguang, Gao Lei, Mi Wenyu. MPC Algorithm Design Based on Improved Macroscopic Traffic Flow Model [J]. Journal of System Simulation, 2021, 33(8): 1875-1881.
[14]	Zhang Ruimin, Chen Qiaoyu. Trajectory Tracking Control of Robotic Manipulators Based on Smooth Second-order Sliding Mode [J]. Journal of System Simulation, 2021, 33(6): 1315-1322.
[15]	Leng Yao, Zhao Shuen. Explicit Model Predictive Control for Intelligent Vehicle Lateral Trajectory Tracking [J]. Journal of System Simulation, 2021, 33(5): 1177-1187.