Robot Arm Control Method Based on Deep Reinforcement Learning

doi:10.16182/j.issn1004731x.joss.19-FZ0378

Abstract

Abstract: Deep reinforcement learning continues to explore in the environment and adjusts the neural network parameters by the reward function. The actual production line can not be used as the trial and error environment for the algorithm, so there is not enough data. For that, this paper constructs a virtual robot arm simulation environment, including the robot arm and the object. The Deep Deterministic Policy Gradient (DDPG),in which the state variables and reward function are set,is trained by deep reinforcement learning algorithm in the simulation environment to realize the target of controlling the robot arm to move the gripper below the object. The new method using neural network can improve the adaptability of the control algorithm and shorten the debugging time. The simulation results show that in the environment constructed in this paper, the deep learning algorithm can converge in a shorter time and control the robot arm to achieve specific goals.

Key words: system simulation, unity, reinforcement learning, neural network

CLC Number:

TP391

Li Heyu, Zhao Zhilong, Gu Lei, Guo Liqin, Zeng Bi, Lin Tingyu. Robot Arm Control Method Based on Deep Reinforcement Learning[J]. Journal of System Simulation, 2019, 31(11): 2452-2457.

References

[1] Wopereis H W, Hoekstra J J, Post T H, et al.Application of substantial and sustained force to vertical surfaces using a quadrotor[C]. 2017 IEEE international conference on robotics and automation (ICRA). Macau: IEEE, 2017: 2704-2709.
[2] 李慧洁, 蔡远利. 基于双幂次趋近律的滑模控制方法[J]. 控制与决策, 2016, 31(3): 498-502.
Li Huijie, Cai Yuanli.Sliding mode control with double power reaching law[J]. Control and Decision, 2016, 31(3): 498-502.
[3] Soltanpour M R, Khooban M H.A particle swarm optimization approach for fuzzy sliding mode control for tracking the robot manipulator[J]. Nonlinear Dynamics (S0924-090X), 2013, 74(1/2): 467-478.
[4] Wang Z, Liu X, Liu K, et al.Backstepping-based Lyapunov function construction using approximate dynamic programming and sum of square techniques[J]. IEEE Transactions on Cybernetics (S1083-4419), 2016, 47(10): 3393-3403.
[5] Yin X G, Wang H P, Wu G.Path planning algorithm for bending robots[C]. 2009 IEEE International Conference on Robotics and Biomimetics (ROBIO). Guilin: IEEE, 2009: 392-395.
[6] Cho H C, Song J B.Null space motion control of a redundant robot arm using matrix augmentation and saturation method[C]. 12^th International Conference on Motion and Vibration Control, MOVIC 2014. Sapporo: Japan Society of Mechanical Engineers, 2014.
[7] Li Y M, Tong S C.Adaptive fuzzy output-feedback stabilization control for a class of switched nonstrict-feedback nonlinear systems[J]. IEEE Transactions on Cybernetics (S1083-4419), 2016, 47(4): 1007-1016.
[8] Li X J, Yang G H.Adaptive decentralized control for a class of interconnected nonlinear systems via backstepping approach and graph theory[J]. Automatica (S0005-1098), 2017, 76: 87-95.
[9] Wang H, Wang Z, Liu Y J, et al.Fuzzy tracking adaptive control of discrete-time switched nonlinear systems[J]. Fuzzy Sets and Systems (S0165-0114), 2017, 316: 35-48.
[10] Mnih V, Kavukcuoglu K, Silver D, et al. Playing atari with deep reinforcement learning[J]. arXiv preprint arXiv:1312.5602, 2013.
[11] Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning[J]. arXiv preprint arXiv:1509.02971, 2015.
[12] Schulman J, Levine S, Abbeel P, et al.Trust region policy optimization[C]. International Conference on Machine Learning. 2015: 1889-1897.
[13] Mnih V, Badia A P, Mirza M, et al.Asynchronous methods for deep reinforcement learning[C]. International conference on machine learning. New York: dblp, 2016: 1928-1937.
[14] Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms[J]. arXiv preprint arXiv:1707.06347, 2017.
[15] Heess N, Sriram S, Lemmon J, et al. Emergence of locomotion behaviours in rich environments[J]. arXiv preprint arXiv:1707.02286, 2017.
[16] Buchli J, Stulp F, Theodorou E, et al.Learning variable impedance control[J]. The International Journal of Robotics Research (S0278-3649), 2011, 30(7): 820-833.
[17] Stulp F, Sigaud O.Robot skill learning: From reinforcement learning to evolution strategies[J]. Paladyn, Journal of Behavioral Robotics (S2081-4836), 2013, 4(1): 49-61.
[18] Wang J.Analysis and design of a k-winners-take-all model with a single state variable and the heaviside step activation function[J]. IEEE Transactions on Neural Networks (S2162-237X), 2010, 21(9): 1496-1506.
[19] Liu Q, Wang J.Finite-Time Convergent Recurrent Neural Network With a Hard-Limiting Activation Function for Constrained Optimization With Piecewise-Linear Objective Functions[J]. IEEE Transactions on Neural Networks (S2162-237X), 2011, 22(4): 601-613.
[20] Kormushev P, Calinon S, Caldwell D G.Imitation Learning of Positional and Force Skills Demonstrated via Kinesthetic Teaching and Haptic Input[J]. Advanced Robotics (S0169-1864), 2011, 25(5): 581-603.
[21] 陈友东, 郭佳鑫, 陶永. 基于高斯过程的机器人自适应抓取策略[J]. 北京航空航天大学学报, 2017, 43(9): 1738-1745.
Chen Youdong, Guo Jiaxin, Tao Yong.Adaptive Grasping Strategy of Robot Based on Gaussian Process[J]. Journal of Beijing University of Aeronautics and Astronautics, 2017, 43(9): 1738-1745.
[22] Ngo T Q, Wang Y N, Mai T L, et al.Robust adaptive neural-fuzzy network tracking control for robot manipulator[J]. International Journal of Computers Communications & Control (S1841-9836), 2012, 7(2): 341-352.
[23] Lee C H, Wang W C.Robust adaptive position and force controller design of robot manipulator using fuzzy neural networks[J]. Nonlinear Dynamics (S0924-090X), 2016, 85(1): 343-354.