Simulation of Robotic Peg-in-hole Assembly Strategy Based on DRL

doi:10.16182/j.issn1004731x.joss.23-0518

Abstract

Abstract:

Aiming at the existing peg-in-hole assembly method problems of dependence on accurate contact state models, difficulties in data acquisition, low sampling efficiency, and poor security, a simulation research method for robot peg-in-hole assembly strategy based on DRL is proposed. A simulation environment of robot peg-in-hole assembly based on ROS-Gazebo is built, and a method of gravity compensation for force/torque sensor based on a least square method is proposed. The reinforcement learning paradigm is employed to model the robot peg-in-hole assembly, and a method based on soft actor-critic(SAC) algorithm is proposed. The communication mechanism between the simulation environment and the deep reinforcement learning algorithm is established through ROS. Simulation experiments show that the proposed SAC algorithm enables robots to accomplish the peg-in-hole assembly task autonomously and compliantly with good generalization ability.

Key words: peg-in-hole assembly, DRL, compliance control, assembly strategy simulation, ROS-Gazebo simulation environment

CLC Number:

TP391.9

Zhu Zilu, Liu Yongkui, Zhang Lin, Wang Lihui, Lin Tingyu. Simulation of Robotic Peg-in-hole Assembly Strategy Based on DRL[J]. Journal of System Simulation, 2024, 36(6): 1414-1424.

Figures/Tables 12

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Table 1

Fig. 5

Fig. 6

Fig. 7

Table 2

Fig. 8

Fig. 9

Table 3

References 22

1	刘乃龙, 刘钊铭, 崔龙. 基于深度强化学习的仿真机器人轴孔装配研究[J]. 计算机仿真, 2019, 36(12): 296-301.
	Liu Nailong, Liu Zhaoming, Cui Long. Deep Reinforcement Learning Based Robotic Assembly in Simulation[J]. Computer Simulation, 2019, 36(12): 296-301.
2	Jiang Jingang, Huang Zhiyuan, Bi Zhuming, et al. State-of-the-art Control Strategies for Robotic PiH Assembly[J]. Robotics and Computer-Integrated Manufacturing, 2020, 65: 101894.
3	Whitney D E. Quasi-static Assembly of Compliantly Supported Rigid Parts[J]. Journal of Dynamic Systems, Measurement, and Control, 1982, 104(1): 65-77.
4	Xu Jing, Hou Zhimin, Wang Wei, et al. Feedback Deep Deterministic Policy Gradient with Fuzzy Reward for Robotic Multiple Peg-in-hole Assembly Tasks[J]. IEEE Transactions on Industrial Informatics, 2019, 15(3): 1658-1667.
5	魏明明, 傅卫平, 蒋家婷, 等. 操作机器人轴孔装配的行为动力学控制策略[J]. 机械工程学报, 2015, 51(5): 14-21.
	Wei Mingming, Fu Weiping, Jiang Jiating, et al. Dynamics of Behavior Control Strategy in Peg-in-hole Assembly Task of Manipulator[J]. Journal of Mechanical Engineering, 2015, 51(5): 14-21.
6	陈婵娟, 赵飞飞, 李承, 等. 多传感器协助机器人精确装配[J]. 机械设计与制造, 2020(3): 281-284.
	Chen Chanjuan, Zhao Feifei, Li Cheng, et al. Multi-sensor Assisted Robotic Accurate Assembly[J]. Machinery Design & Manufacture, 2020(3): 281-284.
7	薛亚东, 陈庆盈, 尹建平. 基于力反馈的轴孔柔顺装配策略[J]. 自动化与仪器仪表, 2021(4): 152-155, 163.
	Xue Yadong, Chen Qingying, Yin Jianping. Peg-in-hole Compliant Assembly Strategy Based on Force Feedback[J]. Automation & Instrumentation, 2021(4): 152-155, 163.
8	Shirinzadeh B, Zhong Yongmin, Tilakaratna P D W, et al. A Hybrid Contact State Analysis Methodology for Robotic-based Adjustment of Cylindrical Pair[J]. The International Journal of Advanced Manufacturing Technology, 2011, 52(1): 329-342.
9	潘柏松, 颜天野, 胡鑫达, 等. 基于几何约束与隐马尔可夫链模型的轴孔装配策略[J]. 计算机集成制造系统, 2022, 28(12): 3766-3776.
	Pan Baisong, Yan Tianye, Hu Xinda, et al. Peg-in-hole Assembly Strategy Based on Geometric Constraint and Hidden Markov Model[J]. Computer Integrated Manufacturing Systems, 2022, 28(12): 3766-3776.
10	李帅龙, 张会文, 周维佳. 模仿学习方法综述及其在机器人领域的应用[J]. 计算机工程与应用, 2019, 55(4): 17-30.
	Li Shuailong, Zhang Huiwen, Zhou Weijia. Review of Imitation Learning Methods and Its Application in Robotics[J]. Computer Engineering and Applications, 2019, 55(4): 17-30.
11	Song Jingzhou, Chen Qingle, Li Zhendong. A Peg-in-hole Robot Assembly System Based on Gauss Mixture Model[J]. Robotics and Computer-Integrated Manufacturing, 2021, 67: 101996.
12	Gao Xiao, Ling Jie, Xiao Xiaohui, et al. Learning Force-relevant Skills from Human Demonstration[J]. Complexity, 2019, 2019: 5262859.
13	Tang Te, Lin H C, Zhao Yu, et al. Teach Industrial Robots Peg-hole-insertion by Human Demonstration[C]//2016 IEEE International Conference on Advanced Intelligent Mechatronics (AIM). Piscataway, NJ, USA: IEEE, 2016: 488-494.
14	Li Fengming, Jiang Qi, Zhang Sisi, et al. Robot Skill Acquisition in Assembly Process Using Deep Reinforcement Learning[J]. Neurocomputing, 2019, 345: 92-102.
15	王竣禾, 姜勇. 基于深度强化学习的动态装配算法[J]. 智能系统学报, 2023, 18(1): 2-11.
	Wang Junhe, Jiang Yong. Dynamic Assembly Algorithm Based on Deep Reinforcement Learning[J]. CAAI Transactions on Intelligent Systems, 2023, 18(1): 2-11.
16	Kim Young-Loul, Ahn Kuk-Hyun, Song Jae-Bok. Reinforcement Learning Based on Movement Primitives for Contact Tasks[J]. Robotics and Computer-Integrated Manufacturing, 2020, 62: 101863.
17	徐德, 秦方博. 机器人自动轴孔装配研究进展[J]. 智能科学与技术学报, 2022, 4(2): 200-211.
	Xu De, Qin Fangbo. Research Development on Automated Robotic Peg-in-hole Assembly[J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 200-211.
18	黄玲涛, 王彬, 倪水, 等. 基于力传感器重力补偿的机器人柔顺控制研究[J]. 农业机械学报, 2020, 51(3): 386-393.
	Huang Lingtao, Wang Bin, Ni Shui, et al. Robotic Compliant Control Based on Force Sensor Gravity Compensation[J]. Transactions of the Chinese Society for Agricultural Machinery, 2020, 51(3): 386-393.
19	Deng Yuelin, Hou Zhimin, Yang Wenhao, et al. Sample-efficiency, Stability and Generalization Analysis for Deep Reinforcement Learning on Robotic Peg-in-hole Assembly[C]//International Conference on Intelligent Robotics and Applications. Cham: Springer International Publishing, 2021: 393-403.
20	Feng Xiaoxin, Shi Tian, Li Weibing, et al. Reinforcement Learning-based Impedance Learning for Robot Admittance Control in Industrial Assembly[C]//2022 International Conference on Advanced Robotics and Mechatronics (ICARM). Piscataway, NJ, USA: IEEE, 2022: 1092-1097.
21	Haarnoja T, Zhou A, Abbeel P, et al. Soft Actor-critic: Off-policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[C]//Proceedings of the 35th International Conference on Machine Learning. Chia Laguna Resort, Sardinia, Italy: PMLR, 2018: 1861-1870.
22	Haarnoja T, Zhou A, Hartikainen K, et al. Soft Actor-critic Algorithms and Applications[EB/OL]. (2019-01-29) [2023-04-26]. .

参数名称	参数值
经验回放池尺寸	10⁶
批量尺寸	256
奖励折扣因子	0.99
软更新因子	0.01
学习率	3×10^-4
目标熵	-2

实验设置	装配时间/s	装配成功率/%
A0+B0+C0	6.27	100
A0+B0+C1	6.09	99
A0+B1+C0	6.33	100
A0+B1+C1	7.94	98
A1+B0+C0	5.88	100
A1+B0+C1	5.41	100
A1+B1+C0	6.01	100
A1+B1+C1	7.61	99

算法	训练时间/h	实验设置	装配成功率/%	装配时间/s	最大接触力/N	最大接触力矩/(N·m)
SAC	6.5	B0	99	6.09	5	0.5
SAC	6.5	B1	98	7.94	5	0.7
DDPG	5.8	B0	100	3.25	50	7.0
DDPG	5.8	B1	74	3.56	65	8.0
PPO	4.3	B0	93	11.79	5	0.8
PPO	4.3	B1	75	14.02	30	3.5

[1]	Wang Ziyi, Zhang Kai, Qian Dianwei, Liu Yuzhen. A DRL⁃based Approach for Distributed Equipment Nodes Selection [J]. Journal of System Simulation, 2025, 37(6): 1565-1573.
[2]	Zhang Sen, Dai Qiangqiang. UAV Path Planning Based on Improved Deep Deterministic Policy Gradients [J]. Journal of System Simulation, 2025, 37(4): 875-881.
[3]	Li Min, Zhang Sen, Zeng Xiangguang, Wang Gang, Zhang Tongwei, Xie Dijie, Ren Wenzhe, Zhang Tao. Trajectory Planning of Quadruped Robot Over Obstacle with Single Leg Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(4): 895-909.
[4]	Zhang Bin, Lei Yonglin, Li Qun, Gao Yuan, Chen Yong, Zhu Jiajun, Bao Chenlong. Reinforcement Learning Modeling of Missile Penetration Decision Based on Combat Simulation [J]. Journal of System Simulation, 2025, 37(3): 763-774.
[5]	Wang He, Xu Jianing, Yan Guangyu. Research on Pedestrian Avoidance Strategy for AGV Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(3): 595-606.
[6]	Fei Shuaidi, Cai Changlong, Liu Fei, Chen Minghui, Liu Xiaoming. Research on the Target Allocation Method for Air Defense and Anti-missile Defense of Naval Ships [J]. Journal of System Simulation, 2025, 37(2): 508-516.
[7]	Huang Sijin, Wen Jia, Chen Zheyi. Intelligent Service Migration towards MEC-based IoV Systems [J]. Journal of System Simulation, 2025, 37(2): 379-391.
[8]	Li Zonggang, Li Yanbo, Jiao Jianjun, Du Yajiang. A Visual Servo Precision Assembly Method for Riveting Parts Based on Adaptive Extended Kalman Filtering [J]. Journal of System Simulation, 2025, 37(1): 107-118.
[9]	Li Chao, Li Jiabao, Ding Caichang, Ye Zhiwei, Zuo Fangwei. Edge Surveillance Task Offloading and Resource Allocation Algorithm Based on DRL [J]. Journal of System Simulation, 2024, 36(9): 2113-2126.
[10]	Wang Hongjun, Lin Junqiang, Zou Xiangjun, Zhang Po, Zhou Mingxuan, Zou Weirui, Tang Yunchao, Luo Lufeng. Construction of a Virtual Interactive System for Orchards Based on Digital Twin [J]. Journal of System Simulation, 2024, 36(6): 1493-1508.
[11]	Wang Yuan, Xu Lin, Gong Xiaoze, Zhang Yongliang, Wang Yongli. Gradient-based Deep Reinforcement Learning Interpretation Methods [J]. Journal of System Simulation, 2024, 36(5): 1130-1140.
[12]	Pan Hainan, Chen Bailiang, Huang Kaihong, Ren Junkai, Cheng Chuang, Lu Huimin, Zhang Hui. Flipper Control Method for Tracked Robot Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(2): 405-414.
[13]	Wang Xinpeng, Fu Huiqiao, Deng Guizhou, Tang Kaiqiang, Chen Chunlin, Liu Canghai. Research on Motion Planning of Hexapod Robot Based on DRL and Free Gait [J]. Journal of System Simulation, 2024, 36(2): 373-384.
[14]	Zhao Yingying, Dong Pusen, Zhu Tianchen, Li Fan, Su Yun, Tai Zhenying, Sun Qingyun, Fan Hang. Efficiency Optimization Method for Data Sampling in Power Grid Topology Scheduling Simulation [J]. Journal of System Simulation, 2024, 36(2): 283-295.
[15]	An Jing, Si Guangya, Zhang Lei. Strategy Optimization Method of Multi-dimension Projection Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(1): 39-49.