基于深度强化学习的机器人轴孔装配策略仿真研究

doi:10.16182/j.issn1004731x.joss.23-0518

系统仿真学报 ›› 2024, Vol. 36 ›› Issue (6): 1414-1424.doi: 10.16182/j.issn1004731x.joss.23-0518

基于深度强化学习的机器人轴孔装配策略仿真研究

朱子璐¹(), 刘永奎¹(), 张霖², 王力翚³, 林廷宇⁴

^1.西安电子科技大学机电工程学院, 陕西西安 710071
^2.北京航空航天大学自动化科学与电气工程学院, 北京 100191
^3.瑞典皇家理工学院生产工程系, 斯德哥尔摩 25175
^4.北京仿真中心北京市复杂产品先进制造系统工程技术研究中心, 北京 100854

收稿日期:2023-05-05 修回日期:2023-06-23 出版日期:2024-06-28 发布日期:2024-06-19
通讯作者: 刘永奎 E-mail:zilu_zhu@163.com;yongkuiliu@163.com
第一作者简介:朱子璐(2001-)，女，本科生，研究方向为人机协作装配、机器人装配技能学习。E-mail：zilu_zhu@163.com
基金资助:
国家自然科学基金(61973243)

Simulation of Robotic Peg-in-hole Assembly Strategy Based on DRL

Zhu Zilu¹(), Liu Yongkui¹(), Zhang Lin², Wang Lihui³, Lin Tingyu⁴

^1.School of Mechano-Electronic Engineering, Xidian University, Xi'an 710071, China
^2.School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China
^3.Department of Production Engineering KTH Royal Institute of Technology, Stockholm 25175, Sweden
^4.Beijing Complex Product Advanced Manufacturing Engineering Research Center, Beijing Simulation Center, Beijing 100854, China

Received:2023-05-05 Revised:2023-06-23 Online:2024-06-28 Published:2024-06-19
Contact: Liu Yongkui E-mail:zilu_zhu@163.com;yongkuiliu@163.com

摘要/Abstract

摘要：

针对现有轴孔装配方法存在的依赖于精确的接触状态模型、数据采集困难、采样效率低、安全性差等问题，提出了一种基于DRL的机器人轴孔装配策略仿真研究方法。搭建了基于ROS-Gazebo机器人轴孔装配仿真环境，提出了基于最小二乘法对力/力矩传感器进行重力补偿的方法；基于RL的范式对轴孔装配问题建模，并提出了一种基于SAC(soft actor-critic)算法的机器人轴孔装配方法；通过ROS建立了仿真环境与深度强化学习算法的通信机制。实验结果表明：该算法能够使机器人自主且柔顺地完成轴孔装配任务，并具有较好的泛化性。

关键词: 轴孔装配, DRL, 柔顺控制, 装配策略仿真, ROS-Gazebo仿真环境

Abstract:

Aiming at the existing peg-in-hole assembly method problems of dependence on accurate contact state models, difficulties in data acquisition, low sampling efficiency, and poor security, a simulation research method for robot peg-in-hole assembly strategy based on DRL is proposed. A simulation environment of robot peg-in-hole assembly based on ROS-Gazebo is built, and a method of gravity compensation for force/torque sensor based on a least square method is proposed. The reinforcement learning paradigm is employed to model the robot peg-in-hole assembly, and a method based on soft actor-critic(SAC) algorithm is proposed. The communication mechanism between the simulation environment and the deep reinforcement learning algorithm is established through ROS. Simulation experiments show that the proposed SAC algorithm enables robots to accomplish the peg-in-hole assembly task autonomously and compliantly with good generalization ability.

Key words: peg-in-hole assembly, DRL, compliance control, assembly strategy simulation, ROS-Gazebo simulation environment

中图分类号:

TP391.9

朱子璐,刘永奎,张霖等 . 基于深度强化学习的机器人轴孔装配策略仿真研究[J]. 系统仿真学报, 2024, 36(6): 1414-1424.

Zhu Zilu,Liu Yongkui,Zhang Lin,et al . Simulation of Robotic Peg-in-hole Assembly Strategy Based on DRL[J]. Journal of System Simulation, 2024, 36(6): 1414-1424.

图/表 12

图1

图2

图3

图4

表1

图5

图6

图7

表2

图8

图9

表3

参考文献 22

1	刘乃龙, 刘钊铭, 崔龙. 基于深度强化学习的仿真机器人轴孔装配研究[J]. 计算机仿真, 2019, 36(12): 296-301.
	Liu Nailong, Liu Zhaoming, Cui Long. Deep Reinforcement Learning Based Robotic Assembly in Simulation[J]. Computer Simulation, 2019, 36(12): 296-301.
2	Jiang Jingang, Huang Zhiyuan, Bi Zhuming, et al. State-of-the-art Control Strategies for Robotic PiH Assembly[J]. Robotics and Computer-Integrated Manufacturing, 2020, 65: 101894.
3	Whitney D E. Quasi-static Assembly of Compliantly Supported Rigid Parts[J]. Journal of Dynamic Systems, Measurement, and Control, 1982, 104(1): 65-77.
4	Xu Jing, Hou Zhimin, Wang Wei, et al. Feedback Deep Deterministic Policy Gradient with Fuzzy Reward for Robotic Multiple Peg-in-hole Assembly Tasks[J]. IEEE Transactions on Industrial Informatics, 2019, 15(3): 1658-1667.
5	魏明明, 傅卫平, 蒋家婷, 等. 操作机器人轴孔装配的行为动力学控制策略[J]. 机械工程学报, 2015, 51(5): 14-21.
	Wei Mingming, Fu Weiping, Jiang Jiating, et al. Dynamics of Behavior Control Strategy in Peg-in-hole Assembly Task of Manipulator[J]. Journal of Mechanical Engineering, 2015, 51(5): 14-21.
6	陈婵娟, 赵飞飞, 李承, 等. 多传感器协助机器人精确装配[J]. 机械设计与制造, 2020(3): 281-284.
	Chen Chanjuan, Zhao Feifei, Li Cheng, et al. Multi-sensor Assisted Robotic Accurate Assembly[J]. Machinery Design & Manufacture, 2020(3): 281-284.
7	薛亚东, 陈庆盈, 尹建平. 基于力反馈的轴孔柔顺装配策略[J]. 自动化与仪器仪表, 2021(4): 152-155, 163.
	Xue Yadong, Chen Qingying, Yin Jianping. Peg-in-hole Compliant Assembly Strategy Based on Force Feedback[J]. Automation & Instrumentation, 2021(4): 152-155, 163.
8	Shirinzadeh B, Zhong Yongmin, Tilakaratna P D W, et al. A Hybrid Contact State Analysis Methodology for Robotic-based Adjustment of Cylindrical Pair[J]. The International Journal of Advanced Manufacturing Technology, 2011, 52(1): 329-342.
9	潘柏松, 颜天野, 胡鑫达, 等. 基于几何约束与隐马尔可夫链模型的轴孔装配策略[J]. 计算机集成制造系统, 2022, 28(12): 3766-3776.
	Pan Baisong, Yan Tianye, Hu Xinda, et al. Peg-in-hole Assembly Strategy Based on Geometric Constraint and Hidden Markov Model[J]. Computer Integrated Manufacturing Systems, 2022, 28(12): 3766-3776.
10	李帅龙, 张会文, 周维佳. 模仿学习方法综述及其在机器人领域的应用[J]. 计算机工程与应用, 2019, 55(4): 17-30.
	Li Shuailong, Zhang Huiwen, Zhou Weijia. Review of Imitation Learning Methods and Its Application in Robotics[J]. Computer Engineering and Applications, 2019, 55(4): 17-30.
11	Song Jingzhou, Chen Qingle, Li Zhendong. A Peg-in-hole Robot Assembly System Based on Gauss Mixture Model[J]. Robotics and Computer-Integrated Manufacturing, 2021, 67: 101996.
12	Gao Xiao, Ling Jie, Xiao Xiaohui, et al. Learning Force-relevant Skills from Human Demonstration[J]. Complexity, 2019, 2019: 5262859.
13	Tang Te, Lin H C, Zhao Yu, et al. Teach Industrial Robots Peg-hole-insertion by Human Demonstration[C]//2016 IEEE International Conference on Advanced Intelligent Mechatronics (AIM). Piscataway, NJ, USA: IEEE, 2016: 488-494.
14	Li Fengming, Jiang Qi, Zhang Sisi, et al. Robot Skill Acquisition in Assembly Process Using Deep Reinforcement Learning[J]. Neurocomputing, 2019, 345: 92-102.
15	王竣禾, 姜勇. 基于深度强化学习的动态装配算法[J]. 智能系统学报, 2023, 18(1): 2-11.
	Wang Junhe, Jiang Yong. Dynamic Assembly Algorithm Based on Deep Reinforcement Learning[J]. CAAI Transactions on Intelligent Systems, 2023, 18(1): 2-11.
16	Kim Young-Loul, Ahn Kuk-Hyun, Song Jae-Bok. Reinforcement Learning Based on Movement Primitives for Contact Tasks[J]. Robotics and Computer-Integrated Manufacturing, 2020, 62: 101863.
17	徐德, 秦方博. 机器人自动轴孔装配研究进展[J]. 智能科学与技术学报, 2022, 4(2): 200-211.
	Xu De, Qin Fangbo. Research Development on Automated Robotic Peg-in-hole Assembly[J]. Chinese Journal of Intelligent Science and Technology, 2022, 4(2): 200-211.
18	黄玲涛, 王彬, 倪水, 等. 基于力传感器重力补偿的机器人柔顺控制研究[J]. 农业机械学报, 2020, 51(3): 386-393.
	Huang Lingtao, Wang Bin, Ni Shui, et al. Robotic Compliant Control Based on Force Sensor Gravity Compensation[J]. Transactions of the Chinese Society for Agricultural Machinery, 2020, 51(3): 386-393.
19	Deng Yuelin, Hou Zhimin, Yang Wenhao, et al. Sample-efficiency, Stability and Generalization Analysis for Deep Reinforcement Learning on Robotic Peg-in-hole Assembly[C]//International Conference on Intelligent Robotics and Applications. Cham: Springer International Publishing, 2021: 393-403.
20	Feng Xiaoxin, Shi Tian, Li Weibing, et al. Reinforcement Learning-based Impedance Learning for Robot Admittance Control in Industrial Assembly[C]//2022 International Conference on Advanced Robotics and Mechatronics (ICARM). Piscataway, NJ, USA: IEEE, 2022: 1092-1097.
21	Haarnoja T, Zhou A, Abbeel P, et al. Soft Actor-critic: Off-policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[C]//Proceedings of the 35th International Conference on Machine Learning. Chia Laguna Resort, Sardinia, Italy: PMLR, 2018: 1861-1870.
22	Haarnoja T, Zhou A, Hartikainen K, et al. Soft Actor-critic Algorithms and Applications[EB/OL]. (2019-01-29) [2023-04-26]. .

参数名称	参数值
经验回放池尺寸	10⁶
批量尺寸	256
奖励折扣因子	0.99
软更新因子	0.01
学习率	3×10^-4
目标熵	-2

实验设置	装配时间/s	装配成功率/%
A0+B0+C0	6.27	100
A0+B0+C1	6.09	99
A0+B1+C0	6.33	100
A0+B1+C1	7.94	98
A1+B0+C0	5.88	100
A1+B0+C1	5.41	100
A1+B1+C0	6.01	100
A1+B1+C1	7.61	99

算法	训练时间/h	实验设置	装配成功率/%	装配时间/s	最大接触力/N	最大接触力矩/(N·m)
SAC	6.5	B0	99	6.09	5	0.5
SAC	6.5	B1	98	7.94	5	0.7
DDPG	5.8	B0	100	3.25	50	7.0
DDPG	5.8	B1	74	3.56	65	8.0
PPO	4.3	B0	93	11.79	5	0.8
PPO	4.3	B1	75	14.02	30	3.5

[1]	彭建伟, 廖哲霖, 姚瀚晨, 万郅玙, 朱利琦, 戴厚德. 基于拓展社会力的机器人柔顺跟随与避障控制[J]. 系统仿真学报, 2023, 35(8): 1776-1787.
[2]	刘延斌, 庞翔元, 张彦斌, 郭冰菁, 韩建海. 踝关节康复机器人主动训练柔顺控制研究[J]. 系统仿真学报, 2020, 32(1): 54-60.
[3]	吕红力, 王仁芳. 基于符号压力函数驱动的活动轮廓图像分割[J]. 系统仿真学报, 2016, 28(1): 174-182.

基于深度强化学习的机器人轴孔装配策略仿真研究

Simulation of Robotic Peg-in-hole Assembly Strategy Based on DRL

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 22

相关文章 3

编辑推荐

Metrics

本文评价