The Synchronous Grasping Method of Virtual-real Assembly Robot Based on Digital Twin

doi:10.16182/j.issn1004731x.joss.23-0575

Abstract

Abstract:

A method based on digital twin for assembly robot virtual-real synchronization and grasping is proposed to address the issues of poor intelligent grasping accuracy and difficult data processing in assembly tasks for industrial robots. Based on the digital twin, a digital twin assembly robot virtual-real synchronization and grasping architecture is designed. The OPC UA information model is built by classifying multi-source heterogeneous data, and the OPC UA communication protocol is used as a bridge for data communication of the assembly robot, achieving virtual-real synchronization. The convolutional neural network is further trained using the virtual robot to improve the grasping accuracy of the materials. The correctness and effectiveness of the proposed method are validated in the developed robot digital twin prototype system, providing a new solution for achieving digital twin virtual-real synchronization and precise grasping for assembly robots.

Key words: digital twin, assembly robot, real-time data, grab, OPC UA, virtual and real synchronization

CLC Number:

TP391.9

Xu Jian, Liu Gaofeng, Zhao Yijian, Zheng Zili, Yan Huanying. The Synchronous Grasping Method of Virtual-real Assembly Robot Based on Digital Twin[J]. Journal of System Simulation, 2024, 36(9): 2181-2192.

Figures/Tables 16

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Table 1

Joint angle data

点位	J1	J2	J3	J4	J5	J6
$A 1$	-70.21	91.32	345.16	112.58	132.98	16.51
$A 2$	-70.12	91.27	345.08	112.47	132.84	16.42
$Δ A$	0.09	0.05	0.08	0.11	0.14	0.09
$B 1$	72.18	121.21	112.93	109.28	39.98	-72.56
$B 2$	72.09	121.08	112.87	109.19	40.05	-72.49
$Δ B$	0.09	0.13	0.06	0.09	0.07	0.07
$C 1$	140.31	-48.68	37.12	-65.89	48.75	94.87
$C 2$	140.23	-48.56	37.19	-65.74	48.63	94.93
$Δ C$	0.08	0.12	0.07	0.15	0.12	0.06

Table 1

Fig. 14

Table 2

References 20

1	Hu Fuwen. Mutual Information-enhanced Digital Twin Promotes Vision-guided Robotic Grasping[J]. Advanced Engineering Informatics, 2022, 52: 101562.
2	Kleeberger Kilian, Bormann Richard, Kraus Werner, et al. A Survey on Learning-based Robotic Grasping[J]. Current Robotics Reports, 2020, 1(4): 239-249.
3	Real E, Moore S, Selle A, et al. Large-scale Evolution of Image Classifiers[C]//Proceedings of the 34th International Conference on Machine Learning. Cambridge: JMLR, 2017: 2902-2911.
4	Akinola I, Angelova A, Lu Yao, et al. Visionary: Vision Architecture Discovery for Robot Learning[C]//2021 IEEE International Conference on Robotics and Automation (ICRA). Piscataway, NJ, USA: IEEE, 2021: 10779-10785.
5	Liu Yongkui, Xu He, Liu Ding, et al. A Digital Twin-based Sim-to-real Transfer for Deep Reinforcement Learning-enabled Industrial Robot Grasping[J]. Robotics and Computer-Integrated Manufacturing, 2022, 78: 102365.
6	Grieves M. Digital Twin: Manufacturing Excellence Through Virtual Factory Replication[J]. White paper, 2014, 1: 1-7.
7	陶飞, 刘蔚然, 张萌, 等. 数字孪生五维模型及十大领域应用[J]. 计算机集成制造系统, 2019, 25(1): 1-18.
	Tao Fei, Liu Weiran, Zhang Meng, et al. Five-dimension Digital Twin Model and Its Ten Applications[J]. Computer Integrated Manufacturing Systems, 2019, 25(1): 1-18.
8	Tipary Bence, Erdős Gábor. Generic Development Methodology for Flexible Robotic Pick-and-place Workcells Based on Digital Twin[J]. Robotics and Computer-Integrated Manufacturing, 2021, 71: 102140.
9	徐健, 宋鑫, 刘秀平, 等. 基于数字孪生的装配机器人建模及系统实现[J]. 系统仿真学报, 2023, 35(7): 1497-1507.
	Xu Jian, Song Xin, Liu Xiuping, et al. Modeling and System Realization of Assembly Robot Based on Digital Twin[J]. Journal of System Simulation, 2023, 35(7): 1497-1507.
10	刘怀兰, 赵文杰, 李世壮, 等. 数字孪生车间机器人虚实驱动系统构建方法[J]. 中国机械工程, 2022, 33(21): 2623-2632.
	Liu Huailan, Zhao Wenjie, Li Shizhuang, et al. Construction Method of Virtual-real Drive Systems for Robots in Digital Twin Workshops[J]. China Mechanical Engineering, 2022, 33(21): 2623-2632.
11	Kuts Vladimir, Otto Tauno, Tähemaa Toivo, et al. Digital Twin Based Synchronised Control and Simulation of the Industrial Robotic Cell Using Virtual Reality[J]. Journal of Machine Engineering, 2019, 19(1): 128-144.
12	王剑, 王好臣, 李学伟, 等. 基于OPC UA的数字孪生车间信息物理融合系统[J]. 现代制造工程, 2023(4): 43-50.
	Wang Jian, Wang Haochen, Li Xuewei, et al. Digital Twin Workshop Information Physical Fusion System Based on OPC UA[J]. Modern Manufacturing Engineering, 2023(4): 43-50.
13	乔峰丽, 苗鸿宾, 纪慧君, 等. 面向数字孪生的工业机械臂手眼标定方法的研究[J]. 机床与液压, 2023, 51(1): 31-35.
	Qiao Fengli, Miao Hongbin, Ji Huijun, et al. Research on Hand-eye Calibration Method for Digital Twin Oriented Industrial Manipulator[J]. Machine Tool & Hydraulics, 2023, 51(1): 31-35.
14	马喜平, 李迪, 姚侠楠, 等. 基于Tsai两步法的视觉点胶系统相机标定方法[J]. 自动化与仪表, 2018, 33(5): 1-4, 18.
	Ma Xiping, Li Di, Yao Xianan, et al. Camera Calibration of Visual Dispensing System Based on Tsai's Two-step Method[J]. Automation & Instrumentation, 2018, 33(5): 1-4, 18.
15	Kyu Tae Park, Lee Donggun, Sang Do Noh. Operation Procedures of a Work-center-level Digital Twin for Sustainable and Smart Manufacturing[J]. International Journal of Precision Engineering and Manufacturing-Green Technology, 2020, 7(3): 791-814.
16	周学广, 吕伟栋, 袁志民. 基于着色Petri网的舰艇指挥控制信息流建模研究[J]. 系统仿真学报, 2019, 31(5): 828-842.
	Zhou Xueguang, Weidong Lü, Yuan Zhimin. Colored Petri Net Based Modeling on Warship Command and Control Information Flow[J]. Journal of System Simulation, 2019, 31(5): 828-842.
17	Perzylo Alexander, Profanter Stefan, Rickert Markus, et al. OPC UA NodeSet Ontologies as a Pillar of Representing Semantic Digital Twins of Manufacturing Resources[C]//2019 24th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA). Piscataway, NJ, USA: IEEE, 2019: 1085-1092.
18	束长宝, 于照, 张继勇. 基于TCP/IP的网络通信及其应用[J]. 微计算机信息, 2006(36): 157-159.
	Shu Changbao, Yu Zhao, Zhang Jiyong. Network Communication and Its Applications Based on TCP/IP[J]. Microcomputer Information, 2006(36): 157-159.
19	Lenz I, Lee H, Saxena A. Deep Learning for Detecting Robotic Grasps[J]. International Journal of Robotics Research, 2015, 34(4/5): 705-724.
20	Wohlkinger Walter, Aldoma Aitor, Rusu R B, et al. 3DNet: Large-scale Object Class Recognition from CAD Models[C]//2012 IEEE International Conference on Robotics and Automation. Piscataway, NJ, USA: IEEE, 2012: 5384-5391.

抓取	原始CNN	数字孪生优化CNN
虚拟	0.58	0.63
真实	0.48	0.56

[1]	Zhao Baiting, Shi Jianguo, Jia Xiaofen. Research on Digital Twin System of Rockshaft Hoist [J]. Journal of System Simulation, 2024, 36(9): 2054-2064.
[2]	Li Dongxue, Liu Yan, Shen Boyao, Jing Yongteng, Ma Qiang, Liu Ran. Carbon Footprint Analysis and Low-carbon Optimization Method Simulation Study of Power Transformer Based on Digital Twin Technology [J]. Journal of System Simulation, 2024, 36(9): 2075-2085.
[3]	Cao Yu, Li Jie, Wang Fang, Liu Zhixiang, Wang Xueliang. Digital Twin Method of Stress Field of Deep Submersible Spherical Shell Based on Simulation Database [J]. Journal of System Simulation, 2024, 36(8): 1764-1779.
[4]	Lei Zhen, Liu Yuhua, Ding Kai, Chen Haoxiang, Li Dongwei. Digital Twin-Driven Structural Thermal Deformation Compensation System for Radio Telescopes [J]. Journal of System Simulation, 2024, 36(8): 1869-1883.
[5]	Ren Qiankun, Xiong Xinli, Liu Jingju, Yao Qian. Reserach on Digital Twins Technology in Cyberspace Security [J]. Journal of System Simulation, 2024, 36(8): 1944-1957.
[6]	Li Ying, Gao Lan, Zhu Zhisong. Digital Twin Modeling and Control of Robots for Intelligent Manufacturing Scenarios [J]. Journal of System Simulation, 2024, 36(7): 1536-1545.
[7]	Dang Hongjie, Yu Wenguang, Yang Huahui. Digital Application of Equipment System Test and Evaluation Based on Digital Twin and Parallel Experiment Theory [J]. Journal of System Simulation, 2024, 36(7): 1729-1736.
[8]	Liang Hongtao, Kong Lingchao, Liu Guozhu, Dong Wenxuan, Liu Xiangyi. ASL-CatBoost Method for Wind Turbine Fault Detection Integrated with Digital Twin [J]. Journal of System Simulation, 2024, 36(4): 873-887.
[9]	Hu Yang, Wang Weiran, Fang Fang, Song Ziqiu, Xu Yuhan, Liu Jizhen. Dynamic Digital Twin Modelling and Semi-Physical Simulation of Wind Turbine Operation [J]. Journal of System Simulation, 2024, 36(3): 636-648.
[10]	Zhao Yingying, Dong Pusen, Zhu Tianchen, Li Fan, Su Yun, Tai Zhenying, Sun Qingyun, Fan Hang. Efficiency Optimization Method for Data Sampling in Power Grid Topology Scheduling Simulation [J]. Journal of System Simulation, 2024, 36(2): 283-295.
[11]	Yin Yanchao, Feng Jiasheng, Yi Bin, Li Wang, Yin Qingwen. Visual Monitoring System of Digital Twin Workshop for Process Manufacturing [J]. Journal of System Simulation, 2024, 36(1): 120-130.
[12]	Zhao Yanbo, Cai Yuanli, Hu Huaizhong. Recursive Subspace-based Model Refinement Method for Digital Twin of Thermal Power Unit [J]. Journal of System Simulation, 2024, 36(1): 195-202.
[13]	Zhong Jinghui, Lin Yutian, Li Wenqiang, Cai Wentong. Intelligent Airport Crowd Management Technology Based on Digital Twin [J]. Journal of System Simulation, 2024, 36(1): 27-38.
[14]	Yuan Biao, Huang Yourui, Xu Shanyong, Rong Xue. Construction and Application of Digital Twin System for Optical Fiber Secondary Coating Workshop [J]. Journal of System Simulation, 2023, 35(9): 2011-2022.
[15]	Lü Nan, Qibing Wang, Jiawei Lu, Juntong Chen, Gang Xiao. Monitoring Method Research on Passenger Behavior on Escalator Based on Digital Twin [J]. Journal of System Simulation, 2023, 35(8): 1737-1747.