Obstacle Avoidance Path Planning and Simulation of Mobile Picking Robot Based on DPPO

doi:10.16182/j.issn1004731x.joss.22-0487

Abstract

Junqiang Lin, Hongjun Wang, Xiangjun Zou, Po Zhang, Chengen Li, Yipeng Zhou, Shujie Yao. Obstacle Avoidance Path Planning and Simulation of Mobile Picking Robot Based on DPPO[J]. Journal of System Simulation, 2023, 35(8): 1692-1704.

Figures/Tables 16

Fig. 1

Table 1

List of state space sets

观测变量	符号	状态描述
位置变量	$p r o b o t$	机器人空间位置
	$p o b s$	障碍物空间位置
	$p t r e e$	果树空间位置
机器人变量	$R v$	线速度
	$R ω$	角速度
	$R t$	移动时间
距离变量	$d R O$	机器人与障碍物的距离
距离变量	$d R G$	机器人与目标点的距离

Table 1

Table 2

Action space collection list

控制方式

符号

动作描述

取值范围

前进控制

a v

$a v > 0$ 加速

$a v = 0$ 静止

[0, 1]

转向控制

a ω

$- 1 < a ω < 0$ 左转

$a ω = 0$ 直行

$0 < a ω < 1$ 右转

[

-

1, 1]

Table 2

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Table 3

Fig. 6

Fig. 7

Table 4

Fig. 8

Table 5

Fig. 9

Fig. 10

Fig. 11

References 25

1	胡广锐, 孔微雨, 齐闯, 等. 果园环境下移动采摘机器人导航路径优化[J]. 农业工程学报, 2021, 37(9): 175-184.
	Hu Guangrui, Kong Weiyu, Qi Chuang, et al. Optimization of the Navigation Path for a Mobile Harvesting Robot in Orchard Environment[J]. Transactions of the Chinese Society of Agricultural Engineering, 2021, 37(9): 175-184.
2	闫皎洁, 张锲石, 胡希平. 基于强化学习的路径规划技术综述[J]. 计算机工程, 2021, 47(10): 16-25.
	Yan Jiaojie, Zhang Qieshi, Hu Xiping. Review of Path Planning Techniques Based on Reinforcement Learning[J]. Computer Engineering, 2021, 47(10): 16-25.
3	Lai Xin, Li Jiahe, Chambers J. Enhanced Center Constraint Weighted A^* Algorithm for Path Planning of Petrochemical Inspection Robot[J]. Journal of Intelligent & Robotic Systems, 2021, 102(4): 78.
4	Orozco-Rosas U, Montiel O, Sepúlveda Roberto. Mobile Robot Path Planning Using Membrane Evolutionary Artificial Potential Field[J]. Applied Soft Computing, 2019, 77: 236-251.
5	陈志梅, 李敏, 邵雪卷, 等. 基于改进RRT算法的桥式起重机避障路径规划[J]. 系统仿真学报, 2021, 33(8): 1832-1838.
	Chen Zhimei, Li Min, Shao Xuejuan, et al. Obstacle Avoidance Path Planning of Bridge Crane Based on Improved RRT Algorithm[J]. Journal of System Simulation, 2021, 33(8): 1832-1838.
6	Ajeil H F, Ibraheem I K, Sahib M A, et al. Multi-objective Path Planning of an Autonomous Mobile Robot Using Hybrid PSO-MFB Optimization Algorithm[J]. Applied Soft Computing, 2020, 89: 106076.
7	AL-Taharwa I, Sheta A, Al-Weshah M. A Mobile Robot Path Planning Using Genetic Algorithm in Static Environment[J]. Journal of Computer Science, 2008, 4(4): 341-344.
8	邓向阳, 张立民, 方伟, 等. 基于双向汇聚引导蚁群算法的机器人路径规划[J]. 系统仿真学报, 2022, 34(5): 1101-1108.
	Deng Xiangyang, Zhang Limin, Fang Wei, et al. Robot Path Planning Based on Bidirectional Aggregation Ant Colony Optimization[J]. Journal of System Simulation, 2022, 34(5): 1101-1108.
9	张文, 刘勇, 张超凡, 等. 基于方向A^*算法的温室机器人实时路径规划[J]. 农业机械学报, 2017, 48(7): 22-28.
	Zhang Wen, Liu Yong, Zhang Chaofan, et al. Real-time Path Planning of Greenhouse Robot Based on Directional A^* Algorithm[J]. Transactions of the Chinese Society for Agricultural Machinery, 2017, 48(7): 22-28.
10	李腾, 丁佩佩, 刘金芳. 货到人拣选系统多阶段可穿行多AGV路径规划[J]. 系统仿真学报, 2022, 34(7): 1512-1523.
	Li Teng, Ding Peipei, Liu Jinfang. Multi-stage Multi-AGV Path Planning With Walk Under Shelves for Robotic Mobile Fulfillment Systems[J]. Journal of System Simulation, 2022, 34(7): 1512-1523.
11	殷建军, 董文龙, 梁利华, 等. 复杂环境下农业机器人路径规划优化方法[J]. 农业机械学报, 2019, 50(5): 17-22.
	Yin Jianjun, Dong Wenlong, Liang Lihua, et al. Optimization Method of Agricultural Robot Path Planning in Complex Environment[J]. Transactions of the Chinese Society for Agricultural Machinery, 2019, 50(5): 17-22.
12	魏武, 韩进, 李艳杰, 等. 基于双树Quick-RRT^*算法的移动机器人路径规划[J]. 华南理工大学学报(自然科学版), 2021, 49(7): 51-58.
	Wei Wu, Han Jin, Li Yanjie, et al. Path Planning of Mobile Robots Based on Dual-tree Quick-RRT^* Algorithm[J]. Journal of South China University of Technology(Natural Science Edition), 2021, 49(7): 51-58.
13	孙辉辉, 胡春鹤, 张军国. 移动机器人运动规划中的深度强化学习方法[J]. 控制与决策, 2021, 36(6): 1281-1292.
	Sun Huihui, Hu Chunhe, Zhang Junguo. Deep Reinforcement Learning for Motion Planning of Mobile Robots[J]. Control and Decision, 2021, 36(6): 1281-1292.
14	张荣霞, 武长旭, 孙同超, 等. 深度强化学习及在路径规划中的研究进展[J]. 计算机工程与应用, 2021, 57(19): 44-56.
	Zhang Rongxia, Wu Changxu, Sun Tongchao, et al. Progress on Deep Reinforcement Learning in Path Planning[J]. Computer Engineering and Applications, 2021, 57(19): 44-56.
15	刘全, 翟建伟, 章宗长, 等. 深度强化学习综述[J]. 计算机学报, 2018, 41(1): 1-27.
	Liu Quan, Zhai Jianwei, Zhang Zongchang, et al. A Survey on Deep Reinforcement Learning[J]. Chinese Journal of Computers, 2018, 41(1): 1-27.
16	Lin Junqiang, Zhang Po, Li Chengen, et al. APF-DPPO: An Automatic Driving Policy Learning Method Based on the Artificial Potential Field Method to Optimize the Reward Function[J]. Machines, 2022, 10(7): 533.
17	徐晓苏, 袁杰. 基于改进强化学习的移动机器人路径规划方法[J]. 中国惯性技术学报, 2019, 27(3): 314-320.
	Xu Xiaosu, Yuan Jie. Path Planning for Mobile Robot Based on Improved Reinforcement Learning Algorithm[J]. Journal of Chinese Inertial Technology, 2019, 27(3): 314-320.
18	王珂, 卜祥津, 李瑞峰, 等. 景深约束下的深度强化学习机器人路径规划[J]. 华中科技大学学报(自然科学版), 2018, 46(12): 77-82.
	Wang Ke, Bu Xiangjin, Li Ruifeng, et al. Path Planning for Robots Based on Deep Reinforcement Learning by Depth Constraint[J]. Journal of Huazhong University of Science and Technology(Natural Science Edition), 2018, 46(12): 77-82.
19	成怡, 郝密密. 改进深度强化学习的室内移动机器人路径规划[J]. 计算机工程与应用, 2021, 57(21): 256-262.
	Cheng Yi, Hao Mimi. Path Planning for Indoor Mobile Robot With Improved Deep Reinforcement Learning[J]. Computer Engineering and Applications, 2021, 57(21): 256-262.
20	Zheng Jianfeng, Mao Shuren, Wu Zhenyu, et al. Improved Path Planning for Indoor Patrol Robot Based on Deep Reinforcement Learning[J]. Symmetry, 2022, 14(1): 132.
21	邹湘军, 罗锡文, 卢俊, 等. 虚拟环境下农业移动机器人行为及其仿真建模[J]. 系统仿真学报, 2006, 18(增2): 551-553, 562.
	Zou Xiangjun, Luo Xiwen, Lu Jun, et al. Modeling for Behavior and Simulation of Agriculture Mobile Robot in Virtual Environment[J]. Journal of System Simulation, 2006, 18(S2): 551-553, 562.
22	Schulman J, Wolski F, Dhariwal P, et al. Proximal Policy Optimization Algorithms[EB/OL]. (2017-08-28)[2022-05-06].
23	Heess N, TB D, Sriram S, et al. Emergence of Locomotion Behaviours in Rich Environments[EB/OL]. (2017-07-10)[2022-05-06].
24	熊俊涛, 李中行, 陈淑绵, 等. 基于深度强化学习的虚拟机器人采摘路径避障规划[J]. 农业机械学报, 2020, 51(增2): 1-10.
	Xiong Juntao, Li Zhongxing, Chen Shumian, et al. Obstacle Avoidance Planning of Virtual Robot Picking Path Based on Deep Reinforcement Learning[J]. Transactions of the Chinese Society for Agricultural Machinery, 2020, 51(S2): 1-10.
25	熊超, 解武杰, 董文瀚. 基于碰撞锥改进人工势场的无人机避障路径规划[J]. 计算机工程, 2018, 44(9): 314-320.
	Xiong Chao, Xie Wujie, Dong Wenhan. Obstacle Avoidance Path Planning for UAV Based on Artificial Potential Field Improved by Collision Cone[J]. Computer Engineering, 2018, 44(9): 314-320.

奖惩机制类型	实验数	成功数	移动采摘机器人每次完成任务所使用时间/s				成功率/%
奖惩机制类型	实验数	成功数	最小值	平均值	最大值	标准差	成功率/%
接触碰撞惩罚	120	104	9.41	9.66	10.17	0.41	86.7
基于人工势场法	120	111	10.03	10.48	11.23	0.67	92.5
改进人工势场法	120	117	8.96	9.15	9.47	0.25	97.5

[1]	Jiayi Liu, Gang Wang, Qiang Fu, Xiangke Guo, Siyuan Wang. Intelligent Air Defense Task Assignment Based on Assignment Strategy Optimization Algorithm [J]. Journal of System Simulation, 2023, 35(8): 1705-1716.
[2]	Laiyi Yang, Jing Bi, Haitao Yuan. Intelligent Path Planning for Mobile Robots Based on SAC Algorithm [J]. Journal of System Simulation, 2023, 35(8): 1726-1736.
[3]	Menglong Cao, Wenbin Zhao, Zhiqiang Chen. Robot Path Planning by Fusing Particle Swarm Algorithm and Improved Grey Wolf Algorithm [J]. Journal of System Simulation, 2023, 35(8): 1768-1775.
[4]	Jianwei Peng, Zhelin Liao, Hanchen Yao, Zhiyu Wan, Liqi Zhu, Houde Dai. A Compliant Robot Control Based on Extended Social-force Model for Human-following and Obstacle Avoidance [J]. Journal of System Simulation, 2023, 35(8): 1776-1787.
[5]	Hailan Yang, Yongqiang Qi, Baolei Wu, Dan Rong, Miaoying Hong, Jun Wang. Path Planning of Mobile Robots Based on Memristor Reinforcement Learning in Dynamic Environment [J]. Journal of System Simulation, 2023, 35(7): 1619-1633.
[6]	Fei Ding, Yuchen Sha, Ying Hong, Xiao Kuai, Dengyin Zhang. Joint Optimization Strategy of Computing Offloading and Edge Caching for Intelligent Connected Vehicles [J]. Journal of System Simulation, 2023, 35(6): 1203-1214.
[7]	Dalei Song, Wenhao Gan, Yingzhi Xu, Xiuqing Qu, Jiangli Cao. Simulation of Real-Time Path Planning and Formation Control for Unmanned Surface Vessel [J]. Journal of System Simulation, 2023, 35(5): 957-970.
[8]	Yuwen Wu, Zhiyue Niu, Zhenping Li. Picking Path Planning of Container Robots Based on Improved Genetic Algorithm [J]. Journal of System Simulation, 2023, 35(5): 1086-1097.
[9]	Yuxuan Dai, Chenggang Cui. Deep Reinforcement Learning-Based Control Strategy for Boost Converter [J]. Journal of System Simulation, 2023, 35(5): 1109-1119.
[10]	Haotian Xu, Long Qin, Junjie Zeng, Yue Hu, Qi Zhang. Research Progress of Opponent Modeling Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2023, 35(4): 671-694.
[11]	Junxiao Xue, Xiangyan Kong, Bowei Dong, Hao Tao, Haiyang Guan, Lei Shi, Mingliang Xu. Obstacle Avoidance and Simulation of Carrier-Based Aircraft on the Deck of Aircraft Carrier [J]. Journal of System Simulation, 2023, 35(3): 592-603.
[12]	Jiajie Shi, Peng Yang, Yannan Pi. Machine Learning-based Simulation Research of On-line Subway Pedestrian Flow Control [J]. Journal of System Simulation, 2023, 35(2): 386-395.
[13]	Guohui Zhang, Xuan Wang, Yanan Zhang, Ang Gao. Research on Cooperative Path Planning Model of Multiple Unmanned Vehicles in Real Environment [J]. Journal of System Simulation, 2023, 35(2): 408-422.
[14]	Yuchao Zhang, Yuan Jiang, Jiyang Dai. Dynamic Obstacle Avoidance Control of Three-order Multi-robot Cooperative Formation [J]. Journal of System Simulation, 2022, 34(8): 1762-1774.
[15]	Teng Li, Peipei Ding, Jinfang Liu. Multi-Stage Multi-AGV Path Planning with Walk under Shelves for Robotic Mobile Fulfillment Systems [J]. Journal of System Simulation, 2022, 34(7): 1512-1523.