Path Planning of Mobile Robots Based on Memristor Reinforcement Learning in Dynamic Environment

doi:10.16182/j.issn1004731x.joss.22-0334

Abstract

Abstract:

In order to solve the path planning problem of mobile robots in dynamic environment, two-layer path planning algorithm based on improved ant colony algorithm and MA-DQN algorithm is proposed. Static global path planning is accomplished by ant colony algorithm that improved the probabilistic transfer function and the pheromone updating principle; the traditional DQN algorithm structure is improved by using the memristor as the synaptic structure of neural network, and then completed the local dynamic obstacle avoidance of the mobile robot. The path planning mechanism is switched according to whether there are dynamic obstacles within the sensing range of the mobile robot, so as to completed the path planning task in the dynamic environment. The simulation results show that the algorithm can effectively plan a feasible path for mobile robots in a dynamic environment in real time.

Key words: dynamic environment, (deep q-network)DQN, memristor, in-memory computing, path planning

CLC Number:

TP242

Hailan Yang, Yongqiang Qi, Baolei Wu, Dan Rong, Miaoying Hong, Jun Wang. Path Planning of Mobile Robots Based on Memristor Reinforcement Learning in Dynamic Environment[J]. Journal of System Simulation, 2023, 35(7): 1619-1633.

Figures/Tables 21

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Table 1

Improved ant colony algorithm parameters

参数	数值
蚂蚁数目 $m$	50
最大迭代次数 $N m a x$	100
信息素启发因子 $α$	1
距离启发因子 $β$	7
拐弯启发因子 $κ$	1
挥发系数 $ρ$	0.5
信息素强度 $Q 1$	10

Table 1

Fig. 12

Table 2

Memristor parameters

参数	数值
最小阻值 $R o n / Ω$	100
最大阻值 $R o f f / Ω$	2 000
忆阻器初值 $x 0$	0.1
线性漂移系数 $μ v / (m 2 ⋅ s - 1 ⋅ V - 1)$	10^-14
忆阻器长度 $D / n m$	10
电压阈值 $V t h / V$	0.1
窗函数系数 $p$	5
频率 $f / H z$	6

Table 2

Fig. 13

Fig. 14

Fig. 15

Fig. 16

Table 3

MA-DQN parameters

参数	数值
学习率 $α$	0.01
折扣因子 $γ$	0.9
探索因子 $ε$	0.1
训练次数episode	1000

Table 3

Fig. 17

Fig. 18

References 24

1	林彬, 韩光辉, 宋晨晨, 等. 基于辐射扫描算法的机器人路径规划与仿真[J]. 系统仿真学报, 2021, 33(1): 84-90.
	Lin Bin, Han Guanghui, Song Chenchen, et al. Traversal Path Planning and Simulation of Robot Based on Radiation Scanning[J]. Journal of System Simulation, 2021, 33(1): 84-90.
2	朱大奇, 朱婷婷, 颜明重. 基于改进神经网络的多AUV全覆盖路径规划[J]. 系统仿真学报, 2020, 32(8): 1505-1514.
	Zhu Daqi, Zhu Tingting, Yan Mingzhong. Multi-AUV Complete Coverage Path Planning Based on Improved Neural Network[J]. Journal of System Simulation, 2020, 32(8): 1505-1514.
3	白天翔, 王帅, 沈震, 等. 平行机器人与平行无人系统:框架、结构、过程、平台及其应用[J]. 自动化学报, 2017, 43(2): 161-175.
	Bai Tianxiang, Wang Shuai, Shen Zhen, et al. Parallel Robotics and Parallel Unmanned Systems: Framework, Structure, Process, Platform and Applications[J]. Acta Automatica Sinica, 2017, 43(2): 161-175.
4	吴鹏, 桑成军, 陆忠华, 等. 基于改进A^*算法的移动机器人路径规划研究[J]. 计算机工程与应用, 2019, 55(21): 226-232, 269.
	Wu Peng, Sang Chengjun, Lu Zhonghua, et al. Research on Mobile Robot Path Planning Based on Improved A^* Algorithm[J]. Computer Engineering and Applications, 2019, 55(21): 226-232, 269.
5	陈继清, 谭成志, 莫荣现, 等. 基于人工势场的A^*算法的移动机器人路径规划[J]. 计算机科学, 2021, 48(11): 327-333.
	Chen Jiqing, Tan Chengzhi, Mo Rongxian, et al. Path Planning of Mobile Robot with A^* Algorithm Based on Artificial Potential Field[J]. Computer Science, 2021, 48(11): 327-333.
6	阮晓钢, 周静, 张晶晶, 等. 基于子目标搜索的机器人目标导向RRT路径规划算法[J]. 控制与决策, 2020, 35(10): 2543-2548.
	Ruan Xiaogang, Zhou Jing, Zhang Jingjing, et al. Robot Goal Guide RRT Path Planning Based on Sub-target Search[J]. Control and Decision, 2020, 35(10): 2543-2548.
7	张毅, 李奎, 黄超. 基于改进蚁群算法的二维码移动机器人路径规划方法[J]. 重庆邮电大学学报(自然科学版), 2021, 33(3): 491-497.
	Zhang Yi, Li Kui, Huang Chao. Path Planning Method for Two-dimensional Code Mobile Robot Based on Improved Ant Colony Algorithm[J]. Journal of Chongqing University of Posts and Telecommunications(Natural Science Edition), 2021, 33(3): 491-497.
8	Xin Junfeng, Zhong Jiabao, Yang Fengru, et al. An Improved Genetic Algorithm for Path-planning of Unmanned Surface Vehicle[J]. Sensors, 2019, 19(11): 2640.
9	王霄汉, 张霖, 任磊, 等. 基于强化学习的车间调度问题研究简述[J]. 系统仿真学报, 2021, 33(12): 2782-2791.
	Wang Xiaohan, Zhang Lin, Ren Lei, et al. Brief Review on Applying Reinforcement Learning to Job Shop Scheduling Problems[J]. Journal of System Simulation, 2021, 33(12): 2782-2791.
10	Gao Junli, Ye Weijie, Guo Jing, et al. Deep Reinforcement Learning for Indoor Mobile Robot Path Planning[J]. Sensors, 2020, 20(19): 5493.
11	王毅然, 经小川, 田涛, 等. 基于强化学习的多Agent路径规划方法研究[J]. 计算机应用与软件, 2019, 36(8): 165-171.
	Wang Yiran, Jing Xiaochuan, Tian Tao, et al. Multi-agent Path Planning Based on Reinforcement Learning[J]. Computer Applications and Software, 2019, 36(8): 165-171.
12	Watkins C J C H, Dayan P. Q-learning[J]. Machine Learning, 1992, 8(3): 279-292.
13	Mao Chao, Shen Zuojun. A Reinforcement Learning Framework for the Adaptive Routing Problem in Stochastic Time-dependent Network[J]. Transportation Research Part C: Emerging Technologies, 2018, 93: 179-197.
14	Mnih V, Kavukcuoglu K, Silver D, et al. Human-level Control Through Deep Reinforcement Learning[J]. Nature, 2015, 518(7540): 529-533.
15	Chua L. Memristor-the Missing Circuit Element[J]. IEEE Transactions on Circuit Theory, 1971, 18(5): 507-519.
16	Li Can, Belkin D, Li Yunning, et al. Efficient and Self-adaptive in-situ Learning in Multilayer Memristor Neural Networks[J]. Nature Communications, 2018, 9(1): 2385.
17	Wang Zhongrui, Li Can, Song Wenhao, et al. Reinforcement Learning With Analogue Memristor Arrays[J]. Nature Electronics, 2019, 2(3): 115-124.
18	张耀中, 胡小方, 周跃, 等. 基于多层忆阻脉冲神经网络的强化学习及应用[J]. 自动化学报, 2019, 45(8): 1536-1547.
	Zhang Yaozhong, Hu Xiaofang, Zhou Yue, et al. A Novel Reinforcement Learning Algorithm Based on Multilayer Memristive Spiking Neural Network With Applications[J]. Acta Automatica Sinica, 2019, 45(8): 1536-1547.
19	Strukov D B, Snider G S, Stewart D R, et al. The Missing Memristor Found[J]. Nature, 2008, 453(7191): 80-83.
20	Joglekar Y N, Wolf S J. The Elusive Memristor: Properties of Basic Electrical Circuits[J]. European Journal of Physics, 2009, 30(4): 661.
21	段书凯, 胡小方, 王丽丹, 等. 忆阻器阻变随机存取存储器及其在信息存储中的应用[J]. 中国科学(信息科学), 2012, 42(6): 754-769.
	Duan Shukai, Hu Xiaofang, Wang Lidan, et al. Memristor-based RRAM With Applications[J]. Scientia Sinica(Informationis), 2012, 42(6): 754-769.
22	王雷, 石鑫. 基于改进蚁群算法的移动机器人动态路径规划[J]. 南京理工大学学报, 2019, 43(6): 700-707.
	Wang Lei, Shi Xin. Dynamic Path Planning of Mobile Robot Based on Improved Ant Colony Algorithm[J]. Journal of Nanjing University of Science and Technology, 2019, 43(6): 700-707.
23	Wang Binyu, Liu Zhe, Li Qingbiao, et al. Mobile Robot Path Planning in Dynamic Environments Through Globally Guided Reinforcement Learning[J]. IEEE Robotics and Automation Letters, 2020, 5(4): 6892-6939.
24	胡飞, 尤志强, 刘鹏, 等. 基于忆阻器交叉阵列的卷积神经网络电路设计[J]. 计算机研究与发展, 2018, 55(5): 1097-1107.
	Hu Fei, You Zhiqiang, Liu Peng, et al. Circuit Design of Convolutional Neural Network Based on Memristor Crossbar Arrays[J]. Journal of Computer Research and Development, 2018, 55(5): 1097-1107.

[1]	Yu Yiran, Lai Huicheng, Gao Guxue, Zhang Guo, Peng Wangyinan, Yang Longfei, Huang Junhao. Optimization Method for Multi Agricultural Machinery Collaborative Operation Based on Genetic Algorithm and A * Algorithm [J]. Journal of System Simulation, 2025, 37(9): 2397-2408.
[2]	Ni Peilong, Mao Pengjun, Wang Ning, Yang Mengjie. Robot Path Planning Based on Improved A-DDQN Algorithm [J]. Journal of System Simulation, 2025, 37(9): 2420-2430.
[3]	Zhang Kaixiang, Mao Jianlin, Wang Niya, Xu Zhihao. Multi-robot Hierarchical Collaborative k-robust Path Planning for Path Interference [J]. Journal of System Simulation, 2025, 37(8): 2074-2088.
[4]	Wan Yuhang, Zhu Zilu, Zhong Chunfu, Liu Yongkui, Lin Tingyu, Zhang Lin. Dynamic Path Planning for Robotic Arms Based on an Improved PPO Algorithm [J]. Journal of System Simulation, 2025, 37(6): 1462-1473.
[5]	Ye Chen, Shao Peng, Zhang Shaoping, Li Wenting, Zhou Tengming. Enhanced Artificial Gorilla Algorithm for Mobile Robot Path Planning [J]. Journal of System Simulation, 2025, 37(6): 1474-1485.
[6]	Zhang Yan, Li Binghua, Huo Tao, Liu Rong. Research on Robot Dynamic Obstacle Avoidance Method Based on Improved A* and Dynamic Window Algorithm [J]. Journal of System Simulation, 2025, 37(6): 1555-1564.
[7]	Zhou Xiaohui, Li Yanqiang, Wang Yong, Zhao Decai, Yang Xiaoyao. Robot Path Planning Based on Ant Colony Algorithm with Dual Heuristic Information [J]. Journal of System Simulation, 2025, 37(5): 1280-1289.
[8]	Yu Die, Bao Baizhong, Si Yan, Duan Jian, Zhan Xiaobin, Shi Tielin. Mobile Robot Path Planning Based on Search-step Optimized A* Algorithm [J]. Journal of System Simulation, 2025, 37(4): 1041-1050.
[9]	Zhang Sen, Dai Qiangqiang. UAV Path Planning Based on Improved Deep Deterministic Policy Gradients [J]. Journal of System Simulation, 2025, 37(4): 875-881.
[10]	He Zhigang, Li Dayan, Wang Niya, Mao Jianlin, Wang Ning. A Multi-robot Collaborative Path Planning Algorithm with Chain Working Mode [J]. Journal of System Simulation, 2025, 37(4): 953-967.
[11]	Lin Guijuan, Li Zihan, Wang Yu. Research on Improved A* Algorithm Path Planning Based on Global Key Point Extraction [J]. Journal of System Simulation, 2025, 37(3): 667-678.
[12]	Bai Yuxin, Chen Zhenya, Shi Ruitao, Su Weitao, Ma Zhuoqiang, Yang Shangjin. Research on Robot Path Planning Based on Improved Harris Hawks Algorithm [J]. Journal of System Simulation, 2025, 37(3): 742-752.
[13]	Jiang Limei, Chen Xinwei. Visual SLAM Algorithm Based on Feature Point Selection in Dynamic Scenes [J]. Journal of System Simulation, 2025, 37(3): 753-762.
[14]	Jin Xu, Mo Yuanbin. Multi-strategy Hybrid Mountain Gazelle Optimizer for Robot Path Planning [J]. Journal of System Simulation, 2025, 37(3): 803-821.
[15]	Li Jiongyi, Li Qiang, Zhang Xinwen, Htet Zin Myo, Cai Yongbin. Improved Bidirectional A* Quadratic Path Planning Algorithm for Mobile Robots [J]. Journal of System Simulation, 2025, 37(2): 498-507.