Multi-AUV Pursuit Algorithm with Phased Guidance Based on MADDPG

doi:10.16182/j.issn1004731x.joss.25-0894

Abstract

Abstract:

To address the problems such as low exploration and sampling efficiency and sparse rewards in the early training stage under the complex target pursuit environment of multiple AUVs based on MADDPG, a phased-guidance curriculum MADDPG (PGC-MADDPG) algorithm was proposed. The pursuit task was divided into two phases, i.e., target tracking and encircling, through curriculum learning. In the target tracking phase, an experience strategy based on the APF method was introduced as a guidance item to provide prior knowledge of the target direction and accelerate the AUV training speed. After entering the encircling phase, the APF guidance item was removed, enabling the AUVs to accomplish the pursuit task relying on their own strategies. Comparative experimental results indicate that compared with the curriculum learning MADDPG algorithm, the proposed algorithm improves the convergence speed by approximately 36% and increases the final pursuit success rate by approximately 5%.

Key words: cooperative pursuit, MADDPG, curriculum learning, APF, AUV

CLC Number:

TP273

Zhang Sen, Shen Sihang, Sun Xiaojie, Shao Jingping, Guo Shuaiqiang, Deng Yingjie. Multi-AUV Pursuit Algorithm with Phased Guidance Based on MADDPG[J]. Journal of System Simulation, 2026, 38(7): 1964-1977.

Figures/Tables 13

Fig. 1

Fig. 2

Table 1

Hyperparameter settings

参数名称	数值
样本批次	256
最大步数	200
经验池大小 $D$	10⁶
APF经验池 $R$	10⁴
APF衰减起始距离 $D s t a r t$	1
APF引导结束距离 $D e n d$	0.5
Actor网络学习率 $α 1$	10^‒4
Critic网络学习率 $α 2$	10^‒3
奖励权重 $ω 1$ ~ $ω 6$	0.8,0.95,0.7,0.8,0.95,0.7
折扣因子 $γ$	0.95
软更新率 $τ$	0.01

Table 1

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Table 2

Fig. 10

Fig. 11

References 21

[1]	Wang Chaoqun, Mei Dan, Wang Yu, et al. Task Allocation for Multi-AUV System: A Review[J]. Ocean Engineering, 2022, 266, Part 3: 112911.
[2]	Gronauer Sven, Diepold Klaus. Multi-agent Deep Reinforcement Learning: A Survey[J]. Artificial Intelligence Review, 2022, 55(2): 895-943.
[3]	闫敬, 陈天明, 关新平, 等. 自主水下航行器协同控制研究现状与发展趋势[J]. 水下无人系统学报, 2023, 31(1): 108-120.
	Yan Jing, Chen Tianming, Guan Xinping, et al. Autonomous Undersea Vehicle Cooperative Control: Current Research Status and Development Trends[J]. Journal of Unmanned Undersea Systems, 2023, 31(1): 108-120.
[4]	Wang Ling, Zhu Daqi, Pang Wen, et al. A Survey of Underwater Search for Multi-target Using Multi-AUV: Task Allocation, Path Planning, and Formation Control[J]. Ocean Engineering, 2023, 278: 114393.
[5]	王景璟, 魏维, 王永越, 等. 水下自主潜航器集群协同围捕技术[J]. 指挥与控制学报, 2023, 9(6): 637-650.
	Wang Jingjing, Wei Wei, Wang Yongyue, et al. Autonomous Underwater Vehicle Swarm Collaborative Hunting Technologies[J]. Journal of Command and Control, 2023, 9(6): 637-650.
[6]	Wang Chengcai, Cai Wenyu, Lu Jun, et al. Design, Modeling, Control, and Experiments for Multiple AUVs Formation[J]. IEEE Transactions on Automation Science and Engineering, 2022, 19(4): 2776-2787.
[7]	郭银景, 孟庆良, 孔芳, 等. AUV路径规划算法研究现状与展望[J]. 计算机科学与探索, 2020, 14(12): 1981-1994.
	Guo Yinjing, Meng Qingliang, Kong Fang, et al. Research Status and Prospect of AUV Path Planning Algorithms[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(12): 1981-1994.
[8]	李艺春, 刘泽娇, 洪艺天, 等. 基于多智能体强化学习的博弈综述[J]. 自动化学报, 2025, 51(3): 540-558.
	Li Yichun, Liu Zejiao, Hong Yitian, et al. Multi-agent Reinforcement Learning Based Game: A Survey[J]. Acta Automatica Sinica, 2025, 51(3): 540-558.
[9]	郑勇明. 改进粒子群优化算法的舰船三维路径规划[J]. 舰船科学技术, 2023, 45(9): 65-68.
	Zheng Yongming. Ship 3D Path Planning Based on Improved Particle Swarm Optimization Algorithm[J]. Ship Science and Technology, 2023, 45(9): 65-68.
[10]	邢博闻, 张昭夷, 王世明, 等. 基于深度强化学习的多无人艇协同目标搜索算法[J]. 兵器装备工程学报, 2023, 44(11): 118-125.
	Xing Bowen, Zhang Zhaoyi, Wang Shiming, et al. Multi-USV Cooperative Target Search Algorithm Based on Deep Reinforcement Learning[J]. Journal of Ordnance Equipment Engineering, 2023, 44(11): 118-125.
[11]	Lowe R, Wu Yi, Tamar A, et al. Multi-agent Actor-critic for Mixed Cooperative-competitive Environments[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2017: 6382-6393.
[12]	成城, 陈智杰, 郭子铭, 等. 多智能体协同决策仿真平台研究与开发[J]. 系统仿真学报, 2023, 35(12): 2669-2679.
	Cheng Cheng, Chen Zhijie, Guo Ziming, et al. Research and Development of Simulation Training Platform for Multi-agent Collaborative Decision-making[J]. Journal of System Simulation, 2023, 35(12): 2669-2679.
[13]	Li Bo, Wang Jianmei, Song Chao, et al. Multi-UAV Roundup Strategy Method Based on Deep Reinforcement Learning CEL-MADDPG Algorithm[J]. Expert Systems with Applications, 2024, 245: 123018.
[14]	Zhou Yuhang, Zhang Baoheng, Yan Qingzhong, et al. Task Assignment Algorithm Based on Improved MADDPG[C]//2024 5th International Conference on Artificial Intelligence and Computer Engineering (ICAICE). Piscataway: IEEE, 2024: 326-332.
[15]	Zhang Meiyan, Chen Hao, Cai Wenyu. Collaborative Hunting Method of Multi-AUV in 3-D IoUT: Searching, Tracking, and Encirclement Keeping[J]. IEEE Internet of Things Journal, 2025, 12(8): 10958-10973.
[16]	Chen Wei, Nie Jing. A MADDPG-based Multi-agent Antagonistic Algorithm for Sea Battlefield Confrontation[J]. Multimedia Systems, 2023, 29(5): 2991-3000.
[17]	Fan Xiaojing, Guo Yinjing, Liu Hui, et al. Improved Artificial Potential Field Method Applied for AUV Path Planning[J]. Mathematical Problems in Engineering, 2020, 2020(1): 6523158.
[18]	李二超, 王玉华. 改进人工势场法的移动机器人避障轨迹研究[J]. 计算机工程与应用, 2022, 58(6): 296-304.
	Li Erchao, Wang Yuhua. Research on Obstacle Avoidance Trajectory of Mobile Robot Based on Improved Artificial Potential Field[J]. Computer Engineering and Applications, 2022, 58(6): 296-304.
[19]	Wang Xin, Chen Yudong, Zhu Wenwu. A Survey on Curriculum Learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(9): 4555-4576.
[20]	林泽阳, 赖俊, 陈希亮. 基于课程学习的深度强化学习研究综述[J]. 计算机技术与发展, 2022, 32(11): 16-23.
	Lin Zeyang, Lai Jun, Chen Xiliang. An Overview of Deep Reinforcement Learning Based on Curriculum Learning[J]. Computer Technology and Development, 2022, 32(11): 16-23.
[21]	Xue Shan, Zhao Ning, Wang Liqi, et al. Multi-agent Self-attention Reinforcement Learning for Multi-USV Hunting Target[J]. Neural Networks, 2025, 189: 107574.

算法	平均回合奖励	高成功率所需回合数	围捕成功率/%	单步推理耗时/ms	训练收敛总时/h
PGC-MADDPG	89	11 312	93	1.30	8.5
CEL-MADDPG	-16	15 427	88	1.20	11.8
MADDPG	-177	10 386	68	1.20	—
MAPPO	-149	13 981	71	1.25	—