基于分阶段引导的MADDPG多AUV围捕算法

doi:10.16182/j.issn1004731x.joss.25-0894

系统仿真学报 ›› 2026, Vol. 38 ›› Issue (7): 1964-1977.doi: 10.16182/j.issn1004731x.joss.25-0894

基于分阶段引导的MADDPG多AUV围捕算法

张森¹, 申四航¹, 孙晓界¹^,², 邵敬平¹, 郭帅强¹, 邓英杰³

^1.河南科技大学信息工程学院，河南洛阳 471000
^2.上海交通大学，上海 200240
^3.燕山大学，河北秦皇岛 066004

收稿日期:2025-09-15 修回日期:2025-12-22 出版日期:2026-07-28 发布日期:2026-07-31
通讯作者: 孙晓界
第一作者简介:张森(1984-)，男，副教授，博士，研究方向为先进机器人与智能控制技术。
基金资助:
中国博士后科学基金面上项目(2025M770290);河南省自然科学基金(252300423316);河南省科技攻关项目(232102220016)

Multi-AUV Pursuit Algorithm with Phased Guidance Based on MADDPG

Zhang Sen¹, Shen Sihang¹, Sun Xiaojie¹^,², Shao Jingping¹, Guo Shuaiqiang¹, Deng Yingjie³

^1.School of Information Engineering, Henan University of Science and Technology, Luoyang 471000, China
^2.Shanghai Jiao Tong University, Shanghai 200240, China
^3.Yanshan University, Qinhuangdao 066004, China

Received:2025-09-15 Revised:2025-12-22 Online:2026-07-28 Published:2026-07-31
Contact: Sun Xiaojie

摘要/Abstract

摘要：

针对多AUV在基于MADDPG的目标围捕复杂环境下会出现训练初期探索和采样效率低下以及奖励稀疏等问题，提出了一种基于分阶段引导与课程学习的MADDPG（phased-guidance curriculum MADDPG，PGC-MADDPG）算法。通过课程学习将围捕任务分为目标跟踪与合围2个阶段；在目标跟踪阶段引入基于APF的经验策略作为引导项，提供目标方向的先验知识，加速AUV训练速度；在进入合围阶段后移除APF引导项，使AUV依靠自身策略完成围捕任务。对比实验结果表明：该算法与课程学习MADDPG算法相比，收敛速度提升了约36%，围捕成功率提高约5%。

关键词: 协同围捕, MADDPG, 课程学习, APF, AUV

Abstract:

To address the problems such as low exploration and sampling efficiency and sparse rewards in the early training stage under the complex target pursuit environment of multiple AUVs based on MADDPG, a phased-guidance curriculum MADDPG (PGC-MADDPG) algorithm was proposed. The pursuit task was divided into two phases, i.e., target tracking and encircling, through curriculum learning. In the target tracking phase, an experience strategy based on the APF method was introduced as a guidance item to provide prior knowledge of the target direction and accelerate the AUV training speed. After entering the encircling phase, the APF guidance item was removed, enabling the AUVs to accomplish the pursuit task relying on their own strategies. Comparative experimental results indicate that compared with the curriculum learning MADDPG algorithm, the proposed algorithm improves the convergence speed by approximately 36% and increases the final pursuit success rate by approximately 5%.

Key words: cooperative pursuit, MADDPG, curriculum learning, APF, AUV

中图分类号:

TP273

张森,申四航,孙晓界等 . 基于分阶段引导的MADDPG多AUV围捕算法[J]. 系统仿真学报, 2026, 38(7): 1964-1977.

Zhang Sen,Shen Sihang,Sun Xiaojie,et al . Multi-AUV Pursuit Algorithm with Phased Guidance Based on MADDPG[J]. Journal of System Simulation, 2026, 38(7): 1964-1977.

图/表 13

图1

图2

表1

超参数设定

参数名称	数值
样本批次	256
最大步数	200
经验池大小 $D$	10⁶
APF经验池 $R$	10⁴
APF衰减起始距离 $D s t a r t$	1
APF引导结束距离 $D e n d$	0.5
Actor网络学习率 $α 1$	10^‒4
Critic网络学习率 $α 2$	10^‒3
奖励权重 $ω 1$ ~ $ω 6$	0.8,0.95,0.7,0.8,0.95,0.7
折扣因子 $γ$	0.95
软更新率 $τ$	0.01

表1

图3

图4

图5

图6

图7

图8

图9

表2

图10

图11

参考文献 21

[1]	Wang Chaoqun, Mei Dan, Wang Yu, et al. Task Allocation for Multi-AUV System: A Review[J]. Ocean Engineering, 2022, 266, Part 3: 112911.
[2]	Gronauer Sven, Diepold Klaus. Multi-agent Deep Reinforcement Learning: A Survey[J]. Artificial Intelligence Review, 2022, 55(2): 895-943.
[3]	闫敬, 陈天明, 关新平, 等. 自主水下航行器协同控制研究现状与发展趋势[J]. 水下无人系统学报, 2023, 31(1): 108-120.
	Yan Jing, Chen Tianming, Guan Xinping, et al. Autonomous Undersea Vehicle Cooperative Control: Current Research Status and Development Trends[J]. Journal of Unmanned Undersea Systems, 2023, 31(1): 108-120.
[4]	Wang Ling, Zhu Daqi, Pang Wen, et al. A Survey of Underwater Search for Multi-target Using Multi-AUV: Task Allocation, Path Planning, and Formation Control[J]. Ocean Engineering, 2023, 278: 114393.
[5]	王景璟, 魏维, 王永越, 等. 水下自主潜航器集群协同围捕技术[J]. 指挥与控制学报, 2023, 9(6): 637-650.
	Wang Jingjing, Wei Wei, Wang Yongyue, et al. Autonomous Underwater Vehicle Swarm Collaborative Hunting Technologies[J]. Journal of Command and Control, 2023, 9(6): 637-650.
[6]	Wang Chengcai, Cai Wenyu, Lu Jun, et al. Design, Modeling, Control, and Experiments for Multiple AUVs Formation[J]. IEEE Transactions on Automation Science and Engineering, 2022, 19(4): 2776-2787.
[7]	郭银景, 孟庆良, 孔芳, 等. AUV路径规划算法研究现状与展望[J]. 计算机科学与探索, 2020, 14(12): 1981-1994.
	Guo Yinjing, Meng Qingliang, Kong Fang, et al. Research Status and Prospect of AUV Path Planning Algorithms[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(12): 1981-1994.
[8]	李艺春, 刘泽娇, 洪艺天, 等. 基于多智能体强化学习的博弈综述[J]. 自动化学报, 2025, 51(3): 540-558.
	Li Yichun, Liu Zejiao, Hong Yitian, et al. Multi-agent Reinforcement Learning Based Game: A Survey[J]. Acta Automatica Sinica, 2025, 51(3): 540-558.
[9]	郑勇明. 改进粒子群优化算法的舰船三维路径规划[J]. 舰船科学技术, 2023, 45(9): 65-68.
	Zheng Yongming. Ship 3D Path Planning Based on Improved Particle Swarm Optimization Algorithm[J]. Ship Science and Technology, 2023, 45(9): 65-68.
[10]	邢博闻, 张昭夷, 王世明, 等. 基于深度强化学习的多无人艇协同目标搜索算法[J]. 兵器装备工程学报, 2023, 44(11): 118-125.
	Xing Bowen, Zhang Zhaoyi, Wang Shiming, et al. Multi-USV Cooperative Target Search Algorithm Based on Deep Reinforcement Learning[J]. Journal of Ordnance Equipment Engineering, 2023, 44(11): 118-125.
[11]	Lowe R, Wu Yi, Tamar A, et al. Multi-agent Actor-critic for Mixed Cooperative-competitive Environments[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2017: 6382-6393.
[12]	成城, 陈智杰, 郭子铭, 等. 多智能体协同决策仿真平台研究与开发[J]. 系统仿真学报, 2023, 35(12): 2669-2679.
	Cheng Cheng, Chen Zhijie, Guo Ziming, et al. Research and Development of Simulation Training Platform for Multi-agent Collaborative Decision-making[J]. Journal of System Simulation, 2023, 35(12): 2669-2679.
[13]	Li Bo, Wang Jianmei, Song Chao, et al. Multi-UAV Roundup Strategy Method Based on Deep Reinforcement Learning CEL-MADDPG Algorithm[J]. Expert Systems with Applications, 2024, 245: 123018.
[14]	Zhou Yuhang, Zhang Baoheng, Yan Qingzhong, et al. Task Assignment Algorithm Based on Improved MADDPG[C]//2024 5th International Conference on Artificial Intelligence and Computer Engineering (ICAICE). Piscataway: IEEE, 2024: 326-332.
[15]	Zhang Meiyan, Chen Hao, Cai Wenyu. Collaborative Hunting Method of Multi-AUV in 3-D IoUT: Searching, Tracking, and Encirclement Keeping[J]. IEEE Internet of Things Journal, 2025, 12(8): 10958-10973.
[16]	Chen Wei, Nie Jing. A MADDPG-based Multi-agent Antagonistic Algorithm for Sea Battlefield Confrontation[J]. Multimedia Systems, 2023, 29(5): 2991-3000.
[17]	Fan Xiaojing, Guo Yinjing, Liu Hui, et al. Improved Artificial Potential Field Method Applied for AUV Path Planning[J]. Mathematical Problems in Engineering, 2020, 2020(1): 6523158.
[18]	李二超, 王玉华. 改进人工势场法的移动机器人避障轨迹研究[J]. 计算机工程与应用, 2022, 58(6): 296-304.
	Li Erchao, Wang Yuhua. Research on Obstacle Avoidance Trajectory of Mobile Robot Based on Improved Artificial Potential Field[J]. Computer Engineering and Applications, 2022, 58(6): 296-304.
[19]	Wang Xin, Chen Yudong, Zhu Wenwu. A Survey on Curriculum Learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(9): 4555-4576.
[20]	林泽阳, 赖俊, 陈希亮. 基于课程学习的深度强化学习研究综述[J]. 计算机技术与发展, 2022, 32(11): 16-23.
	Lin Zeyang, Lai Jun, Chen Xiliang. An Overview of Deep Reinforcement Learning Based on Curriculum Learning[J]. Computer Technology and Development, 2022, 32(11): 16-23.
[21]	Xue Shan, Zhao Ning, Wang Liqi, et al. Multi-agent Self-attention Reinforcement Learning for Multi-USV Hunting Target[J]. Neural Networks, 2025, 189: 107574.

算法	平均回合奖励	高成功率所需回合数	围捕成功率/%	单步推理耗时/ms	训练收敛总时/h
PGC-MADDPG	89	11 312	93	1.30	8.5
CEL-MADDPG	-16	15 427	88	1.20	11.8
MADDPG	-177	10 386	68	1.20	—
MAPPO	-149	13 981	71	1.25	—

基于分阶段引导的MADDPG多AUV围捕算法

Multi-AUV Pursuit Algorithm with Phased Guidance Based on MADDPG

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 21

相关文章 9

编辑推荐

Metrics

本文评价

[1]	陈治润, 袁杰, 加尔肯别克, 张宁宁, 刘超, 叶玉山. 融合RRT*和APF的机械臂自适应路径规划[J]. 系统仿真学报, 2026, 38(6): 1669-1683.
[2]	李彦斌, 潘肇伦, 马新月, 宋明浩, 胡喻杰, 薛晓达. 基于PER-MADDPG算法的发电商容量市场交易策略[J]. 系统仿真学报, 2026, 38(5): 1408-1425.
[3]	章子瑶, 季云峰. 基于Transformer课程RL的机械臂接球策略仿真研究[J]. 系统仿真学报, 2026, 38(2): 321-331.
[4]	肖玮, 高甲博, 柯学良. 基于LDE-MADDPG算法的无人机集群编队集结控制策略[J]. 系统仿真学报, 2025, 37(9): 2335-2351.
[5]	祝靖宇, 张宏立, 匡敏驰, 史恒, 朱纪洪, 乔直, 周文卿. 稀疏奖励下基于课程学习的无人机空战仿真[J]. 系统仿真学报, 2024, 36(6): 1452-1467.
[6]	姜兆祯, 王文龙, 孙文祺. 基于改进RRT^*算法的无人艇路径规划快速求解算法[J]. 系统仿真学报, 2024, 36(4): 888-900.
[7]	余翔, 姜陈, 段思睿, 邓千锐. 改进A*算法和人工势场法的路径规划[J]. 系统仿真学报, 2024, 36(3): 782-794.
[8]	朱大奇, 朱婷婷, 颜明重. 基于改进神经网络的多AUV全覆盖路径规划[J]. 系统仿真学报, 2020, 32(8): 1505-1514.
[9]	曹梦龙, 李飞飞, 刘欣涛. 基于虚拟噪声模型补偿AUV导航算法研究[J]. 系统仿真学报, 2016, 28(1): 242-248.