多智能体协同决策仿真平台研究与开发

doi:10.16182/j.issn1004731x.joss.23-FZ0821

系统仿真学报 ›› 2023, Vol. 35 ›› Issue (12): 2669-2679.doi: 10.16182/j.issn1004731x.joss.23-FZ0821

多智能体协同决策仿真平台研究与开发

成城¹(), 陈智杰¹, 郭子铭², 李妮¹()

^1.北京航空航天大学自动化科学与电气工程学院，北京 100191
^2.中国航空工业集团公司沈阳飞机设计研究所，辽宁沈阳 110035

收稿日期:2023-07-03 修回日期:2023-10-01 出版日期:2023-12-15 发布日期:2023-12-12
通讯作者: 李妮 E-mail:sy2303801@buaa.edu.cn;lini@buaa.edu.cn
第一作者简介:成城(2001-)，男，硕士生，研究方向为系统仿真与智慧制造。E-mail：sy2303801@buaa.edu.cn

Research and Development of Simulation Training Platform for Multi-agent Collaborative Decision-making

Cheng Cheng¹(), Chen Zhijie¹, Guo Ziming², Li Ni¹()

^1.School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China
^2.AVIC Shenyang Aircraft Design and Research Institute, Shenyang 110035, China

Received:2023-07-03 Revised:2023-10-01 Online:2023-12-15 Published:2023-12-12
Contact: Li Ni E-mail:sy2303801@buaa.edu.cn;lini@buaa.edu.cn

摘要/Abstract

摘要：

强化学习仿真平台为强化学习提供交互和训练的环境。为了使仿真平台兼容多智能体强化学习算法，满足军事领域仿真的需求，提炼多智能体强化学习算法中的相似流程，设计统一接口，将多种不同类型深度强化学习算法在仿真平台进行嵌入验证；优化仿真平台后端服务框架以加速算法模型的训练过程。实验结果表明：在仿真平台中统一接口规范，能够兼容多种不同类型的多智能体强化学习算法，显著提升了后端服务框架重构和参数量化后算法训练效率。

关键词: 人工智能, 多智能体, 强化学习, 虚拟仿真, 训练加速

Abstract:

Reinforcement learning simulation platform can be an interactive and training environment for reinforcement learning. In order to make the simulation platform compatible with the multi-agent reinforcement learning algorithms and meet the needs of simulation in military field, the similar processes in multi-agent reinforcement learning algorithms are refined and a unified interface is designed to embed and verify different types of deep reinforcement learning algorithms on the simulation platform and to optimize the back-end service of the simulation platform to accelerate the training process of the algorithm model. The experimental results show that, by unifing the interface, the simulation platform can be compatible with many different types of multi-agent reinforcement learning algorithms, and the algorithm training efficiency can be significantly improved after the back-end service framework reconstruction and parameter quantization.

Key words: artificial intelligence, multi-agent, reinforcement learning, virtual simulation, training acceleration

中图分类号:

TP391.9

成城,陈智杰,郭子铭等 . 多智能体协同决策仿真平台研究与开发[J]. 系统仿真学报, 2023, 35(12): 2669-2679.

Cheng Cheng,Chen Zhijie,Guo Ziming,et al . Research and Development of Simulation Training Platform for Multi-agent Collaborative Decision-making[J]. Journal of System Simulation, 2023, 35(12): 2669-2679.

图/表 17

图1

图2

图3

图4

图5

图6

表1

表2

图7

图8

图9

图10

表3

图11

图12

图13

图14

参考文献 11

1	邹启杰, 蒋亚军, 高兵, 等. 协作多智能体深度强化学习研究综述[J]. 航空兵器, 2022, 29(6): 78-88.
	Zou Qijie, Jiang Yajun, Gao Bing, et al. An Overview of Cooperative Multi-agent Deep Reinforcement Learning[J]. Aero Weaponry, 2022, 29(6): 78-88.
2	Christiano P F, Leike J, Brown T W, et al. Deep Reinforcement Learning from Human Preferences[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates Inc., 2017: 4302-4310.
3	Brockman G, Cheung V, Pettersson L, et al. OpenAI Gym[EB/OL]. (2016-06-05) [2023-05-21]. .
4	Alibaba. Gym StarCraft[EB/OL]. [2023-05-16]. .
5	周亮, 王震, 王冠. 远程过程调用技术在分布式软件系统中的应用[J]. 航空电子技术, 2020, 51(4): 47-52.
	Zhou Liang, Wang Zhen, Wang Guan. Application of Remote Procedure Calling Technology in Distributed Software System[J]. Avionics Technology, 2020, 51(4): 47-52.
6	张红杰. 深度强化学习训练与推理的性能优化[D]. 合肥: 中国科学技术大学, 2021.
	Zhang Hongjie. Performance Optimization of Training and Inference of Deep Reinforcement Learning[D]. Hefei: University of Science and Technology of China, 2021.
7	Abadi Martín, Agarwal A, Barham P, et al. TensorFlow: Large-scale Machine Learning on Heterogeneous Distributed Systems[EB/OL]. (2016-03-16) [2023-04-23]. .
8	Lowe R, Wu Yi, Tamar A, et al. Multi-agent Actor-critic for Mixed Cooperative-competitive Environments[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates Inc., 2017, 30: 6382-6393.
9	Konda V R, Tsitsiklis J N. Actor-critic Algorithms[C]//Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2000: 1008-1014.
10	Terry J, Black B, Grammel N, et al. PettingZoo: Gym for Multi-agent Reinforcement Learning[C]//Advances in Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates, Inc., 2021: 15032-15043.
11	王壮, 艾毅, 文旭光, 等. 航空器智能引导机动决策奖励重塑方法[J]. 科学技术与工程, 2023, 23(8): 3535-3543.
	Wang Zhuang, Ai Yi, Wen Xuguang, et al. Reward Shaping for Intelligent Maneuver Decision Generation in Aircraft Guidance[J]. Science Technology and Engineering, 2023, 23(8): 3535-3543.

序号	原同步框架训练耗时	优化后框架训练耗时
平均值	83.75	59.40
1	83.98	59.41
2	83.56	59.21
3	83.71	59.59

智能体类型	算法模型	奖励设计	无人机/ 舰船数量
无人机	MADDPG	势函数方法设计	2~4
舰船	―	―	1~2

[1]	郭润夏, 王一府. 以维修间隔利用率最优为目标的飞机派遣方法[J]. 系统仿真学报, 2023, 35(9): 1985-1999.
[2]	林俊强, 王红军, 邹湘军, 张坡, 李承恩, 周益鹏, 姚书杰. 基于DPPO的移动采摘机器人避障路径规划及仿真[J]. 系统仿真学报, 2023, 35(8): 1692-1704.
[3]	刘家义, 王刚, 付强, 郭相科, 王思远. 基于分配策略优化算法的智能防空任务分配[J]. 系统仿真学报, 2023, 35(8): 1705-1716.
[4]	杨来义, 毕敬, 苑海涛. 基于SAC算法的移动机器人智能路径规划[J]. 系统仿真学报, 2023, 35(8): 1726-1736.
[5]	马苗苗, 董利鹏, 刘向杰. 基于Q-learning算法的多智能体微电网能量管理策略[J]. 系统仿真学报, 2023, 35(7): 1487-1496.
[6]	徐健, 宋鑫, 刘秀平, 陈博, 闫焕营. 基于数字孪生的装配机器人建模及系统实现[J]. 系统仿真学报, 2023, 35(7): 1497-1507.
[7]	李成兵, 李云飞, 吴鹏. 基于多智能体的城市群客运网络脆弱性动态仿真[J]. 系统仿真学报, 2023, 35(6): 1183-1190.
[8]	丁飞, 沙宇晨, 洪莹, 蒯晓, 张登银. 智能网联汽车计算卸载与边缘缓存联合优化策略[J]. 系统仿真学报, 2023, 35(6): 1203-1214.
[9]	戴宇轩, 崔承刚. 基于深度强化学习的Boost变换器控制策略[J]. 系统仿真学报, 2023, 35(5): 1109-1119.
[10]	刘延东, 黄高翔, 陈文. 基于增强心理行为异质性的改进社会力模型[J]. 系统仿真学报, 2023, 35(5): 1120-1130.
[11]	徐浩添, 秦龙, 曾俊杰, 胡越, 张琪. 基于深度强化学习的对手建模方法研究综述[J]. 系统仿真学报, 2023, 35(4): 671-694.
[12]	石鼎, 燕雪峰, 宫丽娜, 张静宣, 关东海, 魏明强. 强化学习驱动的海战场多智能体协同作战仿真算法[J]. 系统仿真学报, 2023, 35(4): 786-796.
[13]	陈佳昕, 周国辉, 杨建柏. 基于虚拟仿真方法的手部姿态数据集构建技术[J]. 系统仿真学报, 2023, 35(4): 862-870.
[14]	袁国栋, 何明, 马子玉, 张伟士, 刘学达, 李伟. 基于K-means聚类的多智能体跟随多领导者算法[J]. 系统仿真学报, 2023, 35(3): 616-622.
[15]	史佳洁, 杨鹏, 皮雁南. 基于机器学习的地铁行人流在线优化控制研究[J]. 系统仿真学报, 2023, 35(2): 386-395.

多智能体协同决策仿真平台研究与开发

Research and Development of Simulation Training Platform for Multi-agent Collaborative Decision-making

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献 11

相关文章 15

编辑推荐

Metrics

本文评价