Simulation Verification and Decision-making Key Technologies of Unmanned Swarm Game Confrontation: A Survey

doi:10.16182/j.issn1004731x.joss.23-0072

Abstract

Abstract:

Unmanned swarm game confrontation is a new combat mode and plays a crucial role in intelligent warfare. Its core is the autonomous generation of a series of game confrontation decision sequences to "empower" the swarm. The progress of system simulation verification for the unmanned swarm game confrontation is analyzed. The key technologies of the autonomous decision-making are discussed from three aspects, technology based on expert systems and game theory, technology based on swarm intelligence and optimization theory, and technology based on neural networks and reinforcement learning. The key technology research conducted by the author's team on the autonomous decision-making is presented. The development directions of the unmanned swarm game confrontation are proposed.

Key words: unmanned swarm, game confrontation, autonomous decision-making, system simulation validation

CLC Number:

Liang Xiaolong, Yang Aiwu, Zhang Jiaqiang, Hou Yueqi, Wang Ning, Huang Xiao, Gong Junbin. Simulation Verification and Decision-making Key Technologies of Unmanned Swarm Game Confrontation: A Survey[J]. Journal of System Simulation, 2024, 36(4): 805-816.

Figures/Tables 9

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Table 1

Fig. 8

References 35

1	梁晓龙, 胡利平, 张佳强, 等. 航空集群自主空战研究进展[J]. 科技导报, 2020, 38(15): 74-88.
	Liang Xiaolong, Hu Liping, Zhang Jiaqiang, et al. Research Status and Prospect of Aircraft Swarms Autonomou Air Combat[J]. Science & Technology Review, 2020, 38(15): 74-88.
2	梁晓龙, 何吕龙, 张佳强, 等. 航空集群构型控制及其演化方法[J]. 中国科学(技术科学), 2019, 49(3): 277-287.
	Liang Xiaolong, He Lülong, Zhang Jiaqiang, et al. Configuration Control and Evolutionary Mechanism of Aircraft Swarm[J]. Scientia Sinica(Technologica), 2019, 49(3): 277-287.
3	Zhu Xiaoning. Analysis of Military Application of UAV Swarm Technology[C]//2020 3rd International Conference on Unmanned Systems (ICUS). Piscataway, NJ, USA: IEEE, 2020: 1200-1204.
4	刘雷, 刘大卫, 王晓光, 等. 无人机集群与反无人机集群发展现状及展望[J]. 航空学报, 2022, 43(增1): 4-20.
	Liu Lei, Liu Dawei, Wang Xiaoguang, et al. Development Status and Outlook of UAV Clusters and Anti-UAV Clusters[J]. Acta Aeronautica et Astronautica Sinica, 2022, 43(S1): 4-20.
5	Gulden T R, Lamb J, Hagen J, et al. Modeling Rapidly Composable, Heterogeneous, and Fractionated Forces: Findings on Mosaic Warfare from an Agent-based Model[EB/OL]. [2023-03-20]. .
6	O'Donoughue N A, McBirney S, Persons B. Distributed Kill Chains: Drawing Insights for Mosaic Warfare from the Immune System and from the Navy[EB/OL]. [2023-03-20]. .
7	Pope A P, Ide J S, Mićović Daria, et al. Hierarchical Reinforcement Learning for Air-to-air Combat[C]//2021 International Conference on Unmanned Aircraft Systems (ICUAS). Piscataway, NJ, USA: IEEE, 2021: 275-284.
8	周光霞, 周方. 美军人工智能空战系统阿尔法初探[C]//第六届中国指挥控制大会论文集(上册). 北京: 电子工业出版社, 2018: 66-70.
9	Ernest N, Carroll D, Schumacher C, et al. Genetic Fuzzy Based Artificial Intelligence for Unmanned Combat Aerial Vehicle Control in Simulated Air Combat Missions[J]. Journal of Defense Management, 2016, 6(1): 1000144.
10	Ernest N, Cohen K, Kivelevitch E, et al. Genetic Fuzzy Trees and Their Application Towards Autonomous Training and Control of a Squadron of Unmanned Combat Aerial Vehicles[J]. Unmanned Systems, 2015, 3(3): 185-204.
11	Fan D D, Theodorou E A, Reeder J. Model-based Stochastic Search for Large Scale Optimization of Multi-agent UAV Swarms[C]//2018 IEEE Symposium Series on Computational Intelligence (SSCI). Piscataway, NJ, USA: IEEE, 2018: 2216-2222.
12	Ma Yingying, Wang Guoqiang, Hu Xiaoxuan, et al. Cooperative Occupancy Decision Making of Multi-UAV in Beyond-visual-range Air Combat: A Game Theory Approach[J]. IEEE Access, 2020, 8: 11624-11634.
13	Li Shouyi, Chen Mou, Wang Yuhui, et al. Air Combat Decision-making of Multiple UCAVs Based on Constraint Strategy Games[J]. Defence Technology, 2022, 18(3): 368-383.
14	Ji Xiang, Zhang Wanpeng, Xiang Fengtao, et al. A Swarm Confrontation Method Based on Lanchester Law and Nash Equilibrium[J]. Electronics, 2022, 11(6): 896.
15	Liu Fei, Dong Xiwang, Yu Jianglong, et al. Distributed Nash Equilibrium Seeking of N-coalition Noncooperative Games with Application to UAV Swarms[J]. IEEE Transactions on Network Science and Engineering, 2022, 9(4): 2392-2405.
16	Hu Wenfeng, Zhang Qiang, Mao Yaqin. Component-Based Hierarchical State Machine-A Reusable and Flexible Game AI Technology[C]//2011 6th IEEE Joint International Information Technology and Artificial Intelligence Conference. Piscataway, NJ, USA: IEEE, 2011: 319-324.
17	Vannucchi Claudia, Diletta Romana Cacciagrano, Corradini Flavio, et al. A Formal Model for Event-condition-action Rules in Intelligent Environments[C]//12th International Conference on Intelligent Environments-5th International Workshop on the Reliability of Intelligent Environments (WoRIE 2016). Amsterdam, Netherlands: IOS Press, 2016: 56-65.
18	Duan Haibin, Li Pei, Yu Yaxiang. A Predator-prey Particle Swarm Optimization Approach to Multiple UCAV Air Combat Modeled by Dynamic Game Theory[J]. IEEE/CAA Journal of Automatica Sinica, 2015, 2(1): 11-18.
19	Gao Chunqing, Kou Yingxin, Li You, et al. Multi-objective Weapon Target Assignment Based on D-NSGA-III-A[J]. IEEE Access, 2019, 7: 50240-50254.
20	Xing Dongjing, Zhen Ziyang, Gong Huajun. Offense-defense Confrontation Decision Making for Dynamic UAV Swarm Versus UAV Swarm[J]. Proceedings of the Institution of Mechanical Engineers, Part G: Journal of Aerospace Engineering, 2019, 233(15): 5689-5702.
21	Zhen Ziyang, Wen Liangdong, Wang Bolan, et al. Improved Contract Network Protocol Algorithm Based Cooperative Target Allocation of Heterogeneous UAV Swarm[J]. Aerospace Science and Technology, 2021, 119: 107054.
22	Wang Chengcai, Wu Ao, Hou Yueqi, et al. Optimal Deployment of Swarm Positions in Cooperative Interception of Multiple UAV Swarms[J]. Digital Communications and Networks, 2023, 9(2): 567-579.
23	丁超, 魏瑞轩, 周凯. 基于时域映射的多无人机系统给定时间分布式最优集结[J]. 北京航空航天大学学报, 2021, 47(2): 315-322.
	Ding Chao, Wei Ruixuan, Zhou Kai. Distributed Optimal Rendezvous of Multi-UAV Systems in Prescribed Time Based on Time-domain Mapping[J]. Journal of Beijing University of Aeronautics and Astronautics, 2021, 47(2): 315-322.
24	Hou Teng Teck, Hwee Tan Ah, Yuan Sin Tan, et al. Self-organizing Neural Networks for Learning Air Combat Maneuvers[C]//The 2012 International Joint Conference on Neural Networks (IJCNN). Piscataway, NJ, USA: IEEE, 2012: 1-8.
25	Wang Luhe, Hu Jinwen, Xu Zhao, et al. Autonomous Maneuver Strategy of Swarm Air Combat Based on DDPG[J]. Autonomous Intelligent Systems, 2021, 1(1): 15.
26	Hu Jinwen, Wang Luhe, Hu Tianmi, et al. Autonomous Maneuver Decision Making of Dual-UAV Cooperative Air Combat Based on Deep Reinforcement Learning[J]. Electronics, 2022, 11(3): 467.
27	Zhang Jiandong, Yang Qiming, Shi Guoqing, et al. UAV Cooperative Air Combat Maneuver Decision Based on Multi-agent Reinforcement Learning[J]. Journal of Systems Engineering and Electronics, 2021, 32(6): 1421-1438.
28	Xu Dan, Chen Gang. The Research on Intelligent Cooperative Combat of UAV Cluster with Multi-agent Reinforcement Learning[J]. Aerospace Systems, 2022, 5(1): 107-121.
29	Xuan Shuzhe, Ke Liangjun. UAV Swarm Attack-defense Confrontation Based on Multi-agent Reinforcement Learning[C]//Advances in Guidance, Navigation and Control. Singapore: Springer Singapore, 2022: 5599-5608.
30	Wang Baolai, Li Shengang, Gao Xianzhong, et al. Weighted Mean Field Reinforcement Learning for Large-scale UAV Swarm Confrontation[J]. Applied Intelligence, 2023, 53(5): 5274-5289.
31	Xiang Lei, Xie Tao. Research on UAV Swarm Confrontation Task Based on MADDPG Algorithm[C]//2020 5th International Conference on Mechanical, Control and Computer Engineering (ICMCCE). Piscataway, NJ, USA: IEEE, 2020: 1513-1518.
32	Källström Johan, Heintz Fredrik. Agent Coordination in Air Combat Simulation Using Multi-agent Deep Reinforcement Learning[C]//2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). Piscataway, NJ, USA: IEEE, 2020: 2157-2164.
33	Wang Baolai, Li Shengang, Gao Xianzhong, et al. UAV Swarm Confrontation Using Hierarchical Multiagent Reinforcement Learning[J]. International Journal of Aerospace Engineering, 2021, 2021: 3360116.
34	Narvekar S, Peng Bei, Leonetti M, et al. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey[J]. The Journal of Machine Learning Research, 2020, 21(1): 181.
35	Zhu Zhuangdi, Lin Kaixiang, Jain A K, et al. Transfer Learning in Deep Reinforcement Learning: A Survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(11): 13344-13362.

存在的问题	文献
训练效率低，如何引入专家知识来提高训练效率	[11, 25-27, 31]
作战目标属于稀疏奖励或延迟奖励问题	[26-28, 33]
状态空间和动作空间维度大，探索采样效率低	[30, 32-33]
策略网络泛化性、鲁棒性不足	[30, 32]
局部观测信息下作战决策	[27-29]

[1]	Wang Jipeng, Zhang Xing, Wu Hao, Gu Yu, Yang Huijie. Research on Theoretical Framework of Simulative Experiment Evalution for Intelligent Unmanned Swarm Cooperation [J]. Journal of System Simulation, 2024, 36(4): 795-804.
[2]	Wang Ning, Liang Xiaolong, Zhang Jiaqiang, Hou Yueqi, Yang Aiwu. Research on Cross-domain Unmanned Swarm Cooperative Anti-submarine Search Method [J]. Journal of System Simulation, 2024, 36(4): 817-824.
[3]	Yang Huijie, Xiao Taoshun, Wu Chen, Guo Lingfeng. Research on Integration of Virtual-real Hybrid Simulation Experiment Environment for Unmanned Swarm [J]. Journal of System Simulation, 2024, 36(4): 825-833.
[4]	Liqiang Guo, Liang Ma, Hui Zhang, Jing Yang, Xueman Fan, Zhuo Cheng. Research on Modeling and Optimization Method of Torpedo Anti-jamming Attack Based on Game Confrontation [J]. Journal of System Simulation, 2023, 35(8): 1814-1823.
[5]	Zhiqiang Li, Yuanlong Li, Laixiang Yin, Xiangping Ma. Research on Unmanned Swarm Combat System Adaptive Evolution Model Simulation [J]. Journal of System Simulation, 2023, 35(4): 878-886.
[6]	Xin Zhou, Weiping Wang, Yifan Zhu, Tao Wang, Tian Jing. An Unmanned Swarm Search Method Based on Human-Robot Cooperation [J]. Journal of System Simulation, 2022, 34(4): 735-744.