Reinforcement Learning Modeling of Missile Penetration Decision Based on Combat Simulation

doi:10.16182/j.issn1004731x.joss.23-1397

Abstract

Abstract:

Penetration capability is a primary measure of missile systems. In response to the shortcomings of traditional knowledge-based decision-making methods that are difficult to adaptively evolve, an intelligent penetration decision-making based on combat simulation and DRL is proposed. A missile intelligent decision-making training environment is constructed based on the WESS system. Taking missile maneuver penetration decision-making as an example, a maneuver penetration decision-making network model is designed and trained based on the SAC-discrete algorithm and the test of intelligence is conducted. Experimental results show that the intelligent decision model derived from machine learning has a better combat outcome than traditional methods.

Key words: missile penetration, intelligent decision-making, DRL, combat simulation, WESS simulation system

CLC Number:

TP391.9

Zhang Bin, Lei Yonglin, Li Qun, Gao Yuan, Chen Yong, Zhu Jiajun, Bao Chenlong. Reinforcement Learning Modeling of Missile Penetration Decision Based on Combat Simulation[J]. Journal of System Simulation, 2025, 37(3): 763-774.

Figures/Tables 18

Fig. 1

Fig. 2

Fig. 3

Table 1

Input state space design of maneuver penetration decision network

状态变量	符号	数据类型	归一化处理
剩余燃料	$F r e m a i n$	double	当前剩余燃料量/总燃料数量
目标点距离	$d w t$	double	相对距离/地球半径
拦截弹相对位置	$x i, y i, z i$	double[3]	相对位置分量/弹头探测距离
拦截弹相对速度	$v x i, v y i, v z i$	double[3]	相对速度/拦截弹最大速度
速度与视线夹角	$q w i$	double	夹角/π
拦截弹速度与视线夹角	$q i w$	double	夹角/π

Table 1

Fig. 4

Table 2

Output action design of maneuver timing decision ANN

决策动作变量	取值范围	动作解析
机动开始距离 $d$	{1,2,…,10}	基本机动距离( $d m a x - d m i n$ )/10的倍数

Table 2

Table 3

Output action design of maneuver duration decision ANN

决策动作变量

取值范围

动作解析

弹头机动时长t_m,max

{1,2,…,10}

基本机动时长

$t m, m a x / 10$ 的倍数

Table 3

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Table 4

Scenario generation scheme of missile penetration scenario

场景生成变量	取值范围	水平数
海基反导拦截平台初始位置经度 $λ$ /(°)	[ $λ - 5, λ + 5$ ]	10
海基反导拦截平台初始位置纬度 $ϕ$ /(°)	[ $ϕ - 5, ϕ + 5$ ]	10
陆基拦截指控反应时间/s	[30, 90]	10
陆基拦截弹发射间隔/s	[5, 15]	10
陆基拦截瞄准点弧段位置	[0, 1]	10

Table 4

Table 5

Screening experimental design of maneuver penetration decision rules

决策规则变量	取值范围	水平数
机动开始距离 $x 1$ /km	[ $x 1 - 5, x 1 + 5$ ]	10
单次机动时长 $x 2$ (单次最大可机动时长的1/10)	[ $x 2 - 5, x 2 + 5$ ]	10

Table 5

Fig. 9

Fig. 10

Fig. 11

Table 6

Experimental design of new scenarios for intelligence test

实验变量	取值极差/(°)	水平数
海基拦截阵地经度	[ $λ 3 - 4, λ 3 + 5$ ]	10
海基拦截阵地纬度	[ $ϕ 3 - 4, ϕ 3 + 5$ ]	10

Table 6

Table 7

References 14

1	程传浩, 王瑞臣, 路德信. 弹道导弹的突防与拦截[J]. 现代防御技术, 2002, 30(2): 20-23.
	Cheng Chuanhao, Wang Ruichen, Lu Dexin. Penetration and Interception of Ballistic Missile[J]. Modern Defense Technology, 2002, 30(2): 20-23.
2	文苏丽, 陈琦, 苏鑫鑫, 等. 智能化导弹与导弹智能化研究[J]. 战术导弹技术, 2015(6): 21-26.
	Wen Suli, Chen Qi, Su Xinxin, et al. Study of Intelligent Missile and Missile Intelligentization[J]. Tactical Missile Technology, 2015(6): 21-26.
3	Diehl Douglas D, Lt U S N. How to Optimize Joint Theater Ballistic Missile Defense [EB/OL]. (2004-03) [2023-10-03]. .
4	Sutton R S, Barto A G. Reinforcement Learning: An Introduction[M]. Cambridge: MIT Press, 1998.
5	刘全, 翟建伟, 章宗长, 等. 深度强化学习综述[J]. 计算机学报, 2018, 41(1): 1-27.
	Liu Quan, Zhai Jianwei, Zhang Zongchang, et al. A Survey on Deep Reinforcement Learning[J]. Chinese Journal of Computers, 2018, 41(1): 1-27.
6	Wang Xu, Wang Sen, Liang Xingxing, et al. Deep Reinforcement Learning: A Survey[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(4): 5064-5078.
7	Qiu Xiaoqi, Gao Changsheng, Jing Wuxing. Maneuvering Penetration Strategies of Ballistic Missiles Based on Deep Reinforcement Learning[J]. Proceedings of the Institution of Mechanical Engineers, Part G: Journal of Aerospace Engineering, 2022, 236(16): 3494-3504.
8	Shalumov Vitaly. Cooperative Online Guide-launch-guide Policy in a Target-missile-defender Engagement Using Deep Reinforcement Learning[J]. Aerospace Science and Technology, 2020, 104: 105996.
9	Gaudet B, Furfaro R, Linares R, et al. Reinforcement Metalearning for Interception of Maneuvering Exoatmospheric Targets with Parasitic Attitude Loop[J]. Journal of Spacecraft and Rockets, 2021, 58(2): 386-399.
10	左家亮, 杨任农, 张滢, 等. 基于启发式强化学习的空战机动智能决策[J]. 航空学报, 2017, 38(10): 212-225.
	Zuo Jialiang, Yang Rennong, Zhang Ying, et al. Intelligent Decision-making in Air Combat Maneuvering Based on Heuristic Reinforcement Learning[J]. Acta Aeronautica et Astronautica Sinica, 2017, 38(10): 212-225.
11	Zhang Liang, Xu Jia, Gold Dara, et al. Air Dominance Through Machine Learning: A Preliminary Exploration of Artificial Intelligence-assisted Mission Planning[EB/OL]. (2020-05-29) [2023-10-05]. .
12	Arulkumaran K, Cully A, Togelius J. AlphaStar: an Evolutionary Computation Perspective[C]//Proceedings of the Genetic and Evolutionary Computation Conference Companion. New York: ACM, 2019: 314-315.
13	雷永林, 姚剑, 朱宁, 等. 武器装备作战效能仿真系统WESS[J]. 系统仿真学报, 2017, 29(6): 1244-1252.
	Lei Yonglin, Yao Jian, Zhu Ning, et al. Weapon Effectiveness Simulation System (WESS)[J]. Journal of System Simulation, 2017, 29(6): 1244-1252.
14	Christodoulou P. Soft Actor-critic for Discrete Action Settings[EB/OL]. (2019-10-18) [2023-08-06]. .

机动策略	样本总数	成功突防次数	平均突防概率
最优规则策略	1 000	826	0.826
智能机动决策策略	1 000	952	0.952

[1]	Wang He, Xu Jianing, Yan Guangyu. Research on Pedestrian Avoidance Strategy for AGV Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(3): 595-606.
[2]	Huang Sijin, Wen Jia, Chen Zheyi. Intelligent Service Migration towards MEC-based IoV Systems [J]. Journal of System Simulation, 2025, 37(2): 379-391.
[3]	Fei Shuaidi, Cai Changlong, Liu Fei, Chen Minghui, Liu Xiaoming. Research on the Target Allocation Method for Air Defense and Anti-missile Defense of Naval Ships [J]. Journal of System Simulation, 2025, 37(2): 508-516.
[4]	Zhang Peng, Feng Ke, Gong Jiancheng, Yang Xiaoqiang, Shen Jinxing. Combat Effectiveness Evaluation of Air Defense Missile Weapon System Based on RBF Neural Network [J]. Journal of System Simulation, 2025, 37(2): 529-540.
[5]	Jiang Jiachen, Jia Zhengxuan, Xu Zhao, Lin Tingyu, Zhao Pengpeng, Ou Yiming. Decision Modeling and Solution Based on Game Adversarial Complex Systems [J]. Journal of System Simulation, 2025, 37(1): 66-78.
[6]	Li Chao, Li Jiabao, Ding Caichang, Ye Zhiwei, Zuo Fangwei. Edge Surveillance Task Offloading and Resource Allocation Algorithm Based on DRL [J]. Journal of System Simulation, 2024, 36(9): 2113-2126.
[7]	Zhu Zilu, Liu Yongkui, Zhang Lin, Wang Lihui, Lin Tingyu. Simulation of Robotic Peg-in-hole Assembly Strategy Based on DRL [J]. Journal of System Simulation, 2024, 36(6): 1414-1424.
[8]	Wang Hongjun, Lin Junqiang, Zou Xiangjun, Zhang Po, Zhou Mingxuan, Zou Weirui, Tang Yunchao, Luo Lufeng. Construction of a Virtual Interactive System for Orchards Based on Digital Twin [J]. Journal of System Simulation, 2024, 36(6): 1493-1508.
[9]	Wang Yuan, Xu Lin, Gong Xiaoze, Zhang Yongliang, Wang Yongli. Gradient-based Deep Reinforcement Learning Interpretation Methods [J]. Journal of System Simulation, 2024, 36(5): 1130-1140.
[10]	Liu Fei, Lai Peng, Lu Yingbo, Wang Min, Lu Zhifeng. Research on Hybrid Experimental Scheme Design for Combat Simulation [J]. Journal of System Simulation, 2024, 36(3): 735-742.
[11]	Qin Baoxin, Zhang Yuxiao, Wu Sirui, Cao Weichong, Li Zhan. Intelligent Optimization of Coal Terminal Unloading Scheduling Based on Improved D3QN Algorithm [J]. Journal of System Simulation, 2024, 36(3): 770-781.
[12]	Zhao Yingying, Dong Pusen, Zhu Tianchen, Li Fan, Su Yun, Tai Zhenying, Sun Qingyun, Fan Hang. Efficiency Optimization Method for Data Sampling in Power Grid Topology Scheduling Simulation [J]. Journal of System Simulation, 2024, 36(2): 283-295.
[13]	Wang Xinpeng, Fu Huiqiao, Deng Guizhou, Tang Kaiqiang, Chen Chunlin, Liu Canghai. Research on Motion Planning of Hexapod Robot Based on DRL and Free Gait [J]. Journal of System Simulation, 2024, 36(2): 373-384.
[14]	Pan Hainan, Chen Bailiang, Huang Kaihong, Ren Junkai, Cheng Chuang, Lu Huimin, Zhang Hui. Flipper Control Method for Tracked Robot Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(2): 405-414.
[15]	Chen Shikang, Wang Zhimin, Liang Biao, Gu Kunren, Song Keman, Gao Yuan. Modeling Method of Blue Army Warship Formation Air Defense Command and Control for System-of-systems Simulation [J]. Journal of System Simulation, 2024, 36(10): 2246-2256.