A DRL⁃based Approach for Distributed Equipment Nodes Selection

doi:10.16182/j.issn1004731x.joss.24-0222

Abstract

Abstract:

Aiming at the problem of insufficient solution speed and poor generalization of traditional algorithms in large-scale scenarios, this paper intelligently solves the large-scale distributed equipment system preference problem based on deep reinforcement learning. According to the characteristics of distributed equipment system combat, using the complex network to its graph form modeling, and based on the attention mechanism to the equipment between the connecting edge relationship for the characterization, in order to build a distributed equipment system digital simulation environment. Simulation results show that compared with the genetic evolutionary algorithm, the obtained model has obvious advantages in terms of solution time and generalization, which effectively improves the performance of distributed equipment nodes combination selection.

Key words: DRL, graph neural network, attention mechanism, complex network, distributed system of equipment

CLC Number:

TP391.9

Wang Ziyi, Zhang Kai, Qian Dianwei, Liu Yuzhen. A DRL⁃based Approach for Distributed Equipment Nodes Selection[J]. Journal of System Simulation, 2025, 37(6): 1565-1573.

Figures/Tables 12

Table 1

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Table 2

Table 3

Algorithm training hyperparameters

超参数	值	超参数	值
训练总次数	100	学习率	$1 × 10 - 4$
训练批次	640	注意力头数	8
样本数量	64 000	隐藏层维度	128
编码器层数	3

Table 3

Fig. 6

Table 4

Table 5

Table 6

References 19

1	赵仁星, 王玲, 冯明月, 等. "马赛克战"作战概念构想及对策分析[J]. 空天防御, 2021, 4(3): 48-54.
	Zhao Renxing, Wang Ling, Feng Mingyue, et al. Analysis of Combat Concept of "Mosaic Warfare" and Countermeasures[J]. Air & Space Defense, 2021, 4(3): 48-54.
2	郭行, 符文星, 闫杰. 浅析美军马赛克战作战概念及启示[J]. 无人系统技术, 2020, 3(6): 92-106.
	Guo Hang, Fu Wenxing, Yan Jie. Analysis and Inspiration of the U.S. Force's Concept of Mosaic Warfare[J]. Unmanned Systems Technology, 2020, 3(6): 92-106.
3	邵晨曦, 陈小齐, 王行甫, 等. 基于数据场的复杂网络节点影响力建模与仿真[J]. 系统仿真学报, 2020, 32(7): 1257-1266.
	Shao Chenxi, Chen Xiaoqi, Wang Xingfu, et al. Modeling and Simulation on Influence of Complex Network Nodes Based on Data Field in[J]. Journal of System Simulation, 2020, 32(7): 1257-1266.
4	夏博远, 杨克巍, 杨志伟, 等. 基于杀伤网评估的装备组合多目标优化[J]. 系统工程与电子技术, 2021, 43(2): 399-409.
	Xia Boyuan, Yang Kewei, Yang Zhiwei, et al. Multi-objective Optimization of Equipment Portfolio Based on Kill-web Evaluation[J]. Systems Engineering and Electronics, 2021, 43(2): 399-409.
5	Li Yaodong, Xiao Bing, Yu Jingtao. Modeling and Analysis of Mosaic Warfare System Based on Complex Network Theory[C]//International Conference on Intelligent Systems, Communications, and Computer Networks (ISCCN 2022). Bellingham: SPIE, 2022: 1233223.
6	李凯文, 张涛, 王锐, 等. 基于DRL的组合优化研究进展[J]. 自动化学报, 2021, 47(11): 2521-2537.
	Li Kaiwen, Zhang Tao, Wang Rui, et al. Research Reviews of Combinatorial Optimization Methods Based on Deep Reinforcement Learning[J]. Acta Automatica Sinica, 2021, 47(11): 2521-2537.
7	Vinyals O, Fortunato M, Jaitly N. Pointer Networks[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2015: 2692-2700.
8	Bello I, Pham H, Le Q V, et al. Neural Combinatorial Optimization with Reinforcement Learning[EB/OL]. (2017-01-12) [2023-12-22]. .
9	Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2017: 6000-6010.
10	Kool Wouter, van Hoof Herke, Welling Max. Attention, Learn to Solve Routing Problems![EB/OL]. (2019-02-07) [2023-12-22]. .
11	Peng Yun, Choi B, Xu Jianliang. Graph Learning for Combinatorial Optimization: A Survey of State-of-the-art[J]. Data Science and Engineering, 2021, 6(2): 119-141.
12	Scarselli Franco, Gori Marco, Ah Chung Tsoi, et al. The Graph Neural Network Model[J]. IEEE Transactions on Neural Networks, 2009, 20(1): 61-80.
13	Kipf Thomas N, Welling Max. Semi-supervised Classification with Graph Convolutional Networks[EB/OL]. (2017-02-22) [2023-12-22]. .
14	Veličković Petar, Cucurull Guillem, Casanova Arantxa, et al. Graph Attention Networks[EB/OL]. (2018-02-04) [2023-12-22]. .
15	Ma Qiang, Ge Suwen, He Danyang, et al. Combinatorial Optimization by Graph Pointer Networks and Hierarchical Reinforcement Learning[C]//Proceedings of the 1st International Workshop on Deep Learning on Graphs: Methodologies and Applications. New York: AAAI, 2020.
16	Lei Kun, Guo Peng, Wang Yi, et al. Solve Routing Problems with a Residual Edge-graph Attention Neural Network[J]. Neurocomputing, 2022, 508: 79-98.
17	王耀祖, 尚柏林, 宋笔锋, 等. 基于杀伤链的作战体系网络关键节点识别方法[J]. 系统工程与电子技术, 2023, 45(3): 736-744.
	Wang Yaozu, Shang Bailin, Song Bifeng, et al. Identification Method of Key Node in Operational System-of-systems Network Based on Kill Chain[J]. Systems Engineering and Electronics, 2023, 45(3): 736-744.
18	Hu Yaru, Zheng Jinhua, Zou Juan, et al. Dynamic Multi-objective Optimization Algorithm Based Decomposition and Preference[J]. Information Sciences, 2021, 571: 175-190.
19	Jiang Jing, Han Fei, Ling Qinghua, et al. Efficient Network Architecture Search via Multiobjective Particle Swarm Optimization Based on Decomposition[J]. Neural Networks, 2020, 123: 305-316.

连边类型	具体说明
目标节点-侦察节点	目标层与侦察层间连边，表示侦察装备对目标具备侦察能力
侦察节点-指控节点	侦察层与指控层间连边，表示侦察装备能将信息发送至指挥所
指控节点-攻击节点	指控层与攻击层间连边，表示指挥所能将指令下达至装备
攻击节点-目标节点	攻击层与目标层间连边，表示此装备具备攻击目标节点的能力

算法	100节点		150节点		200节点
算法	总指标值	t/s	总指标值	t/s	总指标值	t/s
GAT-RL	0.236 1	0.25	0.224 3	0.31	0.247 9	0.60
GA	0.269 6	751.07	0.326 8	1 763.26	0.389 3	2 892.24

总节点	目标节点	侦察节点	指控节点	攻击节点
90	5	45	10	30
110	5	55	10	40

算法	90节点		110节点		150节点		200节点
算法	总指标值	t/s	总指标值	t/s	总指标值	t/s	总指标值	t/s
GAT-RL	0.261 3	0.60	0.255 6	0.59	0.224 3	0.31	0.247 9	0.60
100节点模型泛化	0.255 4	0.58	0.254 2	0.60	0.236 9	0.26	0.257 1	0.60
GA	0.266 8	515.95	0.280 1	797.53	0.326 8	1 763.26	0.389 3	2 892.24

[1]	Wu Guohua, Zeng Jiaheng, Wang Dezhi, Zheng Long, Zou Wei. A Quadrotor Trajectory Tracking Control Method Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(5): 1169-1187.
[2]	Wang Xiang, Tan Guozhen. Research on Decision-making of Autonomous Driving in Highway Environment Based on Knowledge and Large Language Model [J]. Journal of System Simulation, 2025, 37(5): 1246-1255.
[3]	Li Jie, Liu Yang, Li Liang, Su Bengan, Wei Jialong, Zhou Guangda, Shi Yanmin, Zhao Zhen. Remote Sensing Small Object Detection Based on Cross-stage Two-branch Feature Aggregation [J]. Journal of System Simulation, 2025, 37(4): 1025-1040.
[4]	Zhang Sen, Dai Qiangqiang. UAV Path Planning Based on Improved Deep Deterministic Policy Gradients [J]. Journal of System Simulation, 2025, 37(4): 875-881.
[5]	Li Min, Zhang Sen, Zeng Xiangguang, Wang Gang, Zhang Tongwei, Xie Dijie, Ren Wenzhe, Zhang Tao. Trajectory Planning of Quadruped Robot Over Obstacle with Single Leg Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(4): 895-909.
[6]	Nie Shida, Li Chengbing, He Bowei, Li Xintao. Research on the Resilience of Integrated Urban Passenger Transport Network in Urban Agglomerations Considering the Intra-urban Service Network [J]. Journal of System Simulation, 2025, 37(4): 933-942.
[7]	Zheng Lanyue, Zhang Yujie. Traffic Signal Detection Based on Improved YOLOv7 [J]. Journal of System Simulation, 2025, 37(4): 993-1007.
[8]	Wang He, Xu Jianing, Yan Guangyu. Research on Pedestrian Avoidance Strategy for AGV Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(3): 595-606.
[9]	Zhang Bin, Lei Yonglin, Li Qun, Gao Yuan, Chen Yong, Zhu Jiajun, Bao Chenlong. Reinforcement Learning Modeling of Missile Penetration Decision Based on Combat Simulation [J]. Journal of System Simulation, 2025, 37(3): 763-774.
[10]	Huang Sijin, Wen Jia, Chen Zheyi. Intelligent Service Migration towards MEC-based IoV Systems [J]. Journal of System Simulation, 2025, 37(2): 379-391.
[11]	Li Xiang, Ren Xiaoyu, Zhou Yongbing, Zhang Jian. Research on Flexible Integrated Scheduling Under Stochastic Processing Times Based on Improved D3QN Algorithm [J]. Journal of System Simulation, 2025, 37(2): 474-486.
[12]	Fei Shuaidi, Cai Changlong, Liu Fei, Chen Minghui, Liu Xiaoming. Research on the Target Allocation Method for Air Defense and Anti-missile Defense of Naval Ships [J]. Journal of System Simulation, 2025, 37(2): 508-516.
[13]	Li Chao, Li Jiabao, Ding Caichang, Ye Zhiwei, Zuo Fangwei. Edge Surveillance Task Offloading and Resource Allocation Algorithm Based on DRL [J]. Journal of System Simulation, 2024, 36(9): 2113-2126.
[14]	Liu Peijin, Fu Xuefeng, Sun Haofeng, He Lin, Liu Shujie. A Highly Robust Target Tracking Algorithm Merging CNN and Transformer [J]. Journal of System Simulation, 2024, 36(8): 1854-1868.
[15]	Lu Yang, Liu Pengfei, Xu Siyuan, Liu Qiwang, Gu Fuqian, Wang Peng. Simulation of Rice Disease Recognition Based on Improved Attention Mechanism Embedded in PR-Net Model [J]. Journal of System Simulation, 2024, 36(6): 1322-1333.