DQN-based Joint Scheduling Method of Heterogeneous TT&C Resources

doi:10.16182/j.issn1004731x.joss.21-0879

Abstract

Abstract:

Joint scheduling of heterogeneous TT&C resources as research object, a deep Q network (DQN) algorithm based on reinforcement learning is proposed. The characteristics of the joint scheduling problem of heterogeneous TT&C resources being fully analyzied and mathematical language being used to describe the constraints affecting the solution, a resource joint scheduling model is established. From the perspective of applying reinforcement learning, two neural networks with the same structure and the action selection strategies based on $ε$ greedy algorithm are respectively designed after Markov decision process description, and DQN solution framework is established. The simulation results show that DQN-based heterogeneous TT&C resources scheduling method can identify a TT&C scheduling scheme with better scheduling revenue than the genetic algorithm.

Key words: telemetry, track and command (TT&C), joint scheduling of heterogeneous TT&C resources, deep Q network, scheduling revenue, reinforcement learning

CLC Number:

TP273+.1

Naiyang Xue, Dan Ding, Yutong Jia, Zhiqiang Wang, Yuan Liu. DQN-based Joint Scheduling Method of Heterogeneous TT&C Resources[J]. Journal of System Simulation, 2023, 35(2): 423-434.

Figures/Tables 12

Fig. 1

Fig. 2

Fig. 3

Table 1

TT&C act income statement

测控资源属性 $S T k$	卫星属性 $R T k$	弧段调度收益 $A P r k$
1	1	10
1	2	5
2	1	6
2	2	10

Table 1

Table 2

Fig. 4

Table 3

Partial parameter settings of DQN model

参数名称	参数值
迭代次数	1 050
观察期迭代次数	50
经验池大小	8 000
随机梯度下降采样样本大小	400
目标Q网络更新频率	1 200
学习率	0.000 15
$ε$ 贪婪算法的初始探索值	1
$ε$ 贪婪算法的最终探索值	0.06
$ε$ 衰减因子	0.4

Table 3

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

References 12

1	于志坚. 我国航天测控系统的现状与发展[J]. 中国工程科学, 2006, 8(10): 42-46.
	Yu Zhijian. Status Quo and Development of Spaceflight TT&C Systems[J]. China Engineering Science, 2006, 8(10): 42-46.
2	宋永生, 李铎, 陈劭睿. 商业航天测控管理[J]. 数字通信世界, 2019(7): 29.
3	郭夏锐. 商业卫星测控发展现状及趋势[J]. 国际太空, 2019(10): 44-48.
4	薛乃阳, 丁丹, 王红敏, 等. 基于改进遗传算法的多类测控资源调度方法[J]. 系统工程与电子技术, 2021, 43(9): 2535-2543.
	Xue Naiyang, Ding Dan, Wang Hongmin, et al. Multi-type Measurement and Control Resource Scheduling Method Based on Improved Genetic Algorithm[J]. System Engineering and Electronic Technology, 2021, 43(9): 2535-2543.
5	Stottler R, Richards R. Managed Intelligent Deconfliction and Scheduling for Satellite Communication[C]//2018 IEEE Aerospace Conference. IEEE, 2018: 1-7.
6	张天骄, 李济生, 李晶, 等. 基于混合蚁群优化的天地一体化调度方法[J]. 系统工程与电子技术, 2016, 38(7): 1555-1562.
	Zhang Tianjiao, Li Jisheng, Li Jing, et al. Space-ground Integrated Scheduling Based on the Hybrid ant Colony Optimization[J]. Systems Engineering and Electronics, 2016, 38(7): 1555-1562.
7	李长德, 徐伟, 徐梁, 等. 基于深度神经网络的多星测控调度方法[J]. 中国空间科学技术, 2022, 42(1): 65-72.
	Li Changde, Xu Wei, Xu Liang, et al. Multi-satellite TT&C Scheduling Method Based on DNN[J].Chinese Space Science and Technology, 2022, 42(1): 65-72.
8	朱凤龙. 遗传算法"早熟"现象的探究及改进策略[D]. 重庆: 西南大学, 2010.
	Zhu Fenglong. Research on the Premature Convergence of Genetic Algorithm and Its Improved Politics[D]. Chongqing: Southwest University, 2010.
9	陈仲铭, 何明. 深度强化学习原理与实践[M]. 北京: 人民邮电出版社, 2019: 6-7.
	Chen Zhongming, He Ming. Deep Reinforcement Learning: Principles and Practices[M]. Beijing: Posts & Telecom Press, 2019: 6-7.
10	薛乃阳, 丁丹, 王红敏, 等. 引入微元法思想的混合测控资源联合调度方法[J].系统仿真学报, 2022, 34(4): 826-835.
	Xue Naiyang, Ding Dan, Wang Hongmin, et al. Idea of Infinitesimal Method-introduced Hybrid TT&C Resources Joint Scheduling[J]. Journal of System Simulation, 2022, 34(4): 826-835.
11	杨子璇. 基于深度Q网络的带返工汽车涂装作业重排方法[D]. 大连: 大连理工大学, 2020.
	Yang Zixuan. A Resequencing Method for Automotive Painting Operations Considering Rework Based on Deep Q Network[D]. Dalian: Dalian University of Technology, 2020.
12	安元元, 李伟超, 王伟, 等. 一种低轨卫星星座测控地面站调度策略研究[J]. 时间频率学报, 2021, 44(2): 120-131.
	An Yuanyuan, Li Weichao, Wang wei, et al. Research on Schedule Strategy of Ground Stations for LEO Satellites[J]. Journal of Time and Frequency, 2021, 44(2): 120-131.

神经网络结构名称	神经元个数
输入层	19
隐藏层1	40
隐藏层2	30
输出层	1

[1]	Jiajie Shi, Peng Yang, Yannan Pi. Machine Learning-based Simulation Research of On-line Subway Pedestrian Flow Control [J]. Journal of System Simulation, 2023, 35(2): 386-395.
[2]	Yejian Zhao, Yanhong Wang, Jun Zhang, Hongxia Yu, Zhongda Tian. Application of Improved Q Learning Algorithm in Job Shop Scheduling Problem [J]. Journal of System Simulation, 2022, 34(6): 1247-1258.
[3]	Sen Zhang, Mengyan Zhang, Jingping Shao, Jiexin Pu. Multi-UAVs 3D Path Planning Method Based on Random Strategy Search [J]. Journal of System Simulation, 2022, 34(6): 1286-1295.
[4]	Lingjia Ni, Xiaoxia Huang, Hongga Li, Zibo Zhang. Research on Fire Emergency Evacuation Simulation Based on Cooperative Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(6): 1353-1366.
[5]	Naiyang Xue, Dan Ding, Hongmin Wang, Yile Fan, Zhongqian Liu. Idea of Infinitesimal Method-introduced Hybrid TT&C Resources Joint Scheduling [J]. Journal of System Simulation, 2022, 34(4): 826-835.
[6]	Hongwei Wang, Peng Yang. Research on Optimization of Airport Cargo Business Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(3): 651-660.
[7]	Xiaohan Wang, Lin Zhang, Yuanjun Laili, Kunyu Xie, Tingchun Hu. Constructing the Agent Discrete Simulation Based on DEVS Atomic Model [J]. Journal of System Simulation, 2022, 34(2): 191-200.
[8]	Qirui Li, Xinyi Peng. Job Scheduling and Simulation in Cloud Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(2): 258-268.
[9]	Da Jiang, Zhiqin Cai, Zhongzhen Liu, Haijun Peng, Zhigang Wu. Reinforcement-learning-based Adaptive Tracking Control for a Space Continuum Robot Based on Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(10): 2264-2271.
[10]	Wu Xi, Meng Xianglin, Yang Jingyu. Study on Next-generation Strategic Wargame System [J]. Journal of System Simulation, 2021, 33(9): 2017-2024.
[11]	Zhao Yu, Guo Jifeng, Yan Peng, Bai Chengchao. Self-learning-based Multiple Spacecraft Evasion Decision Making Simulation Under Sparse Reward Condition [J]. Journal of System Simulation, 2021, 33(8): 1766-1774.
[12]	Gao Ang, Dong Zhiming, Zhang Guohui, Liang Tao, Guo Qisheng. Research on Generation Technology of Computer Generated Force in LVC Training System [J]. Journal of System Simulation, 2021, 33(3): 745-752.
[13]	Wang Xiaohan, Zhang Lin, Ren Lei, Xie Kunyu, Wang Kunyu, Ye Fei, Chen Zhen. Brief Review on Applying Reinforcement Learning to Job Shop Scheduling Problems [J]. Journal of System Simulation, 2021, 33(12): 2782-2791.
[14]	Zeng Bi, Fang Xiao, Kong Deshuai, Song Xiangxiang, Jia Zhengxuan, Lin Tingyu. A Data-Driven Modeling Method for Game Adversity Agent [J]. Journal of System Simulation, 2021, 33(12): 2838-2845.
[15]	Ma Jun, Yang Jingyu, Wu Xi. Research on Experimental Method of Joint Operation Simulation Based on Human-machine Hybrid Intelligence [J]. Journal of System Simulation, 2021, 33(10): 2323-2334.