Intelligent Optimization of Coal Terminal Unloading Scheduling Based on Improved D3QN Algorithm

doi:10.16182/j.issn1004731x.joss.22-1320

Abstract

Abstract:

Intelligent decision scheduling can improve the operation efficiency of large ports, which is one of the important research directions for the implementation of artificial intelligence technology in the smart port scenario. This article studies the intelligent unloading scheduling tasks of coal terminals and abstracts them as a Markov sequence decision problem. A deep reinforcement learning model for this problem is established, and an improved D3QN algorithm is proposed to realize intelligent optimization of unloading scheduling decisions by considering the characteristics of high action space dimension and sparse feasible action in the model. The simulation results show that for the same set of random task sequences, the optimized scheduling strategy obviously improves the efficiency compared with the random strategy. At the same time, the trained scheduling strategy is directly applied to the randomly generated new task sequence, and the scheduling efficiency is improved by 5%~7%, which indicates that the optimization method has good generalization ability. In addition, with the increasing complexity of decision models, traditional heuristic optimization algorithms are faced with prominent problems such as difficult modeling and low solving efficiency. This article provides a new idea for studying this kind of problem, which is expected to realize the wider application of deep reinforcement learning-based intelligent decision-making in port scheduling tasks.

Key words: terminal unloading scheduling, scheduling strategy optimization, intelligent decision-making, deep reinforcement learning, Dueling Double DQN algorithm

CLC Number:

TP391.9

Qin Baoxin, Zhang Yuxiao, Wu Sirui, Cao Weichong, Li Zhan. Intelligent Optimization of Coal Terminal Unloading Scheduling Based on Improved D3QN Algorithm[J]. Journal of System Simulation, 2024, 36(3): 770-781.

Figures/Tables 18

Fig. 1

Fig. 2

Table 1

Status information of port unloading process

符号	含义
n	火车总数
F	码头翻车机设备集
D	码头堆料机设备集
k	进港火车的序号
i,j	翻车机与堆料机的序号，i,j =1,2,3,4
$π i j k$	第k辆火车选择的翻车机和堆料机
$A i j k$	$1, 第 k 列火车使用第 i 台翻车机 0, 存在设备占用情况动作不可行$
t_i	翻车机i工作的剩余时间
t_j	堆料机j工作的剩余时间
T^k	第k辆火车的作业耗时
T_all	所有火车的总完成时间, $T a l l = ∑ k = 1 n t k$

Table 1

Table 2

Status information of port unloading process

状态空间	数据结构	维数	类型	含义
T_wait	数组	4	Int	火车的等待时间
M_size	数组	42	Int	42个煤堆的存量
M_kind	数组	42	Int	42个煤堆的种类
X_size	数组	4	Int	火车运煤量
X_kind	数组	4	Int	火车运煤种类
Q	数组	4	Bool	传送带占用情况
Z	数组	4	Bool	翻车机占用情况
Q_i	数组	4	Int	4条传送带对应的煤堆
C_i	数组	4	Int	4列火车对应的翻车机
$T o c c u p y i$	数组	4	Int	4条传送带占用时间
Q_z	数组	4	Int	堆料机与翻车机的对应关系

Table 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Table 3

Fig. 10

Table 4

Table 5

Port history scheduling information

翻车机	车型	煤种	作业实绩
翻车机	车型	煤种	作业时间/h	大票吨
CD2	C80	外购55	111.80	4 320
CD1	C80	神混45	90.38	4 320
CD4	C80	外购55	77.50	4 320
CD3	C64	外石5 000	0	768
$⋮$
CD4	C80	外购45	78.63	4 320
CD3	C64	外石5 000	92.78	4 224
CD2	C80	神混45	82.53	4 320

Table 5

Fig. 11

Fig. 12

Table 6

References 23

1	Ambrosino Daniela, Sciomachen Anna, Tanfani Elena. A Decomposition Heuristics for the Container Ship Stowage Problem[J]. Journal of Heuristics, 2006, 12(3): 211-233.
2	Ambrosino Daniela, Sciomachen Anna, Tanfani Elena. Stowing a Containership: The Master Bay Plan Problem[J]. Transportation Research Part A: Policy and Practice, 2004, 38(2): 81-99.
3	Todd D S, Sen P. A Multiple Criteria Genetic Algorithm for Containership Loading[C]///Proceedings of the 7th International Conference on Genetic Algorithms. [S.l.]: [s.n.], 1997: 674-681.
4	卫家骏. 集装箱船智能配载研究[D]. 大连: 大连海事大学, 2012.
	Wei Jiajun. The Research on Container Ship's Intelligent Stowage[D]. Dalian: Dalian Maritime University, 2012.
5	Briskorn Dirk, Emde Simon, Boysen Nils. Cooperative Twin-crane Scheduling[J]. Discrete Applied Mathematics, 2016, 211: 40-57.
6	魏晨, 胡志华, 高超锋, 等. 自动化集装箱码头堆场内双起重机调度模型与算法[J]. 大连海事大学学报, 2015, 41(4): 75-80, 89.
	Wei Chen, Hu Zhihua, Gao Chaofeng, et al. Scheduling Model and Algorithm of Twin Synchronized Stacking Cranes in Stack Yard of Automated Container Terminal[J]. Journal of Dalian Maritime University, 2015, 41(4): 75-80, 89.
7	黄继伟, 韩晓龙. 基于遗传算法的自动化集装箱码头双轨道吊协同调度优化研究[J]. 计算机应用与软件, 2018, 35(9): 92-98, 143.
	Huang Jiwei, Han Xiaolong. Collaborative Scheduling Optimization of Twin Automated Stacking Cranes in Automatic Container Terminals Based on Genetic Algorithm[J]. Computer Applications and Software, 2018, 35(9): 92-98, 143.
8	Amir Hossein Gharehgozli, Laporte G, Yu Yugang, et al. Scheduling Twin Yard Cranes in a Container Block[J]. Transportation Science, 2015, 49(3): 686-705.
9	魏亚茹, 朱瑾. 自动化码头双场桥调度与集装箱存储选位建模[J]. 计算机应用, 2018, 38(4): 1189-1194, 1206.
	Wei Yaru, Zhu Jin. Modeling of Twin Rail-mounted Gantry Scheduling and Container Slot Selection in Automated Terminal[J]. Journal of Computer Applications, 2018, 38(4): 1189-1194, 1206.
10	初良勇, 李淑娟, 阮志毅. 多箱区多场桥调度优化模型及算法实现[J]. 上海海事大学学报, 2017, 38(1): 37-42.
	Chu Liangyong, Li Shujuan, Ruan Zhiyi. Scheduling Optimization Model and Algorithm Implementation of Multiple Container Blocks with Multiple Yard Cranes[J]. Journal of Shanghai Maritime University, 2017, 38(1): 37-42.
11	蒋静静. 基于深度强化学习的离散型制造企业车间动态调度研究[D]. 西安: 西安理工大学, 2020.
	Jiang Jingjing. Research on Jobshop Dynamic Scheduling of Discrete Manufacturig Enterprises Based on Deep Reinforcement Learning[D]. Xi'an: Xi'an University of Technology, 2020.
12	王凌, 潘子肖. 基于深度强化学习与迭代贪婪的流水车间调度优化[J]. 控制与决策, 2021, 36(11): 2609-2617.
	Wang Ling, Pan Zixiao. Scheduling Optimization for Flow-shop Based on Deep Reinforcement Learning and Iterative Greedy Method[J]. Control and Decision, 2021, 36(11): 2609-2617.
13	Wang Libing, Hu Xin, Wang Yin, et al. Dynamic Job-shop Scheduling in Smart Manufacturing Using Deep Reinforcement Learning[J]. Computer Networks, 2021, 190: 107969.
14	Luo Shu, Zhang Linxuan, Fan Yushun. Dynamic Multi-objective Scheduling for Flexible Job Shop by Deep Reinforcement Learning[J]. Computers and Industrial Engineering, 2021, 159: 107489.
15	Han Baoan, Yang Jianjun. Research on Adaptive Job Shop Scheduling Problems Based on Dueling Double DQN[J]. IEEE Access, 2020, 8: 186474-186495.
16	Hu Liang, Liu Zhenyu, Hu Weifei, et al. Petri-net-based Dynamic Scheduling of Flexible Manufacturing System Via Deep Reinforcement Learning with Graph Convolutional Network[J]. Journal of Manufacturing Systems, 2020, 55: 1-14.
17	Wang Xuelin, Shi Hankun. Research on Intelligent Optimization of Bulk Cargo Terminal Control System[J]. Journal of Physics: Conference Series, 2020, 1601(5): 052044.
18	Alan Dávila de León, Lalla-Ruiz Eduardo, Melián-Batista Belén, et al. A Machine Learning-based System for Berth Scheduling at Bulk Terminals[J]. Expert Systems with Applications, 2017, 87: 170-182.
19	高天佑. 输出型煤炭码头卸车生产调度优化模型和方法研究[D]. 武汉: 武汉理工大学, 2014.
	Gao Tianyou. Optimization Models and Algorithms for Unloading Scheduling of the Export Coal Terminals[D]. Wuhan: Wuhan University of Technology, 2014.
20	Fotuhi F, Huynh N, Vidal J M, et al. Modeling Yard Crane Operators as Reinforcement Learning Agents[J]. Research in Transportation Economics, 2013, 42(1): 3-12.
21	杨奔, 王炜晔, 赵婉婷, 等. 基于DQN的动态深度多分支搜索自动配载算法[J]. 计算机工程, 2020, 46(8): 313-320.
	Yang Ben, Wang Weiye, Zhao Wanting, et al. DQN-based Automatic Stowage Planning Algorithm Using Dynamic Depth Multi-branch Search[J]. Computer Engineering, 2020, 46(8): 313-320.
22	Shen Yifan, Zhao Ning, Xia Mengjue, et al. A Deep Q-learning Network for Ship Stowage Planning Problem[J]. Polish Maritime Research, 2017, 24(S3): 102-109.
23	Li Changan, Wu Sirui, Li Zhan, et al. Intelligent Scheduling Method for Bulk Cargo Terminal Loading Process Based on Deep Reinforcement Learning[J]. Electronics, 2022, 11(9): 1390.

任务	方法决策	非法动作次数	堆料机切换次数	调度总时长/h	优化比例/%
1	随机策略	基准	[20,60,58,19]	321.21	基准
1	训练模型策略	约10	[16,35,31,16]	301.10	6.7
2	随机策略	基准	[33,55,63,25]	332.45	基准
2	训练模型策略	约12	[21,38,30,18]	309.21	6.4

月份	t/h		优化比例/%
月份	人工排产	智能排产	优化比例/%
1	824.03	691.70	16.0
2	721.24	693.79	3.80
3	658.05	620.14	5.80
4	798.50	621.92	22.4
5	710.46	693.23	2.40
6	807.93	632.84	21.7

[1]	Jiang Ming, He Tao. Solving the Vehicle Routing Problem Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(9): 2177-2187.
[2]	Ni Peilong, Mao Pengjun, Wang Ning, Yang Mengjie. Robot Path Planning Based on Improved A-DDQN Algorithm [J]. Journal of System Simulation, 2025, 37(9): 2420-2430.
[3]	Chen Zhen, Wu Zhuoyi, Zhang Lin. Research on Policy Representation in Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(7): 1753-1769.
[4]	Wu Guohua, Zeng Jiaheng, Wang Dezhi, Zheng Long, Zou Wei. A Quadrotor Trajectory Tracking Control Method Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(5): 1169-1187.
[5]	Zhang Bin, Lei Yonglin, Li Qun, Gao Yuan, Chen Yong, Zhu Jiajun, Bao Chenlong. Reinforcement Learning Modeling of Missile Penetration Decision Based on Combat Simulation [J]. Journal of System Simulation, 2025, 37(3): 763-774.
[6]	Bai Zhenzu, Hou Yizhi, He Zhangming, Wei Juhui, Zhou Haiyin, Wang Jiongqi. Optimization of Dynamic Weapon Target Assignment Considering Random Disturbances [J]. Journal of System Simulation, 2025, 37(12): 2967-2980.
[7]	Zheng Jiayu, Mai Zhuxue, Chen Zheyi. Optimization of Service Caching and Computation Offloading in Digital Twin Cloud-edge Networks [J]. Journal of System Simulation, 2025, 37(11): 2741-2753.
[8]	Di Jian, Wan Xue, Jiang Limei. Evolutionary Reinforcement Learning Based on Elite Instruction and Random Search [J]. Journal of System Simulation, 2025, 37(11): 2877-2887.
[9]	Xu Zhongkai, Chu Chenyang, Xie Kai, Zhao Ruizhuo, Ke Wenjun. Optimization Dispatch Method for High-proportion Renewable Energy Power Systems Based on SC-PPO [J]. Journal of System Simulation, 2025, 37(10): 2511-2521.
[10]	Liang Xiuman, Liu Ziliang, Liu Zhendong. Path Planning of Improved RRT Algorithm Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(10): 2578-2593.
[11]	Jiang Jiachen, Jia Zhengxuan, Xu Zhao, Lin Tingyu, Zhao Pengpeng, Ou Yiming. Decision Modeling and Solution Based on Game Adversarial Complex Systems [J]. Journal of System Simulation, 2025, 37(1): 66-78.
[12]	Li Ming, Ye Wangzhong, Yan Jiehua. Path Planning of Desert Robot Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(12): 2917-2925.
[13]	Zhang Yongfu, Liu Yang, Yuan He. A Method for Key Node Identification in Operational Target System Based on War Gaming [J]. Journal of System Simulation, 2024, 36(11): 2654-2661.
[14]	An Jing, Si Guangya, Zhang Lei. Strategy Optimization Method of Multi-dimension Projection Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2024, 36(1): 39-49.
[15]	Laiyi Yang, Jing Bi, Haitao Yuan. Intelligent Path Planning for Mobile Robots Based on SAC Algorithm [J]. Journal of System Simulation, 2023, 35(8): 1726-1736.