AGV Scheduling Problem at Automated Terminals Based on Improved DQN Algorithm

doi:10.16182/j.issn1004731x.joss.23-0912

Abstract

Abstract:

A future tasks considering deep Q-network (F-DQN) algorithm was proposed to output real-time scheduling results of automated guided vehicles (AGVs) at automated terminals. This algorithm combined the advantages of real-time scheduling and static scheduling, improving the system status by considering static future task information when making real-time decisions, so as to obtain a better scheduling solution. In this study, the actual layout and equipment conditions of the Yangshan phase IV automated terminal were considered, and a series of simulation experiments were conducted using the Plant Simulation software. The experimental results show that the F-DQN algorithm can effectively solve the real-time scheduling problem of AGVs at automated terminals. Furthermore, the F-DQN algorithm significantly reduces the waiting time of quay cranes compared to the traditional DQN algorithm.

Key words: automated terminal, AGV scheduling, DQN, MDP, simulation model

CLC Number:

TP391.9

Liang Chengji, Zhang Shidong, Wang Yu, Lu Bin. AGV Scheduling Problem at Automated Terminals Based on Improved DQN Algorithm[J]. Journal of System Simulation, 2024, 36(11): 2592-2603.

Figures/Tables 15

Fig. 1

Table 1

Parameters and comments

参数	注释
$V$	AGV $i$ 的集合， $i ∈ V$
$Q$	岸桥 $q$ 的集合， $q ∈ Q$
$Y$	场桥 $y$ 的集合， $y ∈ Y$
$N$	集装箱 $n$ 的集合， $n ∈ N$
$M$	经验回放池容量
$E M$	训练迭代次数(episodes)
$L R$	学习率
$U F$	目标Q网络更新频率
$ε$	贪婪因子
$τ$	目标Q网络参数更新系数
$α$	完工时间折扣因子
$β$	岸桥等待时间折扣因子
$μ$	奖励函数中的比例因子
$ω$	奖励重构因子

Table 1

Table 2

Variables and comments

变量	注释
K	调度决策阶段 $k$ 的集合， $k ∈ K$
t_k	决策阶段k的时刻
s_k	状态变量，代表t_k 时刻的环境信息
a_k	决策阶段k的动作
s_k₊₁	下一决策阶段的状态变量
r_k	状态变量从s_k 转换到s_k₊₁的奖励值
$s k + 1'$	改进后的状态变量
$r k'$	重构后的奖励值

Table 2

Table 3

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Table 4

Fig. 7

Table 5

Fig. 8

Fig. 9

Table 6

References 15

1	张素云, 杨勇生, 梁承姬, 等. 自动化码头多AGV路径冲突的优化控制研究[J]. 交通运输系统工程与信息, 2017, 17(2): 83-89.
	Zhang Suyun, Yang Yongsheng, Liang Chengji, et al. Optimal Control of Multiple AGV Path Conflict in Automated Terminals[J]. Journal of Transportation Systems Engineering and Information Technology, 2017, 17(2): 83-89.
2	He Junliang, Huang Youfang, Yan Wei, et al. Integrated Internal Truck, Yard Crane and Quay Crane Scheduling in a Container Terminal Considering Energy Consumption[J]. Expert Systems with Applications, 2015, 42(5): 2464-2487.
3	谢旦岚, 郭笛, 纪媛, 等. 自动化码头运输设备充电策略优化的仿真研究[J]. 系统仿真学报, 2020, 32(10): 1927-1935.
	Xie Danlan, Guo Di, Ji Yuan, et al. Simulation Research on Optimization of AGV Charging Strategy for Automated Terminal[J]. Journal of System Simulation, 2020, 32(10): 1927-1935.
4	Hwan Kim Kap, Jong Wook Bae. A Look-ahead Dispatching Method for Automated Guided Vehicles in Automated Port Container Terminals[J]. Transportation Science, 2004, 38(2): 224-234.
5	Homayouni Mahdi, Tang Saihong, Ismail Napsiah, et al. Using Simulated Annealing Algorithm for Optimization of Quay Cranes and Automated Guided Vehicles Scheduling[J]. International Journal of the Physical Sciences, 2011, 6(27): 6286-6294.
6	Shiue Y R, Lee K C, Su C T. Real-time Scheduling for a Smart Factory Using a Reinforcement Learning Approach[J]. Computers & Industrial Engineering, 2018, 125: 604-614.
7	Wang Haoxiang, Sarker B R, Li Jing, et al. Adaptive Scheduling for Assembly Job Shop with Uncertain Assembly Times Based on Dual Q-learning[J]. International Journal of Production Research, 2021, 59(19): 5867-5883.
8	Mnih V, Kavukcuoglu K, Silver D, et al. Human-level Control Through Deep Reinforcement Learning[J]. Nature, 2015, 518(7540): 529-533.
9	Hu Hao, Jia Xiaoliang, He Qixuan, et al. Deep Reinforcement Learning Based AGVs Real-time Scheduling with Mixed Rule for Flexible Shop Floor in Industry 4.0[J]. Computers & Industrial Engineering, 2020, 149: 106749.
10	Wei Qinglai, Yan Yutian, Zhang Jie, et al. A Self-attention-based Deep Reinforcement Learning Approach for AGV Dispatching Systems[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(6): 7911-7922.
11	Chen Xuchao, He Shiwei, Zhang Yongxiang, et al. Yard Crane and AGV Scheduling in Automated Container Terminal: A Multi-robot Task Allocation Framework[J]. Transportation Research Part C: Emerging Technologies, 2020, 114: 241-271.
12	Egbelu P J, Tanchoco J M A. Characterization of Automatic Guided Vehicle Dispatching Rules[J]. International Journal of Production Research, 1984, 22(3): 359-374.
13	Sutton R S, Barto A G. Reinforcement Learning: An Introduction[M]. Cambridge: MIT Press, 2018.
14	李凯文, 张涛, 王锐, 等. 基于深度强化学习的组合优化研究进展[J]. 自动化学报, 2021, 47(11): 2521-2537.
	Li Kaiwen, Zhang Tao, Wang Rui, et al. Research Reviews of Combinatorial Optimization Methods Based on Deep Reinforcement Learning[J]. Acta Automatica Sinica, 2021, 47(11): 2521-2537.
15	Banino A, Barry C, Uria B, et al. Vector-based Navigation Using Grid-like Representations in Artificial Agents[J]. Nature, 2018, 557(7705): 429-433.

规则名称	规则描述
NV	将首先选择距离当前任务最近的AGV
FV	将首先选择距离当前任务最远的AGV
LIV	将首先选择等待时间最长的AGV
LUV	将首先选择利用率最低的AGV

算法	最大完工时间/s	岸桥等待时间/s	AGV等待时间/s
F-DQN	5 438	423	288
DQN	5 453	457	304
NV	5 510	599	398
FV	5 637	1 198	224
LIV	5 601	1 281	321
LUV	5 604	1 743	232

算法	最大完工时间/s	岸桥等待时间/s	AGV等待时间/s
DQN	13 814	680	6 493
F-DQN-1	13 746	593	6 376
F-DQN-2	13 601	477	6 224
F-DQN-3	13 659	568	6 312
F-DQN-4	13 638	524	6 286

[1]	Qiu Jiandong, Tang Yi, Ji Yuxiong, Liu Heng, Luo Junshao. Research on Simulation Model of Double-layer Expansion Design of Expressway [J]. Journal of System Simulation, 2024, 36(5): 1072-1080.
[2]	Qin Baoxin, Zhang Yuxiao, Wu Sirui, Cao Weichong, Li Zhan. Intelligent Optimization of Coal Terminal Unloading Scheduling Based on Improved D3QN Algorithm [J]. Journal of System Simulation, 2024, 36(3): 770-781.
[3]	Chen Shikang, Wang Zhimin, Liang Biao, Gu Kunren, Song Keman, Gao Yuan. Modeling Method of Blue Army Warship Formation Air Defense Command and Control for System-of-systems Simulation [J]. Journal of System Simulation, 2024, 36(10): 2246-2256.
[4]	Bao Zhe, Zhang Xiaofang, Li Wei, Xu Ye, Wang Xu. Research on Flexible Operational Optimization of CCHP System Based on Intelligent Fusion Algorithm [J]. Journal of System Simulation, 2024, 36(10): 2330-2344.
[5]	Luo Yucheng, Zhang Ming'en, Liu Fei, Lu Yingbo, Ye Feng. Result Validation Method of Simulation Models Based on Piecewise Feature Extraction [J]. Journal of System Simulation, 2024, 36(1): 272-281.
[6]	Yingxuan Li, Zhongxun Wang, Yunlong Dong. Two New Maneuvering Target Simulation Methods [J]. Journal of System Simulation, 2023, 35(7): 1581-1589.
[7]	Hailan Yang, Yongqiang Qi, Baolei Wu, Dan Rong, Miaoying Hong, Jun Wang. Path Planning of Mobile Robots Based on Memristor Reinforcement Learning in Dynamic Environment [J]. Journal of System Simulation, 2023, 35(7): 1619-1633.
[8]	Yuxuan Dai, Chenggang Cui. Deep Reinforcement Learning-Based Control Strategy for Boost Converter [J]. Journal of System Simulation, 2023, 35(5): 1109-1119.
[9]	Xianwei Zhou, Qixu Gong, Songsen Yu. Computation Offloading Strategy Based on Stackelberg Game and DRL [J]. Journal of System Simulation, 2023, 35(2): 372-385.
[10]	Yarong Chen, Shuchen Guan, Chengjun Huang, Lixia Zhu, Chou FuhDer. Simulation-Based Adaptive Dynamic Scheduling for Bi-objective Parallel Multi-processor Open Shop [J]. Journal of System Simulation, 2023, 35(1): 69-81.
[11]	Aibin Chen, Fubo Ding, Guoxiong Zhou, Bo Zhou. Simulation Model of Forest Fire Spread Based on Swarm Intelligence [J]. Journal of System Simulation, 2022, 34(7): 1439-1448.
[12]	Hongwei Wang, Peng Yang. Research on Optimization of Airport Cargo Business Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(3): 651-660.
[13]	Boyuan Zhang, Guanghong Gong, Ze Wang, Ni Li. Research on Parameter Construction Method of Blue Army Equipment Model Based on a Deep Network [J]. Journal of System Simulation, 2022, 34(12): 2629-2638.
[14]	Zhang Luan, Luo Zhengjun, Zhou Dequn. Agent- based Research on Power Absorption Simulation Analysis of Renewable Energy [J]. Journal of System Simulation, 2022, 34(1): 170-178.
[15]	Li Rongqiang, Wen Aibing, Hua Bin, Li Jiajun, Jiang Bing. Rapid Development Technology of Virtual Maintenance Training Simulation Model for Aviation Equipment [J]. Journal of System Simulation, 2021, 33(5): 1167-1176.