基于改进D3QN算法的随机工时下柔性综合调度问题研究

doi:10.16182/j.issn1004731x.joss.23-1164

摘要/Abstract

摘要：

针对离散制造车间的工时不确定性问题，在考虑设备和工序约束的基础上，以最小化最大完工时间为优化目标构建综合调度数学模型，并提出一种改进双竞争深度Q网络算法(ID3QN)求解随机工时下的柔性综合调度问题。从工序、机器及整体层面分别设计了三组状态特征；将与工时、加工顺序相关的工序规则以及与优化目标相关的机器规则组成的8组复合调度规则作为动作集，并根据平均机器利用率差值进行即时奖励；引入自注意力机制与混合采样策略，以进一步提升算法稳定性和泛化性。仿真结果表明：所提算法在求解随机工时柔性综合调度问题时，平均偏差比现有深度强化学习算法平均提高了54.63%，验证了算法的有效性。

关键词: 柔性综合调度, ID3QN, 自注意力机制, 随机工时

Abstract:

Aiming at the problem of time uncertainty in discrete manufacturing workshops, we construct an integrated scheduling mathematical model with the optimization objective of minimizing the maximum completion time based on the consideration of equipment and process constraints, and propose an improved dual-competitive deep Q-network algorithm (ID3QN) to solve the flexible integrated scheduling problem under stochastic working hours. The levels of process, machine, and overall scheduling are designed as features. Eight composite scheduling rules are formed as the action space by combining process rules based on processing times, processing sequences, and process structure tree, along with machine rules relevant to optimization objectives. The difference in average machine utilization set as rewards instantly. Simultaneously, the self-attention mechanism and the mixed sampling strategy are introduced to improve the stability and generalization of the algorithm. The empirical results demonstrate that the average variation of the ID3QN is increased by 54.63% compared to existing deep reinforcement learning algorithms under stochastic processing times, thus confirming the effectiveness of the proposed algorithm.

Key words: flexible integrated scheduling, ID3QN, self-attention mechanism, stochastic processing times

中图分类号:

TH186

李想,任晓羽,周永兵等 . 基于改进D3QN算法的随机工时下柔性综合调度问题研究[J]. 系统仿真学报, 2025, 37(2): 474-486.

Li Xiang,Ren Xiaoyu,Zhou Yongbing,et al . Research on Flexible Integrated Scheduling Under Stochastic Processing Times Based on Improved D3QN Algorithm[J]. Journal of System Simulation, 2025, 37(2): 474-486.

图/表 16

图1

表1

数学符号表

符号	描述
$m$	机器总数
$Ω$	总的机器集合
$i$	机器序号， $i = 1,2, ⋯, m$
$n$	产品工序总数
$h, l$	工序索引号， $h, l = 1,2, ⋯, n$
$Ω h$	产品第 $h$ 道工序的可选加工机器集
$m h$	产品第 $h$ 道工序的可选加工机器数
$O h$	产品的第 $h$ 道工序
$M i h$	产品第 $h$ 道工序在机器 $i$ 上加工
$d i h$	产品第 $h$ 道工序在机器 $i$ 的实际加工时间
$d i h *$	产品第 $h$ 道工序在机器 $i$ 的期望加工时间
$s h$	产品第 $h$ 道工序开始加工时间
$c h$	产品第 $h$ 道工序加工完成时间
$P h$	产品第 $h$ 道工序的直接紧前工序集合
$L$	一个足够大的正数
$C m a x$	产品的最大完工时间
$x i h$	决策变量，工序 $O h$ 是否选择机器 $i$
$y i h l$	决策变量，工序 $O i h$ 是否先于 $O i l$ 加工

表1

图2

图3

表2

图4

图5

图6

表3

随机工时分布信息

分布	分布范围	分布方差
U1	$U (d i h * - d i h ), d i h + d i h *)$	$d i h * / 3$
U2	$U (d i h , 2 d i h )$	$(d i h *) 2 / 3$
B1	$B (d i h * / 2,2 d i h , d i h / 2 - 1 / 3, d i h * - 2 / 3)$	$d i h * / 3$
B2	$B (d i h * / 2,2 d i h *, 1 / 6,1 / 3)$	$(d i h *) 2 / 3$
E	$E (d i h *)$	$(d i h *) 2$

表3

表4

超参数

参数名称	值
折扣因子 $γ$	0.9
学习率r_l	10^-3
训练回合数M	2 500
记忆池容量N	10⁵
批量大小B	32
更新目标网络步数C	4
贪婪策略概率 $ε$	从0.9~0.1线性下降
裁剪系数	0.3

表4

图7

表5

表6

表7

不同方差下的随机工时期望偏差

算例	分布	PR1	PR2	PR3	PR4	PR5	PR6	PR7	AC	DQN	DDQN	D3QN	IGA	IMFFA	ID3QN
MK01	U1	0.96	2.42	1.26	1.84	1.11	2.87	1.02	0.63	0.96	0.75	0.33	0.20	0.25	0.35
	U2	1.97	4.44	2.64	3.65	2.39	5.05	2.10	1.35	1.75	1.68	0.72	0.86	0.96	0.69
	B1	0.53	1.77	0.71	1.30	0.62	1.89	0.57	0.14	0.49	0.29	0.11	0.20	0.24	0.53
	B2	0.36	1.87	0.75	1.32	0.51	2.26	0.48	0.14	0.64	0.19	0.15	0.12	0.06	0.44
	E	1.34	2.61	1.36	1.93	1.57	3.28	1.45	0.72	1.22	0.86	0.50	0.38	0.29	0.48
MK02	U1	0.86	4.19	1.54	2.50	1.25	5.07	1.29	1.58	1.26	0.82	0.22	0.14	0.24	0.25
	U2	1.95	7.40	2.78	4.48	2.98	8.16	2.98	3.08	2.51	2.31	0.64	0.90	1.03	0.64
	B1	0.49	2.99	0.74	1.59	0.90	3.38	0.99	1.02	0.81	0.62	0.53	0.38	0.32	0.55
	B2	0.39	3.69	1.11	1.98	0.72	4.16	0.76	1.32	1.24	1.10	0.46	0.30	0.33	0.47
	E	1.45	5.03	1.90	2.92	1.94	5.84	1.89	1.96	2.27	1.45	0.68	0.45	0.57	0.54
MK03	U1	0.99	3.49	0.93	2.34	0.93	4.58	1.14	0.82	0.62	0.78	0.37	0.26	0.31	0.37
	U2	1.90	5.60	1.85	3.96	1.78	7.34	2.12	1.61	1.33	1.58	0.78	0.84	0.89	0.70
	B1	0.45	2.36	0.44	1.36	0.48	3.24	0.57	0.31	0.09	0.13	0.04	0.09	0.04	0.02
	B2	0.27	2.98	0.68	1.80	0.37	3.79	0.47	0.60	0.62	0.90	0.05	0.11	0.09	0.03
	E	0.95	2.70	0.70	1.70	0.95	4.02	1.05	0.57	0.42	0.66	0.09	0.12	0.14	0.08
MK04	U1	0.77	1.59	0.56	0.86	0.83	1.89	1.04	0.81	0.84	0.58	0.04	0.05	0.03	0.07
	U2	1.52	2.94	1.41	1.90	1.86	3.73	2.19	2.08	1.93	1.46	0.41	0.46	0.53	0.40
	B1	0.47	0.94	0.18	0.40	0.53	1.15	0.66	0.39	0.33	0.16	0.27	0.23	0.20	0.22
	B2	0.27	1.25	0.29	0.59	0.33	1.47	0.46	0.59	0.46	0.48	0.21	0.26	0.20	0.20
	E	1.00	1.76	0.58	0.83	1.24	1.90	1.46	0.86	0.83	0.92	0.16	0.10	0.07	0.11
MK05	U1	1.37	1.83	0.71	0.91	1.41	2.13	1.58	0.94	0.74	0.93	0.40	0.31	0.34	0.38
	U2	2.50	3.54	1.72	1.93	2.74	3.94	3.13	1.89	1.79	2.02	0.77	1.00	1.03	0.81
	B1	0.82	1.15	0.36	0.42	0.89	1.38	1.08	0.45	0.29	0.46	0.60	0.50	0.53	0.60
	B2	0.61	1.57	0.44	0.68	0.72	1.75	0.89	0.47	0.40	0.54	0.05	0.05	0.02	0.51
	E	1.43	1.93	0.72	0.81	1.54	1.97	1.73	0.91	0.82	0.93	0.59	0.31	0.35	0.25
MK06	U1	1.18	5.44	1.76	3.44	1.32	6.23	1.45	3.31	3.07	2.62	0.38	0.12	0.21	0.37
	U2	2.77	8.96	3.33	5.74	3.04	9.18	3.48	5.54	5.17	4.76	1.01	0.86	0.95	0.83
	B1	0.82	3.57	1.11	2.31	1.01	4.25	1.18	2.06	1.93	1.61	0.08	0.08	0.02	0.63
	B2	0.48	4.65	1.29	3.09	0.71	5.44	0.78	2.84	2.35	2.13	0.66	0.46	0.49	0.08
	E	2.04	4.88	1.76	2.99	2.41	5.68	2.66	2.74	2.35	2.54	0.36	0.46	0.49	0.36
MK07	U1	1.15	3.69	0.98	2.00	1.42	4.09	1.27	1.42	0.92	0.84	0.38	0.21	0.27	0.36
	U2	2.14	5.96	1.89	3.24	2.83	6.19	2.48	2.27	1.67	1.50	0.78	0.77	0.89	0.77
	B1	0.69	2.33	0.41	1.04	1.06	2.57	0.81	0.67	0.32	0.33	0.56	0.09	0.05	0.01
	B2	0.37	2.87	0.68	1.51	0.63	3.33	0.53	1.02	0.56	1.08	0.45	0.18	0.13	0.05
	E	1.12	2.49	0.71	1.27	1.58	2.89	1.36	0.89	0.64	0.57	0.44	0.16	0.21	0.14
MK08	U1	1.04	1.84	0.57	0.71	1.65	2.45	2.00	0.60	0.79	1.02	0.38	0.30	0.32	0.37
	U2	2.03	3.36	1.31	1.55	2.97	4.09	3.40	1.40	1.66	1.88	0.73	0.93	0.95	0.65
	B1	0.63	1.06	0.15	0.24	0.98	1.42	1.20	0.16	0.31	0.35	0.58	0.03	0.03	0.01
	B2	0.53	1.10	0.14	0.28	0.91	1.55	1.14	0.22	0.39	0.48	0.50	0.04	0.03	0.01
	E	0.86	1.24	0.35	0.41	1.30	1.78	1.62	0.34	0.45	0.56	0.36	0.20	0.12	0.10
MK09	U1	1.87	3.78	1.15	2.03	2.31	4.76	3.03	0.67	1.11	1.32	0.53	0.33	0.32	0.55
	U2	3.31	6.11	2.00	3.24	4.00	7.11	4.96	1.52	2.27	2.30	0.97	1.01	1.04	1.22
	B1	1.16	2.53	0.56	1.19	1.56	3.00	2.12	0.30	0.62	0.56	0.70	0.11	0.12	0.14
	B2	0.96	2.55	0.60	1.11	1.33	3.46	1.60	0.23	0.85	0.75	0.17	0.08	0.06	0.02
	E	1.76	2.88	0.83	1.44	2.12	3.47	2.81	0.58	0.81	0.86	0.29	0.19	0.17	0.46
MK10	U1	1.63	4.11	1.23	2.16	1.89	5.22	2.19	1.42	1.84	1.37	0.45	0.21	0.24	0.48
	U2	2.68	6.36	2.18	3.59	3.32	8.17	3.71	2.41	3.43	2.58	0.90	0.81	0.86	1.31
	B1	1.11	2.72	0.75	1.46	1.30	3.64	1.59	0.74	1.24	0.62	0.62	0.17	0.15	0.21
	B2	0.66	3.21	0.66	1.48	0.84	3.87	1.03	0.92	1.31	1.32	0.13	0.16	0.14	0.06
	E	1.58	2.99	0.99	1.81	1.78	4.05	2.10	1.06	1.44	0.86	0.47	0.15	0.13	0.26

表7

表8

图8

参考文献 33

1	尚旭东. 存在相同多功能设备的综合调度算法研究[D]. 哈尔滨: 哈尔滨理工大学, 2022.
	Shang Xudong. Research on Integrated Scheduling Algorithm with the Same Multi-functional Machine[D]. Harbin: Harbin University of Science and Technology, 2022.
2	谢志强, 王茜. 基于逆序层优先的柔性综合调度算法[J]. 电子与信息学报, 2022, 44(5): 1554-1562.
	Xie Zhiqiang, Wang Qian. Flexible Integrated Scheduling Algorithm Based on Reverse Order Layer Priority[J]. Journal of Electronics & Information Technology, 2022, 44(5): 1554-1562.
3	谢志强, 周伟, 杨静. 考虑层级调度次序的资源协同综合调度算法[J]. 计算机集成制造系统, 2022, 28(11): 3391-3402.
	Xie Zhiqiang, Zhou Wei, Yang Jing. Resource Cooperative Integrated Scheduling Algorithm Considering Hierarchical Scheduling Order[J]. Computer Integrated Manufacturing Systems, 2022, 28(11): 3391-3402.
4	谢志强, 张晓欢, 辛宇, 等. 考虑后续工序的择时综合调度算法[J]. 自动化学报, 2018, 44(2): 344-362.
	Xie Zhiqiang, Zhang Xiaohuan, Xin Yu, et al. Time-selective Integrated Scheduling Algorithm Considering Posterior Processes[J]. Acta Automatica Sinica, 2018, 44(2): 344-362.
5	Zhang Cong, Song Wen, Cao Zhiguang, et al. Learning to Dispatch for Job Shop Scheduling Via Deep Reinforcement Learning[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2020: 1621-1632.
6	邵炜世. 基于元启发式的分布式车间调度方法研究[D]. 南京: 南京航空航天大学, 2018.
	Shao Weishi. Research on Meta-heuristic Methods for Distributed Shop Scheduling[D]. Nanjing: Nanjing University of Aeronautics and Astronautics, 2018.
7	石飞, 赵诗奎. 基于工序约束链编码的遗传算法求解产品综合调度问题[J]. 中国机械工程, 2017, 28(20): 2483-2492.
	Shi Fei, Zhao Shikui. Product Comprehensive Scheduling Problems Solved by Genetic Algorithm Based on Operation Constraint Chain Coding[J]. China Mechanical Engineering, 2017, 28(20): 2483-2492.
8	张剑, 袁铭辉, 陈浩杰, 等. 基于改进烟花算法的焊接综合调度研究[J]. 华中科技大学学报(自然科学版), 2022, 50(5): 130-137.
	Zhang Jian, Yuan Minghui, Chen Haojie, et al. Research on Welding Integrated Scheduling Based on Improved Fireworks Algorithm[J]. Journal of Huazhong University of Science and Technology(Nature Science Edition), 2022, 50(5): 130-137.
9	赵诗奎, 韩青, 王桂从. 基于虚拟零部件级别分区编码的产品综合调度算法[J]. 计算机集成制造系统, 2015, 21(9): 2435-2445.
	Zhao Shikui, Han Qing, Wang Guicong. Product Comprehensive Scheduling Algorithm Based on Virtual Component Level Division Coding[J]. Computer Integrated Manufacturing Systems, 2015, 21(9): 2435-2445.
10	陈浩杰. 面向随机工时资源受限项目调度的遗传规划算法研究[D]. 成都: 西南交通大学, 2022.
	Chen Haojie. Research on Genetic Programming for Resource Constrained Project Scheduling with Stochastic Activity Duration[D]. Chengdu: Southwest Jiaotong University, 2022.
11	Zhan Ming, Lu Yang, Hu Youxi, et al. Dynamic Scheduling Method for Job-shop Manufacturing Systems by Deep Reinforcement Learning with Proximal Policy Optimization[J]. Sustainability, 2022, 14(9): 5177.
12	Song Wen, Chen Xinyang, Li Qiang, et al. Flexible Job-shop Scheduling via Graph Neural Network and Deep Reinforcement Learning[J]. IEEE Transactions on Industrial Informatics, 2023, 19(2): 1600-1610.
13	Nour El Houda Hammami, Lardeux Benoit, Hadj-Alouane Atidel B, et al. Job Shop Scheduling: A Novel DRL Approach for Continuous Schedule-generation Facing Real-time Job Arrivals[J]. IFAC-PapersOnLine, 2022, 55(10): 2493-2498.
14	Han B A, Yang J J. A Deep Reinforcement Learning Based Solution for Flexible Job Shop Scheduling Problem[J]. International Journal of Simulation Modelling, 2021, 20(2): 375-386.
15	蒋静静. 基于深度强化学习的离散型制造企业车间动态调度研究[D]. 西安: 西安理工大学, 2020.
	Jiang Jingjing. Research on Jobshop Dynamic Scheduling of Discrete Manufacturig Enterprises Based on Deep Reinforcement Learning[D]. Xi'an: Xi'an University of Technology, 2020.
16	张希肴. 基于深度强化学习的置换流水车间调度方法研究[D]. 武汉: 华中科技大学, 2021.
	Zhang Xiyao. Research on Permutation Flow Shop Scheduling Approaches Based on Deep Reinforcement Learning[D]. Wuhan: Huazhong University of Science and Technology, 2021.
17	Luo Shu. Dynamic Scheduling for Flexible Job Shop with New Job Insertions by Deep Reinforcement Learning[J]. Applied Soft Computing, 2020, 91: 106208.
18	Yan Qi, Wang Hongfeng, Wu Fang. Digital Twin-enabled Dynamic Scheduling with Preventive Maintenance Using a Double-layer Q-learning Algorithm[J]. Computers & Operations Research, 2022, 144: 105823.
19	Chang Xiao, Jia Xiaoliang, Fu Shifeng, et al. Digital Twin and Deep Reinforcement Learning Enabled Real-time Scheduling for Complex Product Flexible Shop-floor[J]. Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture, 2023, 237(8): 1254-1268.
20	Liu Renke, Piplani Rajesh, Toro Carlos. Deep Reinforcement Learning for Dynamic Scheduling of a Flexible Job Shop[J]. International Journal of Production Research, 2022, 60(13): 4049-4069.
21	Johnson Dazzle, Chen Gang, Lu Yuqian. Multi-agent Reinforcement Learning for Real-time Dynamic Production Scheduling in a Robot Assembly Cell[J]. IEEE Robotics and Automation Letters, 2022, 7(3): 7684-7691.
22	Wang Hao, Cheng Junfu, Liu Chang, et al. Multi-objective Reinforcement Learning Framework for Dynamic Flexible Job Shop Scheduling Problem with Uncertain Events[J]. Applied Soft Computing, 2022, 131: 109717.
23	Zhang Zhicong, Zheng Li, Li Na, et al. Minimizing Mean Weighted Tardiness in Unrelated Parallel Machine Scheduling with Reinforcement Learning[J]. Computers & Operations Research, 2012, 39(7): 1315-1324.
24	柳旭东, 赵夙, 朱晓荣. 基于强化学习的自适应网络覆盖优化算法研究[J]. 光通信研究, 2022(5): 66-73.
	Liu Xudong, Zhao Su, Zhu Xiaorong. An Adaptive Network Coverage Optimization Method Based on Reinforcement Learning[J]. Study on Optical Communications, 2022(5): 66-73.
25	Han Baoan, Yang Jianjun. Research on Adaptive Job Shop Scheduling Problems Based on Dueling Double DQN[J]. IEEE Access, 2020, 8: 186474-186495.
26	张铃, 张钹. 神经网络中BP算法的分析[J]. 模式识别与人工智能, 1994, 7(3): 191-195.
	Zhang Ling, Zhang Bo. On the BP Algorithm of Feedforward Neural Networks[J]. Pattern Recognition and Artificial Intelligence, 1994, 7(3): 191-195.
27	高御尧, 石明全, 秦渝, 等. 基于改进非全连接神经网络的站点客流预测模型[J]. 计算机工程, 2023, 49(9): 43-51.
	Gao Yuyao, Shi Mingquan, Qin Yu, et al. Station Ridership Prediction Model Based on Improved Non-fully Connected Neural Network[J]. Computer Engineering, 2023, 49(9): 43-51.
28	Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2017: 6000-6010.
29	王冬雪, 闻翔, 孙慧妮, 等. 基于密集连接注意力一维卷积的航班延误预测[J]. 北京交通大学学报, 2023, 47(2): 95-105.
	Wang Dongxue, Wen Xiang, Sun Huini, et al. Flight Delay Prediction Based on Densely Connected Attention One-dimensional Convolution[J]. Journal of Beijing Jiaotong University, 2023, 47(2): 95-105.
30	吴浩, 陈运涛, 朱赵辉, 等. 改进一维卷积神经网络的隧道围岩收敛变形分级预测[J]. 应用基础与工程科学学报, 2024, 32(1): 145-159.
	Wu Hao, Chen Yuntao, Zhu Zhaohui, et al. Prediction of Tunnel Squeezing Classification Based on Improved One-dimensional Convolutional Neural Network[J]. Journal of Basic Science and Engineering, 2024, 32(1): 145-159.
31	Fortunato M, Azar M G, Piot B, et al. Noisy Networks For Exploration[C]//ICLR 2018. New York: ICLR, 2018: 1-21.
32	杨兴锐, 赵寿为, 张如学, 等. 结合自注意力和残差的BiLSTM_CNN文本分类模型[J]. 计算机工程与应用, 2022, 58(3): 172-180.
	Yang Xingrui, Zhao Shouwei, Zhang Ruxue, et al. BiLSTM_CNN Classification Model Based on Self-attention and Residual Network[J]. Computer Engineering and Applications, 2022, 58(3): 172-180.
33	閤泰梓, 唐秋华, 成丽新. 基于DQN协同进化算法的柔性作业车间能效调度优化[J/OL]. 计算机集成制造系统. (2023-02-27) [2023-09-23]. .
	Ge Taizi, Tang Qiuhua, Cheng Lixin. Energy-efficient Optimization of Flexible Job-shop Scheduling Based on DQN Co-evolutionary Algorithm[J/OL]. Computer Integrated Manufacturing Systems. (2023-02-27) [2023-09-23]. .

算法	MK01	MK02	MK03	MK04	MK05	MK06	MK07	MK08	MK09	MK10	平均值
PR1	105	69	350	189	335	148	305	748	788	603	364.0
PR2	175	157	757	193	410	258	537	977	1 249	1 133	584.6
PR3	92	80	361	189	245	156	313	680	473	582	317.1
PR4	154	122	610	140	278	200	391	714	798	970	437.7
PR5	100	88	398	218	396	178	426	1 153	998	735	469.0
PR6	183	184	1 142	243	441	378	714	1 401	1 440	1 526	765.2
PR7	101	84	456	213	414	196	365	1 296	1 136	888	514.9
PR8	200	162	1 019	309	475	414	788	1 509	1 537	1 532	794.5
AC	80	63	342	139	246	179	312	616	439	565	298.1
DQN	75	50	270	118	229	96	243	652	534	634	290.1
DDQN	73	44	292	119	218	97	249	651	464	444	265.1
D3QN	76	46	333	114	220	98	219	719	390	410	262.5
IGA	57	42	262	108	202	82	201	587	363	316	222.0
IMFFA	54	41	279	108	196	82	203	583	368	311	222.5
ID3QN	74	47	296	111	216	89	221	606	425	390	247.5

算法	MK01	MK02	MK03	MK04	MK05	MK06	MK07	MK08	MK09	MK10	平均值
PR1	0.032	0.033	0.025	0.033	0.026	0.032	0.026	0.027	0.026	0.031	0.029
PR2	0.033	0.027	0.027	0.025	0.026	0.025	0.029	0.025	0.030	0.029	0.028
PR3	0.026	0.025	0.024	0.024	0.024	0.033	0.024	0.029	0.025	0.024	0.027
PR4	0.029	0.024	0.027	0.027	0.029	0.035	0.024	0.025	0.030	0.024	0.027
PR5	0.029	0.024	0.027	0.027	0.029	0.035	0.024	0.025	0.030	0.024	0.027
PR6	0.029	0.024	0.027	0.027	0.029	0.035	0.024	0.025	0.030	0.024	0.027
PR7	0.029	0.024	0.027	0.027	0.029	0.035	0.024	0.025	0.030	0.024	0.027
PR8	0.025	0.026	0.027	0.027	0.026	0.027	0.038	0.033	0.030	0.026	0.028
AC	1.843	0.064	0.381	0.139	0.213	0.402	0.160	1.198	1.327	1.353	0.708
DQN	1.794	0.071	0.429	0.150	0.195	0.412	0.179	1.172	1.389	1.397	0.719
DDQN	1.001	0.109	0.562	0.203	0.385	0.693	0.358	1.694	1.892	2.003	0.890
D3QN	1.976	0.134	0.541	0.207	0.283	0.543	0.236	1.363	1.714	1.900	0.890
IGA	8.358	9.413	28.492	14.316	15.151	26.888	10.132	50.630	56.899	52.328	27.261
IMFFA	5.716	6.520	26.304	11.511	15.064	26.865	13.975	54.626	58.324	56.907	27.581
ID3QN	0.978	0.106	0.544	0.215	0.255	0.506	0.244	1.353	1.506	1.614	0.732