基于DEVS原子模型的智能体离散仿真构建方法

doi:10.16182/j.issn1004731x.joss.21-0263

系统仿真学报 ›› 2022, Vol. 34 ›› Issue (2): 191-200.doi: 10.16182/j.issn1004731x.joss.21-0263

基于DEVS原子模型的智能体离散仿真构建方法

王霄汉^1,²(), 张霖^1,²(), 赖李媛君^1,², 谢堃钰^1,², 胡听春¹

^1.北京航空航天大学，北京 100191
^2.复杂产品先进制造系统教育部工程研究中心，北京 100191

收稿日期:2021-03-29 修回日期:2021-04-01 出版日期:2022-02-18 发布日期:2022-02-23
通讯作者: 张霖 E-mail:by1903042@buaa.edu.cn;johnlin9999@163.com
作者简介:王霄汉(1998-)，男，博士生，研究方向为离散仿真、多智能体系统和强化学习。E-mail：by1903042@buaa.edu.cn
基金资助:
国家重点研发计划(2018YFB1701600)

Constructing the Agent Discrete Simulation Based on DEVS Atomic Model

Xiaohan Wang^1,²(), Lin Zhang^1,²(), Yuanjun Laili^1,², Kunyu Xie^1,², Tingchun Hu¹

^1.Beihang University, Beijing 100191, China
^2.Engineering Research Center of Complex Product Advanced Manufacturing Systems, Ministry of Education, Beijing 100191, China

Received:2021-03-29 Revised:2021-04-01 Online:2022-02-18 Published:2022-02-23
Contact: Lin Zhang E-mail:by1903042@buaa.edu.cn;johnlin9999@163.com

摘要/Abstract

摘要：

智能体由于自身交互行为与学习行为的复杂性，难以直接被建模和仿真。针对智能体离散仿真中的常见问题，借助DEVS（discrete event system specification）原子模型的事件转移机制表示智能体的交互与学习过程，通过对智能体交互模式、多状态外部事件转移控制、端口连接模式、以及强化学习事件转移表示等原子模型下智能体建模技术的介绍，给出了基于DEVS原子模型的智能体离散仿真构建方法。在网格世界与倒立摆2个环境中进行仿真验证，实验结果证明了提出方法在构建智能体交互行为和学习行为的可行性和有效性。

关键词: 智能体, DEVS, 离散仿真, 强化学习, 状态转移, 原子模型

Abstract:

Agents are difficult to be directly modeled and simulated due to the complexity of their own interaction and learning behaviors. Aiming at the common problems in the discrete simulation of the agent, the event transfer mechanism of the discrete event system specification (DEVS) atomic model is applied to express the interaction and learning of an agent. Through the interaction mode of the agent, the transfer control of multi-state external events, the port connection mode, as well as the introduction of reinforcement learning event transfer representation, a discrete simulation construction method of the agent based on the DEVS atomic model is provided. The simulation verification is carried out in the grid world and the cart-pole environment. The experimental results prove the feasibility and effectiveness of the proposed method in constructing the interactive and learning behaviors of the agent.

Key words: agent, discrete event system specification(DEVS), discrete simulation, reinforcement learning, state transition, atomic model

中图分类号:

TP391.9

王霄汉, 张霖, 赖李媛君, 谢堃钰, 胡听春. 基于DEVS原子模型的智能体离散仿真构建方法[J]. 系统仿真学报, 2022, 34(2): 191-200.

Xiaohan Wang, Lin Zhang, Yuanjun Laili, Kunyu Xie, Tingchun Hu. Constructing the Agent Discrete Simulation Based on DEVS Atomic Model[J]. Journal of System Simulation, 2022, 34(2): 191-200.

图/表 17

图1

图2

图3

图4

图5

图6

表1

智能体Q-learning参数设置

超参数名	值
学习率 $α$	0.7
衰减因子 $γ$	0.5
探索率 $ε$	0.5
算法迭代次数	200

表1

图7

图8

图9

图10

图11

图12

图13

图14

表2

图15

参考文献 19

1	Xie J, Liu C C. Multi-agent Systems and their Applications[J]. Journal of International Council on Electrical Engineering(S2234-8972), 2017, 7(1): 188-197.
2	Van Tendeloo Y, Vangheluwe H. Extending the DEVS Formalism with Initialization Information[J]. arXiv preprint arXiv:, 2018.
3	Seo C, Zeigler B P, Kim D. DEVS Markov Modeling and Simulation: Formal Definition and Implementation[C]//Proceedings of the 4th ACM International Conference of Computing for Engineering and Sciences, 2018: 1-12.
4	Bae J W, Lee G H, Moon I C. Formal Specification Supporting Incremental and Flexible Agent-based Modeling[C]//Proceedings of the 2012 Winter Simulation Conference (WSC). IEEE, 2012: 1-12.
5	Müller J P. Towards a Formal Semantics of Event-based Multi-agent Simulations[C]//International Workshop on Multi-Agent Systems and Agent-Based Simulation. Springer, Berlin, Heidelberg, 2008: 110-126.
6	Barbieri E, Capocchi L, Santucci J F. DEVS Modeling and Simulation of Financial Leverage Effect Based on Markov Decision Process[C]//2018 4th International Conference on Universal Village (UV). IEEE, 2018: 1-5.
7	Capocchi L, Santucci J F, Zeigler B P. Discrete Event Modeling and Simulation Aspects to Improve Machine Learning Systems[C]//2018 4th International Conference on Universal Village (UV). IEEE, 2018: 1-6.
8	Kessler C, Capocchi L, Santucci J F, et al. Hierarchical Markov Decision Process based on DEVS Formalism[C]//2017 Winter Simulation Conference (WSC). IEEE, 2017: 1001-1012.
9	Zhang M. Constructing a Cognitive Agent Model using DEVS Framework for Multi-agent Simulation[C]// Proc. 15th Eur. Agent Syst. Summer School (EASSS), 2013: 1-5.
10	Akplogan M, Quesnel G, Garcia F, et al. Towards a Deliberative Agent System based on DEVS Formalism for Application in Agriculture[C]//Proceedings of the 2010 Summer Computer Simulation Conference. 2010: 250-257.
11	Chow A C H, Zeigler B P. Parallel DEVS: A Parallel, Hierarchical, Modular Modeling Formalism[C]//Proceedings of Winter Simulation Conference. IEEE, 1994: 716-722.
12	孙长银, 穆朝絮. 多智能体深度强化学习的若干关键科学问题[J]. 自动化学报, 2020, 45: 1-12.
	Sun Changyin, Mu Chaoxu. Important Scientific Problems of Multi-agent deep Reinforcement Learning[J]. Acta Automatica Sinica, 2020, 45: 1-12.
13	Haarnoja T, Zhou A, Abbeel P, et al. Soft Actor-critic: Off-policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[C]//International Conference on Machine Learning. PMLR, 2018: 1861-1870.
14	张红旗, 杨峻楠, 张传富. 基于不完全信息随机博弈与 Q-learning 的防御决策方法[J]. 通信学报, 2018, 39(8): 56-68.
	Zhang Hongqi, Yang Junnan, Zhang Chuanfu. Defense Decision-making Method based on Incomplete Information Stochastic Game and Q-learning[J]. Journal on Communications, 2018, 39(8): 56-68.
15	蒲玮, 李雄. 基于扩展 FIPA-ACL 的装备保障 Agent 通信语言[J]. 系统工程理论与实践, 2018, 38(1): 220-228.
	Pu Wei, Li Xiong. Equipment Support Agent Communication Language based on Extended FIPA-ACL[J]. System Engineering Theory&Practice, 2018, 38(1): 220-228.
16	梁凯, 陈志军, 闫学勤. 移动机器人路径规划仿真研究[J]. 现代电子技术, 2018, 41(17): 6.
	Liang Kai, Chen Zhijun, Yan Xueqin. Simulation Study on Effective Path Planning for Mobile Robot[J]. Modern Electronics Technique, 2018, 41(17): 6.
17	陈建平, 邹锋, 刘全, 等. 一种基于生成对抗网络的强化学习算法[J]. 计算机科学, 2019, 46(10): 265-272.
	Chen Jianping, Zou Feng, Liu Quan, etc. Reinforcement Learning Algorithm Based on Generative Adversarial Networks[J]. Computer Science, ,2019, 46(10): 265-272.
18	Van Tendeloo Y, Vangheluwe H. The Modular Architecture of the Python (P) DEVS Simulation Kernel[C]//Proceedings of the 2014 Symposium on Theory of Modeling and Simulation-DEVS. 2014: 387-392.
19	Sutton R S, Barto A G. Reinforcement Learning: An Introduction[J]. IEEE Transactions on Neural Networks(S1045-9227), 1998, 9(5): 1054.

超参数名	值
学习率α	0.000 1
衰减因子γ	0.99
策略网络模型尺寸	4×64×128×1
算法迭代次数	700

[1]	罗俊仁, 张万鹏, 袁唯淋, 胡振震, 陈少飞, 陈璟. 面向多智能体博弈对抗的对手建模框架[J]. 系统仿真学报, 2022, 34(9): 1941-1955.
[2]	谢启苗, 郭帅帅. 基于社会力模型的客船人员疏散仿真研究[J]. 系统仿真学报, 2022, 34(8): 1710-1724.
[3]	陆淼嘉, 黄承媛, 滕靖. 基于多智能体的网购生鲜无人车配送调度仿真[J]. 系统仿真学报, 2022, 34(6): 1185-1195.
[4]	赵也践, 王艳红, 张俊, 于洪霞, 田中大. 改进Q学习算法在作业车间调度问题中的应用[J]. 系统仿真学报, 2022, 34(6): 1247-1258.
[5]	张森, 张孟炎, 邵敬平, 普杰信. 基于随机策略搜索的多机三维路径规划方法[J]. 系统仿真学报, 2022, 34(6): 1286-1295.
[6]	倪凌佳, 黄晓霞, 李红旮, 张子博. 基于协作式深度强化学习的火灾应急疏散仿真研究[J]. 系统仿真学报, 2022, 34(6): 1353-1366.
[7]	王红微, 杨鹏. 基于深度强化学习的机场货运业务优化研究[J]. 系统仿真学报, 2022, 34(3): 651-660.
[8]	李锋, 魏莹. 社会学习和参照点效应对企业产品决策的影响[J]. 系统仿真学报, 2022, 34(2): 234-246.
[9]	李启锐, 彭心怡. 基于深度强化学习的云作业调度及仿真研究[J]. 系统仿真学报, 2022, 34(2): 258-268.
[10]	吴曦, 孟祥林, 杨镜宇. 下一代战略博弈推演系统研究[J]. 系统仿真学报, 2021, 33(9): 2017-2024.
[11]	贺筱媛, 郭圣明, 吴琳, 李东, 许霄, 李丽. 面向智能化兵棋的认知行为建模方法研究[J]. 系统仿真学报, 2021, 33(9): 2037-2047.
[12]	赵毓, 郭继峰, 颜鹏, 白成超. 稀疏奖励下多航天器规避决策自学习仿真[J]. 系统仿真学报, 2021, 33(8): 1766-1774.
[13]	桂欣冬, 吉鸿江, 范玲玲, 刘世达. 信任驱动的自适应协调控制算法及应用[J]. 系统仿真学报, 2021, 33(8): 1809-1817.
[14]	李文翔, 李晔, 董洁霜, 李一鸣. 引入碳交易机制的新能源汽车发展路径研究[J]. 系统仿真学报, 2021, 33(6): 1451-1465.
[15]	李锋, 魏莹. 复杂网络对羊群效应现象影响的仿真研究[J]. 系统仿真学报, 2021, 33(3): 539-553.

基于DEVS原子模型的智能体离散仿真构建方法

Constructing the Agent Discrete Simulation Based on DEVS Atomic Model

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献 19

相关文章 15

编辑推荐

Metrics

本文评价