Constructing the Agent Discrete Simulation Based on DEVS Atomic Model

doi:10.16182/j.issn1004731x.joss.21-0263

Abstract

Abstract:

Agents are difficult to be directly modeled and simulated due to the complexity of their own interaction and learning behaviors. Aiming at the common problems in the discrete simulation of the agent, the event transfer mechanism of the discrete event system specification (DEVS) atomic model is applied to express the interaction and learning of an agent. Through the interaction mode of the agent, the transfer control of multi-state external events, the port connection mode, as well as the introduction of reinforcement learning event transfer representation, a discrete simulation construction method of the agent based on the DEVS atomic model is provided. The simulation verification is carried out in the grid world and the cart-pole environment. The experimental results prove the feasibility and effectiveness of the proposed method in constructing the interactive and learning behaviors of the agent.

Key words: agent, discrete event system specification(DEVS), discrete simulation, reinforcement learning, state transition, atomic model

CLC Number:

TP391.9

Xiaohan Wang, Lin Zhang, Yuanjun Laili, Kunyu Xie, Tingchun Hu. Constructing the Agent Discrete Simulation Based on DEVS Atomic Model[J]. Journal of System Simulation, 2022, 34(2): 191-200.

Figures/Tables 17

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Table 1

Q-learning parameter settings

超参数名	值
学习率 $α$	0.7
衰减因子 $γ$	0.5
探索率 $ε$	0.5
算法迭代次数	200

Table 1

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Fig. 14

Table 2

Fig. 15

References 19

1	Xie J, Liu C C. Multi-agent Systems and their Applications[J]. Journal of International Council on Electrical Engineering(S2234-8972), 2017, 7(1): 188-197.
2	Van Tendeloo Y, Vangheluwe H. Extending the DEVS Formalism with Initialization Information[J]. arXiv preprint arXiv:, 2018.
3	Seo C, Zeigler B P, Kim D. DEVS Markov Modeling and Simulation: Formal Definition and Implementation[C]//Proceedings of the 4th ACM International Conference of Computing for Engineering and Sciences, 2018: 1-12.
4	Bae J W, Lee G H, Moon I C. Formal Specification Supporting Incremental and Flexible Agent-based Modeling[C]//Proceedings of the 2012 Winter Simulation Conference (WSC). IEEE, 2012: 1-12.
5	Müller J P. Towards a Formal Semantics of Event-based Multi-agent Simulations[C]//International Workshop on Multi-Agent Systems and Agent-Based Simulation. Springer, Berlin, Heidelberg, 2008: 110-126.
6	Barbieri E, Capocchi L, Santucci J F. DEVS Modeling and Simulation of Financial Leverage Effect Based on Markov Decision Process[C]//2018 4th International Conference on Universal Village (UV). IEEE, 2018: 1-5.
7	Capocchi L, Santucci J F, Zeigler B P. Discrete Event Modeling and Simulation Aspects to Improve Machine Learning Systems[C]//2018 4th International Conference on Universal Village (UV). IEEE, 2018: 1-6.
8	Kessler C, Capocchi L, Santucci J F, et al. Hierarchical Markov Decision Process based on DEVS Formalism[C]//2017 Winter Simulation Conference (WSC). IEEE, 2017: 1001-1012.
9	Zhang M. Constructing a Cognitive Agent Model using DEVS Framework for Multi-agent Simulation[C]// Proc. 15th Eur. Agent Syst. Summer School (EASSS), 2013: 1-5.
10	Akplogan M, Quesnel G, Garcia F, et al. Towards a Deliberative Agent System based on DEVS Formalism for Application in Agriculture[C]//Proceedings of the 2010 Summer Computer Simulation Conference. 2010: 250-257.
11	Chow A C H, Zeigler B P. Parallel DEVS: A Parallel, Hierarchical, Modular Modeling Formalism[C]//Proceedings of Winter Simulation Conference. IEEE, 1994: 716-722.
12	孙长银, 穆朝絮. 多智能体深度强化学习的若干关键科学问题[J]. 自动化学报, 2020, 45: 1-12.
	Sun Changyin, Mu Chaoxu. Important Scientific Problems of Multi-agent deep Reinforcement Learning[J]. Acta Automatica Sinica, 2020, 45: 1-12.
13	Haarnoja T, Zhou A, Abbeel P, et al. Soft Actor-critic: Off-policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[C]//International Conference on Machine Learning. PMLR, 2018: 1861-1870.
14	张红旗, 杨峻楠, 张传富. 基于不完全信息随机博弈与 Q-learning 的防御决策方法[J]. 通信学报, 2018, 39(8): 56-68.
	Zhang Hongqi, Yang Junnan, Zhang Chuanfu. Defense Decision-making Method based on Incomplete Information Stochastic Game and Q-learning[J]. Journal on Communications, 2018, 39(8): 56-68.
15	蒲玮, 李雄. 基于扩展 FIPA-ACL 的装备保障 Agent 通信语言[J]. 系统工程理论与实践, 2018, 38(1): 220-228.
	Pu Wei, Li Xiong. Equipment Support Agent Communication Language based on Extended FIPA-ACL[J]. System Engineering Theory&Practice, 2018, 38(1): 220-228.
16	梁凯, 陈志军, 闫学勤. 移动机器人路径规划仿真研究[J]. 现代电子技术, 2018, 41(17): 6.
	Liang Kai, Chen Zhijun, Yan Xueqin. Simulation Study on Effective Path Planning for Mobile Robot[J]. Modern Electronics Technique, 2018, 41(17): 6.
17	陈建平, 邹锋, 刘全, 等. 一种基于生成对抗网络的强化学习算法[J]. 计算机科学, 2019, 46(10): 265-272.
	Chen Jianping, Zou Feng, Liu Quan, etc. Reinforcement Learning Algorithm Based on Generative Adversarial Networks[J]. Computer Science, ,2019, 46(10): 265-272.
18	Van Tendeloo Y, Vangheluwe H. The Modular Architecture of the Python (P) DEVS Simulation Kernel[C]//Proceedings of the 2014 Symposium on Theory of Modeling and Simulation-DEVS. 2014: 387-392.
19	Sutton R S, Barto A G. Reinforcement Learning: An Introduction[J]. IEEE Transactions on Neural Networks(S1045-9227), 1998, 9(5): 1054.

超参数名	值
学习率α	0.000 1
衰减因子γ	0.99
策略网络模型尺寸	4×64×128×1
算法迭代次数	700

[1]	Junren Luo, Wanpeng Zhang, Weilin Yuan, Zhenzhen Hu, Shaofei Chen, Jing Chen. Research on Opponent Modeling Framework for Multi-agent Game Confrontation [J]. Journal of System Simulation, 2022, 34(9): 1941-1955.
[2]	Qimiao Xie, Shuaishuai Guo. Research on Passenger Ship Evacuation Simulation Based on Social Force Model [J]. Journal of System Simulation, 2022, 34(8): 1710-1724.
[3]	Miaojia Lu, Chengyuan Huang, Jing Teng. Multi-agent Simulation for Online Fresh Food Autonomous Delivery [J]. Journal of System Simulation, 2022, 34(6): 1185-1195.
[4]	Yejian Zhao, Yanhong Wang, Jun Zhang, Hongxia Yu, Zhongda Tian. Application of Improved Q Learning Algorithm in Job Shop Scheduling Problem [J]. Journal of System Simulation, 2022, 34(6): 1247-1258.
[5]	Sen Zhang, Mengyan Zhang, Jingping Shao, Jiexin Pu. Multi-UAVs 3D Path Planning Method Based on Random Strategy Search [J]. Journal of System Simulation, 2022, 34(6): 1286-1295.
[6]	Lingjia Ni, Xiaoxia Huang, Hongga Li, Zibo Zhang. Research on Fire Emergency Evacuation Simulation Based on Cooperative Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(6): 1353-1366.
[7]	Xin Zhou, Weiping Wang, Yifan Zhu, Tao Wang, Tian Jing. An Unmanned Swarm Search Method Based on Human-Robot Cooperation [J]. Journal of System Simulation, 2022, 34(4): 735-744.
[8]	Hongwei Wang, Peng Yang. Research on Optimization of Airport Cargo Business Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(3): 651-660.
[9]	Feng Li, Ying Wei. Product Decisions in Presence of Social Learning and Reference Point Effect [J]. Journal of System Simulation, 2022, 34(2): 234-246.
[10]	Qirui Li, Xinyi Peng. Job Scheduling and Simulation in Cloud Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2022, 34(2): 258-268.
[11]	Qi Xiaolong, Yang Xuguang. Time-varying Output Formation Tracking Control of Discrete-time Heterogeneous Multi-agent Systems [J]. Journal of System Simulation, 2022, 34(1): 36-44.
[12]	Zhao Zheng, Hu Li, Qian Yuanyuan, Jin Hui, Jia Aiping. Research on Optimal Allocation Method of Aircraft Towing Rules Based on Multi-agent [J]. Journal of System Simulation, 2022, 34(1): 113-125.
[13]	Wu Xi, Meng Xianglin, Yang Jingyu. Study on Next-generation Strategic Wargame System [J]. Journal of System Simulation, 2021, 33(9): 2017-2024.
[14]	He Xiaoyuan, Guo Shengming, Wu Lin, Li Dong, Xu Xiao, Li Li. Modeling Research of Cognition Behavior for Intelligent Wargaming [J]. Journal of System Simulation, 2021, 33(9): 2037-2047.
[15]	Cong Jiping, Cui Lijie, Ding Gang, Ren Bo. Simulation on Aviation Maintenance Support System Based on Goal-driven [J]. Journal of System Simulation, 2021, 33(9): 2157-2165.