Constructing the Agent Discrete Simulation Based on DEVS Atomic Model

doi:10.16182/j.issn1004731x.joss.21-0263

Abstract

Abstract:

Agents are difficult to be directly modeled and simulated due to the complexity of their own interaction and learning behaviors. Aiming at the common problems in the discrete simulation of the agent, the event transfer mechanism of the discrete event system specification (DEVS) atomic model is applied to express the interaction and learning of an agent. Through the interaction mode of the agent, the transfer control of multi-state external events, the port connection mode, as well as the introduction of reinforcement learning event transfer representation, a discrete simulation construction method of the agent based on the DEVS atomic model is provided. The simulation verification is carried out in the grid world and the cart-pole environment. The experimental results prove the feasibility and effectiveness of the proposed method in constructing the interactive and learning behaviors of the agent.

Key words: agent, discrete event system specification(DEVS), discrete simulation, reinforcement learning, state transition, atomic model

CLC Number:

TP391.9

Xiaohan Wang, Lin Zhang, Yuanjun Laili, Kunyu Xie, Tingchun Hu. Constructing the Agent Discrete Simulation Based on DEVS Atomic Model[J]. Journal of System Simulation, 2022, 34(2): 191-200.

Figures/Tables 17

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Table 1

Q-learning parameter settings

超参数名	值
学习率 $α$	0.7
衰减因子 $γ$	0.5
探索率 $ε$	0.5
算法迭代次数	200

Table 1

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Fig. 14

Table 2

Fig. 15

References 19

1	Xie J, Liu C C. Multi-agent Systems and their Applications[J]. Journal of International Council on Electrical Engineering(S2234-8972), 2017, 7(1): 188-197.
2	Van Tendeloo Y, Vangheluwe H. Extending the DEVS Formalism with Initialization Information[J]. arXiv preprint arXiv:, 2018.
3	Seo C, Zeigler B P, Kim D. DEVS Markov Modeling and Simulation: Formal Definition and Implementation[C]//Proceedings of the 4th ACM International Conference of Computing for Engineering and Sciences, 2018: 1-12.
4	Bae J W, Lee G H, Moon I C. Formal Specification Supporting Incremental and Flexible Agent-based Modeling[C]//Proceedings of the 2012 Winter Simulation Conference (WSC). IEEE, 2012: 1-12.
5	Müller J P. Towards a Formal Semantics of Event-based Multi-agent Simulations[C]//International Workshop on Multi-Agent Systems and Agent-Based Simulation. Springer, Berlin, Heidelberg, 2008: 110-126.
6	Barbieri E, Capocchi L, Santucci J F. DEVS Modeling and Simulation of Financial Leverage Effect Based on Markov Decision Process[C]//2018 4th International Conference on Universal Village (UV). IEEE, 2018: 1-5.
7	Capocchi L, Santucci J F, Zeigler B P. Discrete Event Modeling and Simulation Aspects to Improve Machine Learning Systems[C]//2018 4th International Conference on Universal Village (UV). IEEE, 2018: 1-6.
8	Kessler C, Capocchi L, Santucci J F, et al. Hierarchical Markov Decision Process based on DEVS Formalism[C]//2017 Winter Simulation Conference (WSC). IEEE, 2017: 1001-1012.
9	Zhang M. Constructing a Cognitive Agent Model using DEVS Framework for Multi-agent Simulation[C]// Proc. 15th Eur. Agent Syst. Summer School (EASSS), 2013: 1-5.
10	Akplogan M, Quesnel G, Garcia F, et al. Towards a Deliberative Agent System based on DEVS Formalism for Application in Agriculture[C]//Proceedings of the 2010 Summer Computer Simulation Conference. 2010: 250-257.
11	Chow A C H, Zeigler B P. Parallel DEVS: A Parallel, Hierarchical, Modular Modeling Formalism[C]//Proceedings of Winter Simulation Conference. IEEE, 1994: 716-722.
12	孙长银, 穆朝絮. 多智能体深度强化学习的若干关键科学问题[J]. 自动化学报, 2020, 45: 1-12.
	Sun Changyin, Mu Chaoxu. Important Scientific Problems of Multi-agent deep Reinforcement Learning[J]. Acta Automatica Sinica, 2020, 45: 1-12.
13	Haarnoja T, Zhou A, Abbeel P, et al. Soft Actor-critic: Off-policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[C]//International Conference on Machine Learning. PMLR, 2018: 1861-1870.
14	张红旗, 杨峻楠, 张传富. 基于不完全信息随机博弈与 Q-learning 的防御决策方法[J]. 通信学报, 2018, 39(8): 56-68.
	Zhang Hongqi, Yang Junnan, Zhang Chuanfu. Defense Decision-making Method based on Incomplete Information Stochastic Game and Q-learning[J]. Journal on Communications, 2018, 39(8): 56-68.
15	蒲玮, 李雄. 基于扩展 FIPA-ACL 的装备保障 Agent 通信语言[J]. 系统工程理论与实践, 2018, 38(1): 220-228.
	Pu Wei, Li Xiong. Equipment Support Agent Communication Language based on Extended FIPA-ACL[J]. System Engineering Theory&Practice, 2018, 38(1): 220-228.
16	梁凯, 陈志军, 闫学勤. 移动机器人路径规划仿真研究[J]. 现代电子技术, 2018, 41(17): 6.
	Liang Kai, Chen Zhijun, Yan Xueqin. Simulation Study on Effective Path Planning for Mobile Robot[J]. Modern Electronics Technique, 2018, 41(17): 6.
17	陈建平, 邹锋, 刘全, 等. 一种基于生成对抗网络的强化学习算法[J]. 计算机科学, 2019, 46(10): 265-272.
	Chen Jianping, Zou Feng, Liu Quan, etc. Reinforcement Learning Algorithm Based on Generative Adversarial Networks[J]. Computer Science, ,2019, 46(10): 265-272.
18	Van Tendeloo Y, Vangheluwe H. The Modular Architecture of the Python (P) DEVS Simulation Kernel[C]//Proceedings of the 2014 Symposium on Theory of Modeling and Simulation-DEVS. 2014: 387-392.
19	Sutton R S, Barto A G. Reinforcement Learning: An Introduction[J]. IEEE Transactions on Neural Networks(S1045-9227), 1998, 9(5): 1054.

超参数名	值
学习率α	0.000 1
衰减因子γ	0.99
策略网络模型尺寸	4×64×128×1
算法迭代次数	700

[1]	Dong Zhiming, Hu Zhongqi, Dai Haoran, Gao Jiancheng. An Automated Generation Method for Combat Simulation Scenarios Based on Large Language Models [J]. Journal of System Simulation, 2026, 38(5): 1129-1145.
[2]	Zhou Zicong, Zeng Junjie, Hu Yue, Zhu Zhengqiu, Yin Quanjun. Multi-agent Reinforcement Learning Method for Wargame Simulation Based on Suboptimal Demonstration Guidance [J]. Journal of System Simulation, 2026, 38(5): 1277-1289.
[3]	Li Guozheng, Wang Rui, Fan Shichao, Cai Xintong, Zhai Xinyue. Review on Optimization of Simulation Modeling Strategies for Spacecraft Orbit Avoidance [J]. Journal of System Simulation, 2026, 38(4): 855-868.
[4]	Li Dequan, Xiong Wan. Robot Path Planning by Reinforcement Learning Based on SAC3Q-HDM [J]. Journal of System Simulation, 2026, 38(3): 714-724.
[5]	Liu Dayong, Dong Zhiming, Guo Qisheng, Gao Ang, Qiu Xuehuan. Construction Approach of LLM-empowered Tactical Wargame Decision-making Agents [J]. Journal of System Simulation, 2026, 38(3): 758-775.
[6]	Li Jiting, Sun Yi, Wang Yirong, Lin Yiqin, Jia Jun, Ding Gangsong. LLM-driven Multi-agent Social Network Simulation: Interdisciplinary Integration and Cutting-edge Development [J]. Journal of System Simulation, 2026, 38(2): 235-260.
[7]	Zhang Mingxin, Wu Jinxuan, Zhu Rui, Wang Yunlong, Meng Wenjuan, Liu Zhe, Li Xu, Chen Xiaolei, Liang Yuxuan, Zheng Yi, Xue Xiangyang. Social Cognition Simulation with Large Language Model-driven Agents [J]. Journal of System Simulation, 2026, 38(2): 261-277.
[8]	Yang Can, Chen Kai, Zhu Feng. Reinforcement Learning Based Method for UAV Team Orienteering Optimization under Multi-constraint Condition [J]. Journal of System Simulation, 2026, 38(2): 360-371.
[9]	Zheng Wei, Tang Jiahao, Xiong Xiaoping, Fan Xin. Intelligent Decision-making Method in Imbalanced Air Combat Based on Asymmetric Self-play [J]. Journal of System Simulation, 2026, 38(2): 433-446.
[10]	Ding Zhengkun, Liu Jiaqi, Xu Junzheng, Xu Yuezhu, Wang Xingmei. Intelligent Air Combat Decision-making Method Based on BiGRU and Priority Dynamic Sampling [J]. Journal of System Simulation, 2026, 38(2): 447-459.
[11]	Tao Caixia, Chen Naikun, Gao Fengyang, Zhang Jiangang. Distributed Optimization for Integrated Energy Based on Multi-agent Reinforcement Learning [J]. Journal of System Simulation, 2026, 38(2): 476-487.
[12]	Wang Yifan, Yang Bin, Wang Congjun. Simulation Method for Multi-crew Construction Processes Based on Large Language Model-powered Agent [J]. Journal of System Simulation, 2026, 38(2): 488-500.
[13]	Jiang Ming, He Tao. Solving the Vehicle Routing Problem Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(9): 2177-2187.
[14]	Ni Peilong, Mao Pengjun, Wang Ning, Yang Mengjie. Robot Path Planning Based on Improved A-DDQN Algorithm [J]. Journal of System Simulation, 2025, 37(9): 2420-2430.
[15]	Chen Zhen, Wu Zhuoyi, Zhang Lin. Research on Policy Representation in Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(7): 1753-1769.