Constructing the Agent Discrete Simulation Based on DEVS Atomic Model

doi:10.16182/j.issn1004731x.joss.21-0263

Abstract

Abstract:

Agents are difficult to be directly modeled and simulated due to the complexity of their own interaction and learning behaviors. Aiming at the common problems in the discrete simulation of the agent, the event transfer mechanism of the discrete event system specification (DEVS) atomic model is applied to express the interaction and learning of an agent. Through the interaction mode of the agent, the transfer control of multi-state external events, the port connection mode, as well as the introduction of reinforcement learning event transfer representation, a discrete simulation construction method of the agent based on the DEVS atomic model is provided. The simulation verification is carried out in the grid world and the cart-pole environment. The experimental results prove the feasibility and effectiveness of the proposed method in constructing the interactive and learning behaviors of the agent.

Key words: agent, discrete event system specification(DEVS), discrete simulation, reinforcement learning, state transition, atomic model

CLC Number:

TP391.9

Xiaohan Wang, Lin Zhang, Yuanjun Laili, Kunyu Xie, Tingchun Hu. Constructing the Agent Discrete Simulation Based on DEVS Atomic Model[J]. Journal of System Simulation, 2022, 34(2): 191-200.

Figures/Tables 17

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Table 1

Q-learning parameter settings

超参数名	值
学习率 $α$	0.7
衰减因子 $γ$	0.5
探索率 $ε$	0.5
算法迭代次数	200

Table 1

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

Fig. 13

Fig. 14

Table 2

Fig. 15

References 19

1	Xie J, Liu C C. Multi-agent Systems and their Applications[J]. Journal of International Council on Electrical Engineering(S2234-8972), 2017, 7(1): 188-197.
2	Van Tendeloo Y, Vangheluwe H. Extending the DEVS Formalism with Initialization Information[J]. arXiv preprint arXiv:, 2018.
3	Seo C, Zeigler B P, Kim D. DEVS Markov Modeling and Simulation: Formal Definition and Implementation[C]//Proceedings of the 4th ACM International Conference of Computing for Engineering and Sciences, 2018: 1-12.
4	Bae J W, Lee G H, Moon I C. Formal Specification Supporting Incremental and Flexible Agent-based Modeling[C]//Proceedings of the 2012 Winter Simulation Conference (WSC). IEEE, 2012: 1-12.
5	Müller J P. Towards a Formal Semantics of Event-based Multi-agent Simulations[C]//International Workshop on Multi-Agent Systems and Agent-Based Simulation. Springer, Berlin, Heidelberg, 2008: 110-126.
6	Barbieri E, Capocchi L, Santucci J F. DEVS Modeling and Simulation of Financial Leverage Effect Based on Markov Decision Process[C]//2018 4th International Conference on Universal Village (UV). IEEE, 2018: 1-5.
7	Capocchi L, Santucci J F, Zeigler B P. Discrete Event Modeling and Simulation Aspects to Improve Machine Learning Systems[C]//2018 4th International Conference on Universal Village (UV). IEEE, 2018: 1-6.
8	Kessler C, Capocchi L, Santucci J F, et al. Hierarchical Markov Decision Process based on DEVS Formalism[C]//2017 Winter Simulation Conference (WSC). IEEE, 2017: 1001-1012.
9	Zhang M. Constructing a Cognitive Agent Model using DEVS Framework for Multi-agent Simulation[C]// Proc. 15th Eur. Agent Syst. Summer School (EASSS), 2013: 1-5.
10	Akplogan M, Quesnel G, Garcia F, et al. Towards a Deliberative Agent System based on DEVS Formalism for Application in Agriculture[C]//Proceedings of the 2010 Summer Computer Simulation Conference. 2010: 250-257.
11	Chow A C H, Zeigler B P. Parallel DEVS: A Parallel, Hierarchical, Modular Modeling Formalism[C]//Proceedings of Winter Simulation Conference. IEEE, 1994: 716-722.
12	孙长银, 穆朝絮. 多智能体深度强化学习的若干关键科学问题[J]. 自动化学报, 2020, 45: 1-12.
	Sun Changyin, Mu Chaoxu. Important Scientific Problems of Multi-agent deep Reinforcement Learning[J]. Acta Automatica Sinica, 2020, 45: 1-12.
13	Haarnoja T, Zhou A, Abbeel P, et al. Soft Actor-critic: Off-policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[C]//International Conference on Machine Learning. PMLR, 2018: 1861-1870.
14	张红旗, 杨峻楠, 张传富. 基于不完全信息随机博弈与 Q-learning 的防御决策方法[J]. 通信学报, 2018, 39(8): 56-68.
	Zhang Hongqi, Yang Junnan, Zhang Chuanfu. Defense Decision-making Method based on Incomplete Information Stochastic Game and Q-learning[J]. Journal on Communications, 2018, 39(8): 56-68.
15	蒲玮, 李雄. 基于扩展 FIPA-ACL 的装备保障 Agent 通信语言[J]. 系统工程理论与实践, 2018, 38(1): 220-228.
	Pu Wei, Li Xiong. Equipment Support Agent Communication Language based on Extended FIPA-ACL[J]. System Engineering Theory&Practice, 2018, 38(1): 220-228.
16	梁凯, 陈志军, 闫学勤. 移动机器人路径规划仿真研究[J]. 现代电子技术, 2018, 41(17): 6.
	Liang Kai, Chen Zhijun, Yan Xueqin. Simulation Study on Effective Path Planning for Mobile Robot[J]. Modern Electronics Technique, 2018, 41(17): 6.
17	陈建平, 邹锋, 刘全, 等. 一种基于生成对抗网络的强化学习算法[J]. 计算机科学, 2019, 46(10): 265-272.
	Chen Jianping, Zou Feng, Liu Quan, etc. Reinforcement Learning Algorithm Based on Generative Adversarial Networks[J]. Computer Science, ,2019, 46(10): 265-272.
18	Van Tendeloo Y, Vangheluwe H. The Modular Architecture of the Python (P) DEVS Simulation Kernel[C]//Proceedings of the 2014 Symposium on Theory of Modeling and Simulation-DEVS. 2014: 387-392.
19	Sutton R S, Barto A G. Reinforcement Learning: An Introduction[J]. IEEE Transactions on Neural Networks(S1045-9227), 1998, 9(5): 1054.

超参数名	值
学习率α	0.000 1
衰减因子γ	0.99
策略网络模型尺寸	4×64×128×1
算法迭代次数	700

[1]	Jiang Ming, He Tao. Solving the Vehicle Routing Problem Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(9): 2177-2187.
[2]	Ni Peilong, Mao Pengjun, Wang Ning, Yang Mengjie. Robot Path Planning Based on Improved A-DDQN Algorithm [J]. Journal of System Simulation, 2025, 37(9): 2420-2430.
[3]	Chen Zhen, Wu Zhuoyi, Zhang Lin. Research on Policy Representation in Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(7): 1753-1769.
[4]	Piao Jinghua, Gao Chen, Zhang Fang, Su Jun, Li Yong. Large-scale Social Simulator: Frontiers and Perspectives [J]. Journal of System Simulation, 2025, 37(7): 1770-1790.
[5]	Zhang Yue, Zhang Wenliang, Feng Qiang, Guo Xing, Ren Yi, Wang Zili. Combat-oriented Comprehensive Simulation and Verification Technology for Equipment System RMS [J]. Journal of System Simulation, 2025, 37(7): 1823-1835.
[6]	Wan Yuhang, Zhu Zilu, Zhong Chunfu, Liu Yongkui, Lin Tingyu, Zhang Lin. Dynamic Path Planning for Robotic Arms Based on an Improved PPO Algorithm [J]. Journal of System Simulation, 2025, 37(6): 1462-1473.
[7]	Gu Xueqiang, Luo Junren, Zhou Yanzhong, Zhang Wanpeng. Survey on Large Language Agent Technologies for Intelligent Game Theoretic Decision-making [J]. Journal of System Simulation, 2025, 37(5): 1142-1157.
[8]	Wu Guohua, Zeng Jiaheng, Wang Dezhi, Zheng Long, Zou Wei. A Quadrotor Trajectory Tracking Control Method Based on Deep Reinforcement Learning [J]. Journal of System Simulation, 2025, 37(5): 1169-1187.
[9]	Yao Changhua, Bi Shanning, Ma Rufei, Yu Xiaohan, Li Jiaqiang, Chen Jinli. Method for Dynamic Coalition Formation of Wargame Agent for Force Cooperation [J]. Journal of System Simulation, 2025, 37(5): 1188-1196.
[10]	Xu Ming, Li Jinye, Zuo Dongyu, Zhang Jing. Signal Timing Optimization via Reinforcement Learning with Traffic Flow Prediction [J]. Journal of System Simulation, 2025, 37(4): 1051-1062.
[11]	Wang Xin, Cui Chenggang, Wang Xiangxiang, Zhu Ping. Research on Economic Dispatching Strategy of CHP Units Based on SRL [J]. Journal of System Simulation, 2025, 37(4): 968-981.
[12]	Ding Dian, Wei Guangfen, Cao Zheng, Wen Shaohui. Simulation Environment Construction of Track Segment Association and Algorithm Performance Evaluation [J]. Journal of System Simulation, 2025, 37(4): 982-992.
[13]	Zhang Lei, Zhang Xuechao, Wang Chao, Bo Xianglei. An Intelligent Ambulance Regulation Model Based on Online Reinforcement Learning Algorithm [J]. Journal of System Simulation, 2025, 37(3): 584-594.
[14]	Bai Zhenzu, Hou Yizhi, He Zhangming, Wei Juhui, Zhou Haiyin, Wang Jiongqi. Optimization of Dynamic Weapon Target Assignment Considering Random Disturbances [J]. Journal of System Simulation, 2025, 37(12): 2967-2980.
[15]	Xiong Jun, Zhang Wenbo, Xiong Zhi, Zhou Feng, Yang Bo. Survey of Cooperative Multi-Agent Path Finding [J]. Journal of System Simulation, 2025, 37(12): 3033-3049.