Research on Requirements and Methods for Intelligent Assessment of Simulation Credibility

doi:10.16182/j.issn1004731x.joss.25-0510

Abstract

Abstract:

The accuracy of simulations in representing real-world systems is a critical concern for users. Simulation credibility assessment ensures trustworthiness by evaluating the correctness and effectiveness of simulations to meet application requirements. As simulation technologies are widely adopted, and new simulation paradigms emerge, traditional assessment methods are increasingly showing limitations in their dependence on experts, data processing capabilities, and assessment efficiency. This paper systematically reviewed the research demands, current progress, new technologies, and future trends of intelligent simulation credibility assessment. Based on the simulation credibility assessment process and problem analysis, the requirements for intelligent credibility assessment were discussed. Intelligent technologies were classified, and their applications in credibility assessment were summarized, with four emerging intelligent assessment methods being presented. Potential research directions for intelligent simulation credibility assessment were proposed.

Key words: simulation credibility, intelligent assessment, deep learning, perceptual computing, knowledge graph

CLC Number:

TP391.9

Wang Bingheng, Liu Tingrui, Yang Fan, Zhang Huan, Li Wei, Ma Ping, Yang Ming. Research on Requirements and Methods for Intelligent Assessment of Simulation Credibility[J]. Journal of System Simulation, 2025, 37(7): 1710-1722.

Figures/Tables 12

Fig. 1

Fig. 2

Table 1

Fig. 3

Fig. 4

Fig. 5

Table 2

Fig. 6

Table 3

Evaluation results and weights of indicators

评估指标	可信度		权重
评估指标	可信度	STD	权重	STD
$I 1$	—	—	—	—
$I 2$	76.8	[3.5，0]	非常不重要	—
$I 3$	80.4	[5，1.5]	中等重要	—
$I 4$	[80，85]	[1，2.2]	76.5	[4，0.5]
$I 5$	缺乏可信	—	极其重要	—

Table 3

Table 4

Fig. 7

Table 5

References 27

[1]	王子才. 关于仿真理论的探讨[J]. 系统仿真学报, 2000, 12(6): 604-608.
	Wang Zicai. Research on Simulation Theory[J]. Journal of System Simulation, 2000, 12(6): 604-608.
[2]	李伟, 林圣琳, 周玉臣, 等. 复杂仿真系统可信度评估研究进展[J]. 中国科学(信息科学), 2018, 48(7): 767-782.
	Li Wei, Lin Shenglin, Zhou Yuchen, et al. Research Progress on Credibility Assessment of a Complex Simulation System[J]. Scientia Sinica(Informationis), 2018, 48(7): 767-782.
[3]	张伟, 王行仁. 仿真可信度[J]. 系统仿真学报, 2001, 13(3): 312-314.
	Zhang Wei, Wang Xingren. Simulation Credibility[J]. Journal of System Simulation, 2001, 13(3): 312-314.
[4]	Xu Yongjun, Liu Xin, Cao Xin, et al. Artificial Intelligence: A Powerful Paradigm for Scientific Research[J]. The Innovation, 2021, 2(4): 100179.
[5]	LeCun Y, Bengio Y, Hinton G. Deep Learning[J]. Nature, 2015, 521(7553): 436-444.
[6]	Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Generative Adversarial Nets[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2014: 2672-2680.
[7]	Xu Peng, Zhu Xiatian, Clifton D A. Multimodal Learning with Transformers: A Survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(10): 12113-12132.
[8]	Martens Jurgen, Pauwels Karl, Put Ferdi. A Neural Network Approach to the Validation of Simulation Models[C]//Proceedings of the 2006 Winter Simulation Conference. Piscataway: IEEE, 2006: 905-910.
[9]	尹爱军, 王昱, 戴宗贤, 等. 基于变分自编码器的轴承健康状态评估[J]. 振动、测试与诊断, 2020, 40(5): 1011-1016.
	Yin Aijun, Wang Yu, Dai Zongxian, et al. Evaluation Method of Bearing Health State Based on Variational Auto-encoder[J]. Journal of Vibration, Measurement & Diagnosis, 2020, 40(5): 1011-1016.
[10]	李宁. 基于强化学习的CBTC系统信息安全风险评估方法[J]. 电子设计工程, 2024, 32(6): 151-155.
	Li Ning. Cyber Security Risk Assessment Method of CBTC System Based on Reinforcement Learning[J]. Electronic Design Engineering, 2024, 32(6): 151-155.
[11]	Min Feiyan, Yang Ming, Wang Zicai. Knowledge-based Method for the Validation of Complex Simulation Models[J]. Simulation Modelling Practice and Theory, 2010, 18(5): 500-515.
[12]	Franz Georg Listl, Fischer Jan, Weyrich Michael. An Architecture for Knowledge Graph Based Simulation Support[C]//2023 IEEE 28th International Conference on Emerging Technologies and Factory Automation (ETFA). Piscataway: IEEE, 2023: 1-8.
[13]	Tang J. A Dynamic System Model Validation Scheme with Fuzzy Logic Techniques[C]//Proceedings of the 10th International Conference on Accelerator and Large Experimental Physics Control Systems. Geneva: CERN, 2005: 45-51.
[14]	Lin Shenglin, Li Wei, Ma Ping, et al. Structural Modelling and Bayesian Inference for Model Validation and Confidence Extrapolation[J]. Journal of Statistical Computation and Simulation, 2020, 90(2): 211-233.
[15]	Han Yongming, Cui Shiying, Geng Zhiqiang, et al. Food Quality and Safety Risk Assessment Using a Novel HMM Method Based on GRA[J]. Food Control, 2019, 105: 180-189.
[16]	Aigrain S, Foreman-Mackey D. Gaussian Process Regression for Astronomical Time Series[J]. Annual Review of Astronomy and Astrophysics, 2023, 61: 329-371.
[17]	鹿亮. 发动机轴承仿真系统可信度评估方法和系统实现[D]. 大连: 大连理工大学, 2013.
	Lu Liang. Credibility Evaluation Method and System Implementation for Simulation System of Engine Bearing[D]. Dalian: Dalian University of Technology, 2013.
[18]	Wu Yuanzhong. Optimization Design and Application of Training Evaluation Model for Power Enterprises Based on Ant Colony Algorithm[C]//2023 World Conference on Communication & Computing (WCONF). Piscataway: IEEE, 2023: 1-6.
[19]	Li Wei, Lu Lingyun, Xie Xiaotian, et al. A Novel Extension Algorithm for Optimized Latin Hypercube Sampling[J]. Journal of Statistical Computation and Simulation, 2017, 87(13): 2549-2559.
[20]	Song Jiao, Wei Li, Ming Yang. The Validation Method of Simulation Model Based on K-means Clustering and Fisher Discriminant Analysis[C]//Proceedings of 2013 International Conference on Virtual Reality and Visualization. Piscataway: IEEE, 2013: 313-316.
[21]	Teferra K, Shields M D, Hapij A, et al. Mapping Model Validation Metrics to Subject Matter Expert Scores for Model Adequacy Assessment[J]. Reliability Engineering & System Safety, 2014, 132: 9-19.
[22]	周玉臣, 方可, 马萍, 等. 基于集成学习的复杂仿真模型验证方法[J]. 系统工程与电子技术, 2018, 40(9): 2124-2130.
	Zhou Yuchen, Fang Ke, Ma Ping, et al. Complex Simulation Model Validation Method Based on Ensemble Learning[J]. Systems Engineering and Electronics, 2018, 40(9): 2124-2130.
[23]	聂凯, 栾瑞鹏. 基于数据增强的仿真模型验证方法[J]. 指挥控制与仿真, 2019, 41(3): 92-96.
	Nie Kai, Luan Ruipeng. Validation Method of Simulation Models Based on Data Augmentation[J]. Command Control & Simulation, 2019, 41(3): 92-96.
[24]	Song Jiao, Wei Li, Ming Yang. A Method for Simulation Model Validation Based on Theil's Inequality Coefficient and Principal Component Analysis[C]//AsiaSim 2013. Berlin: Springer Berlin Heidelberg, 2013: 126-135.
[25]	Li Yunlong, Wang Xiaojun, Wang Chong, et al. Non-probabilistic Bayesian Update Method for Model Validation[J]. Applied Mathematical Modelling, 2018, 58: 388-403.
[26]	王玉龙, 董志明, 彭文成, 等. 基于融合先验信息的仿真交互可信度评估方法[J]. 装甲兵工程学院学报, 2019, 33(3): 103-108.
	Wang Yulong, Dong Zhiming, Peng Wencheng, et al. A Simulation Interactive Credibility Evaluation Method Based on Fusion Prior Information[J]. Journal of Academy of Armored Force Engineering, 2019, 33(3): 103-108.
[27]	陆营波, 钱晓超, 陈伟, 等. 数据驱动的装备效能评估模型构建方法研究[J]. 系统仿真学报, 2018, 30(12): 4587-4594, 4601.
	Lu Yingbo, Qian Xiaochao, Chen Wei, et al. Research on Construction Method of Data-driven Equipment Effectiveness Evaluation Model[J]. Journal of System Simulation, 2018, 30(12): 4587-4594, 4601.

智能技术类型	可信度智能评估方法	解决的主要问题
机器学习	基于概率神经网络的仿真验证方法^[8]	多候选仿真模型验证问题
	基于聚类和判别分析的排序评估方法^[20]	多候选仿真模型验证问题
	基于PNN的指标-专家映射方法^[21]	仿真模型半自动充分性评估
	基于集成学习的复杂仿真模型验证方法^[22]	评估数据相似性度量结果与可信度等级映射关系
	基于生成对抗网络的数据增强方法^[23]	面向小样本评估数据
	基于主成分分析的特征提取方法^[24]	面向多元相关变量的高维特征
知识推理	基于知识的专家系统^[11]	考虑多种行为的复杂仿真模型的验证问题
	基于知识图谱的评估框架^[12]	数字孪生模型的集成与验证问题
	基于仿真模型预期行为知识的验证模糊知识库^[13]	模型行为智能验证问题
贝叶斯推理	基于贝叶斯推理的模型验证方法^[25]	考虑不确定性的模型验证问题
	基于贝叶斯网络的可信度外推方法^[14]	复杂仿真系统可信度推理问题
	基于融合先验信息的仿真交互可信度评估方法^[26]	仿真交互行为可信度实时评估问题
智能优化	基于自适应克里金法的结构可靠性评估方法^[27]	评估数据高效优化问题
	基于蚁群算法的评价模型优化设计方法^[18]	评估模型优化问题
	用于优化拉丁超立方采样的扩展算法^[19]	昂贵仿真模型高效采样问题
	基于遗传算法的区间数层次分析法^[17]	评估指标权重优化问题