Review of Simulation Model Validation Methods

doi:10.16182/j.issn1004731x.joss.19-0250

Abstract

Abstract: Modeling and simulation has become an important way to cognize and research objective world, and the credibility of simulation models is the premise and basics for carrying out simulation activities. The ever-increasing complexity of the simulation objects and demands of simulation applications for users lead to a great challenge in validation of complex simulation model, which characterize a variety of complex features. This paper aims to summarize the concepts and development of simulation model validation, classify existing validation methods, analysis the applicable conditions of these methods and provide a brief review of some new validation methods. Furthermore, some emerging issues in the simulation model validation are analyzed to shed some light on the future research directions.

Key words: modeling and simulation, complex simulation, model validation, simulation result validation

CLC Number:

TP391.9

Li Wei, Zhou Yuchen, Lin Shenglin, Ma Ping, Yang Ming. Review of Simulation Model Validation Methods[J]. Journal of System Simulation, 2019, 31(7): 1249-1256.

References

[1] 李伟, 林圣琳, 周玉臣, 等. 复杂仿真系统可信度评估研究进展[J]. 中国科学: 信息科学, 2018, 48(7): 767-782.
Li Wei, Lin Shenglin, Zhou Yuchen, et al.Research progress on credibility assessment of a complex simulation system[J]. SCIENTIA SINICA Informationis, 2018, 48(7): 767-782.
[2] Robinson S.Simulation: the practice of model development and use[M]. England: Wiley, 2014.
[3] Sargent R G.Verification and validation of simulation models[J]. Journal of Simulation (S1747-7778), 2013, 7(1): 12-24.
[4] 王子才, 张冰, 杨明. 仿真系统的校核、验证与验收(VV&A): 现状与未来[J]. 系统仿真学报, 1999, 11(5): 321-325.
Wang Zicai, Zhang Bing, Yang Ming.Verification, validation and accreditation (VV&A) for simulation system: current status and future[J]. Journal of System Simulation, 1999, 11(5): 321-326.
[5] Conway R W, Johnson B M, Maxwell W L.Some problems of digital systems simulation[J]. Management Science (S0025-1909), 1959, 6(1): 92-110.
[6] Fishman G S, Kiviat P J.The analysis of simulation generated time series[J]. Management Science (S0025-1909), 1967, 3(7): 525-557.
[7] Schruben L W.Establishing the credibility of simulations[J]. Simulation (S0037-5497), 1980, 34(3): 101-105.
[8] Teorey T J.Validation criteria for computer system simulations[C]. Proceedings of the 3rd Symposium on Simulation of Computer Systems. Boulder, USA: ACM, 1975: 161-160.
[9] Balci O, Sargent R G.Some examples of simulation model validation using hypothesis testing[C]. Proceedings of the 1982 Winter Simulation Conference. San Diego, USA: IEEE, 1982: 621-629.
[10] Kheir N A, Holmes W M.On validating simulation models of missile systems[J]. Simulation (S0037-5497), 1978, 30(4): 117-128.
[11] Montgonmery D C, Conard R G.Comparison of simulation and flight test data for missile systems[J]. Simulation (S0037-5497), 1980, 34(2): 63-72.
[12] Balci O.Validation, verification, and testing techniques through the life cycle of a simulation study[J]. Annals of Operations Research (S0254-5330), 1994, 53: 121-173.
[13] Li W, Chen W, Jiang Z, et al.New validation metrics for models with multiple correlated responses[J]. Reliability Engineering & System Safety (S0951-8320), 2014, 127(6): 1-11.
[14] Roy C J, Oberkampf W L.A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing[J]. Computer Methods in Applied Mechanics and Engineering (S0045-7825), 2011, 200(25/28): 2131-2144.
[15] Jiang X, Mahadevan S, Urbina A.Bayesian nonlinear structural equation modeling for hierarchical validation of dynamical systems[J]. Mechanical Systems and Signal Processing (S0888-3270), 2010, 24(4): 957-975.
[16] Wu D, Lu Z, Wang Y, et al.Model validation and calibration based on component functions of model output[J]. Reliability Engineering & System Safety (S0951-8320), 2015, 140: 59-70.
[17] Claudia S, Teo Y M.On validation of semantic composability in data-driven simulation[C]. 2010 IEEE Workshop on Principles of Advanced and Distributed Simulation. Atlanta, USA: IEEE, 2010: 1-8.
[18] Lizcano D, Soriano J, Lopez G, et al.Automatic verification and validation wizard in web-centred end-user software engineering[J]. Journal of Systems and Software (S0164-1212), 2017, 125: 47-67.
[19] 查亚兵. 仿真结果验证的频谱比较方法[J]. 系统仿真学报, 1997, 9(1): 22-26.
Zha Yabing.Frequency Spectrum Comparison Method for Simulation Result Validation[J]. Journal of System Simulation, 1997, 9(1): 22-26.
[20] 吴静, 吴晓燕, 陈永兴, 等. 基于改进灰色关联分析的仿真模型验证方法[J]. 系统工程与电子技术, 2010, 32(8): 1677-1679.
Wu Jing, Wu Xiaoyan, Chen Yongxing, et al.Validation of simulation models based on improved grey relational analysis[J]. Systems Engineering and Electronics, 2010, 32(8): 1677-1679.
[21] 魏华梁, 李钟武. 灰色关联分析及其在导弹系统仿真模型验证中的应用[J]. 系统工程与电子技术. 1997(2): 55-61.
Wei Hualiang, Li Zhongwu.Grey relational analysis and its application to the validation of computet simulation models for missile systems[J]. Systems Engineering and Electronic, 1997(2): 55-61.
[22] 傅惠民. 仿真结果统计检验方法[J]. 机械强度, 2005, 27(5): 598-603.
Fu Huimin.Statistical test method for simulation results[J]. Journal of Mechanical Strength, 2005, 27(5): 598-603.
[23] 杨明, 钱晓超, 李伟, 等. 基于数据特征的仿真动态输出验证方法[J]. 系统工程与电子技术, 2016, 38(2): 457-463.
Yang Ming, Qian Xiaochao, Li Wei, et al.Simulation dynamic output validation method based on the data feature[J]. Systems Engineering and Electronics, 2016, 38(2): 457-463.
[24] 詹振飞, 胡洁, 符雁, 等. 虚拟样机环境下不确定性多元动态系统模型验证[J]. 机械工程学报, 2012, 48(5): 138-146.
Zhan Zhenfei, Hu Jie, Fu Yan, et al.Bayesian based model validation method for uncertain multivariate dynamic systems under virtual prototype environment[J]. Journal of Mechanical Engineering, 2012, 48(5): 138-146.
[25] Ferson S, Oberkampf W L, Ginzburg L.Model validation and predictive capability for the thermal challenge problem[J]. Computer Methods in Applied Mechanics and Engineering (S0045-7825), 2008, 197(29/32): 2408-2430.
[26] Dorobantu A, Balas G J, Georgiou T T.Validating aircraft models in the gap metric[J]. Journal of Aircraft (S0021-8669), 2014, 51(6): 1665-1672.
[27] 宁小磊, 吴颖霞, 于天朋, 等. 基于改进灰色关联分析的仿真模型综合验证方法[J]. 兵工学报, 2016, 37(2), 338-347.
Ning Xiaolei, Wu Yingxia, Yu Tianpeng, et al.Research on comprehensive validation of simulation models based on improved grey relational analysis[J]. Acta Armamentarii, 2016, 37(2): 338-347.
[28] Notton G, Paoli C, Ivanova L, et al.Neural network approach to estimate 10-min solar global irradiation values on tilted planes[J]. Renewable Energy (S0960-1481), 2013, 50: 576-584.
[29] Song K S, Li L, Tedesco L P, et al.Remote estimation of chlorophylla in turbid inland waters: three-band model versus GA-PLS model[J]. Remote Sensing of Environment (S0034-4257), 2013, 136: 342-357.
[30] Consonni V, Ballabio D, Todeschini R.Evaluation of model predictive ability by external validation techniques[J]. Journal of Chemometrics (S0886-9383), 2010, 24: 194-201.
[31] Roy K, Das R N, Ambure P, et al.Be aware of error measures: further studies on validation of predictive QSAR models[J]. Chemometrics and Intelligent Laboratory Systems (S0169-7439), 2016, 152: 18-33.
[32] 刘鹏, 陈长兴, 白云. 使用窗谱估计法验证仿真系统模型[J]. 空军工程大学学报, 2000, 1(2): 69-71.
Liu Peng, Chen Changxing, Bai Yun.Verifying simulation model using window spectrum estimation[J]. Journal of Air Force Engineering University, 2000, 1(2): 69-71.
[33] 刘文钊, 洪丽娜, 戚宗锋. 基于谱估计的雷达信号模型验证方法[J]. 微计算机信息, 2012, 28(5): 161-163.
Liu Wenzhao, Hong Lina, Qi Zongfeng.Model validation method of radar signal model based on spectrum estimation[J]. Microcomputer Information, 2012, 28(5): 161-163.
[34] Min F Y, Yang M, Wang Z C.Knowledge-based method for the validation of complex simulation models[J]. Simulation Modelling Practice and Theory (S1569-190X), 2010, 18(5): 500-515.
[35] 焦松, 李伟, 杨明. 基于三角模糊数的仿真模型验证方法[J]. 中南大学学报(自然科学版), 2014, 45(1): 124-131.
Jiao Song, Li Wei, Yang Ming.Validation method of simulation model based on triangular fuzzy number[J]. Journal of Central South University (Science and Technology), 2014, 45(1): 124-131.
[36] 周玉臣, 方可, 马萍, 等. 基于集成学习的复杂仿真模型验证方法[J]. 系统工程与电子技术, 2018, 40(9): 2124-2130.
Zhou Yuchen, Fang Ke, Ma Ping, et al.Complex simulation model validation method based on ensemble learning[J]. Systems Engineering and Electronic, 2018, 40(9): 2124-2130.
[37] 林圣琳, 李伟, 杨明, 等.考虑相关性的多元输出仿真模型验证方法[J/OL]. 自动化学报 [2019-06-10].https://doi.org/10.16383/j.aas.c180456.
Lin Shenglin, Li Wei, Yang Ming, et al.Multivariate validation method for simulation model under correlation[J/OL]. Acta Automatica Sinica, [2019-06-10].https: //doi.org/10.16383/j.aas.c180456.
[38] Zhou Y, Fang K, Ma P, et al.Simulation credibility evaluation based on multi-source data fusion[C]. 18th Asia Simulation Conference, Communications in Computer and Information Science. Singapore: Springer, 2018, 946: 18-31.