A Framework on Equipment Digital Twin Credibility Assessment

doi:10.16182/j.issn1004731x.joss.23-0057

Abstract

Abstract:

A key bottleneck in the large-scale application of equipment digital twin is the lack of systematic and effective credibility assessment methods. This paper analyzes the dynamic evolution, virtual-real interactivity and other key features of the equipment digital twin. A credibility assessment framework for the equipment digital twin is proposed, including the credibility connotation of the digital twin, a multi-dimensional and multi-level credibility assessment index system and a credibility assessment methodology. The whole assessment process is illustrated with the robotic arm digital twin as an example, which can provide directional guidance for the assessment and construction of the digital twin.

Key words: equipment digital twin, credibility assessment, dynamic evolution, index system, modeling and simulation

CLC Number:

TP 391.9

Han Lu, Lin Zhang, Kunyu Wang, Zejun Huang, Hongbo Cheng, Jin Cui. A Framework on Equipment Digital Twin Credibility Assessment[J]. Journal of System Simulation, 2023, 35(7): 1455-1471.

Figures/Tables 17

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Table 1

Data indicators

考察对象	指标	含义
数据源	适配性 $Z s p S$	产生、采集、传递数据等符合性能要求
数据源	权威性 $Z q w S$	设备厂商、人员机构、软件等权威可靠
数据体	真实性 $Z z s S$	到达系统的数据与实际情况一致且及时有效
数据体	完备性 $Z w b S$	数据充分包含所需信息，且规范、取用便捷

Table 1

Table 2

Static indicators

考察对象	指标	含义
需模映射	规范性 $Z g f J$	采用规范的形式化方法分析表征需求与模型
需模映射	完整性 $Z w z J$	需求被系统完整地关联映射到了多维模型
单维属性	准确性 $Z z q 1 J$	图2各单维度机理、参数准确反映物理装备属性
单维属性	独立性 $Z d l J$	各单维模型及其内部属性、功能相对独立
多维耦合	准确性 $Z z q 2 J$	对耦合效应的描述清晰、简洁，与实际结果一致
多维耦合	有界性 $Z y j J$	耦合过程无环或有合理的时值边界约束

Table 2

Table 3

Dynamic indicators

考察对象	指标	含义
演化驱动	辨识性 $Z b s D$	物理系统发生变化时能够正确辨识模型应变部位
演化驱动	灵敏性 $Z l m D$	实现变化部位辨识所需的最小数据波动量
演化过程	时效性 $Z s x D$	演化复杂度与相应演化所耗时间的比值
演化过程	收敛性 $Z s l D$	虚实差异性在演化与不确定性影响下的收敛情况
演化结果	系统性 $Z x t D$	与驱动部位相关且影响需求的部位参与演化传递
演化结果	准确性 $Z z q D$	可测对象的多时间窗输出与实际输出的差异程度

Table 3

Table 4

Fig. 6

Fig. 7

Table 5

List of credibility assessment methods

一级目录	二级目录	三级目录	四级目录
软件评估F1	代码检查F₁₁	词法检查F₁₁₁
		语法检查F₁₁₂
		语义检查F₁₁₃
	算法分析F₁₂	可读性分析F₁₂₁
		健壮性分析F₁₂₂
		时间复杂度分析F₁₂₃
		空间复杂度分析F₁₂₄
		神经网络结构分析F₁₂₅
	执行测试F₁₃	执行监控F₁₃₁
		执行分析F₁₃₂
		边界测试F₁₃₃
		压力测试F₁₃₄
		执行定位F₁₃₅
		分离测试F₁₃₆
	数据集验证F₁₄	数据集划分法F₁₄₁	留出法F₁₄₁₁
			留一法F₁₄₁₂
			k折交叉验证F₁₄₁₃
		分类问题评估法F₁₄₂	准确率F₁₄₂₁
			精准率F₁₄₂₂
			召回率F₁₄₂₃
			F1分数F₁₄₂₄
			ROC曲线F₁₄₂₅
			AUC F₁₄₂₆
		回归问题评估法F₁₄₃	MAE F₁₄₃₁
			MSE F₁₄₃₂
			RMSE F₁₄₃₃
			MSLE F₁₄₃₄
			MAPE F₁₄₃₅
主观方法F₂	人工审核F₂₁
	图灵测试F₂₂
	专家打分F₂₃
	多方校验F₂₄
	经验参数对比F₂₅
半形式化方法F₃	一致性检查F₃₁
	图分析F₃₂	数据流图分析F₃₂₁
		时序图分析F₃₂₂
		连接图分析F₃₂₃
		状态机图分析F₃₂₄
		结构图分析F₃₂₅
		键合图分析F₃₂₆
		因果图分析F₃₂₇
形式化方法F₄	归纳推理F₄₁
	断言检查F₄₂
	证据理论F₄₃
	形式审查F₄₄
	逻辑推断F₄₅
结果比较F₅	灵敏度分析F₅₁
	概率统计F₅₂	假设性检验F₅₂₁	置信区间法F₅₂₁₁
			T检验F₅₂₁₂
			卡方检验F₅₂₁₃
			Hotelling's T2 tests F₅₂₁₄
		非假设性检验F₅₂₂	Theil's inequality F₅₂₂₁
			Multivariate analysis of variance F₅₂₂₂
			回归测试 F₅₂₂₃
	确定性数据分析F₅₃	时域数据分析F₅₃₁
		频域数据分析F₅₃₂
		时频域数据分析F₅₃₃
		机器学习F₅₃₄
	误差效应评估F₅₄
集成分析F₆	AHP类F₆₁
	贝叶斯网络F₆₂
	灰色关联度分析F₆₃
	复杂网络分析F₆₄
演化分析F₇	演化时效性F₇₁	控制系统实时性测试F₇₁₁
	演化时效性F₇₁	软件系统实时性测试F₇₁₂
	演化准确性F₇₂	时间域分析F₇₂₁
	演化准确性F₇₂	频域分析F₇₂₂
	演化平稳性F₇₃	李雅普诺夫稳定性理论F₇₃₁
	演化平稳性F₇₃	算法收敛性分析F₇₃₂
	演化系统性F₇₄
数据检查F₈	数据认证F₈₁	协议认证F₈₁₁
		机构认证F₈₁₂
		时效认证F₈₁₃
	数据校验F₈₂	逻辑关系校验F₈₂₁
		自身冗余信息校验F₈₂₂
		行业标准对照F₈₂₃
		数据价值校验F₈₂₄
		实验校验F₈₂₅
	数据源分析F₈₃	性能分析F₈₃₁
		环境分析F₈₃₂
		布局关联分析F₈₃₃

Table 5

Fig. 8

Fig. 9

Fig. 10

Fig. 11

Fig. 12

References 43

1	Zhang Lin, Zhou Longfei, Horn B K P. Building a Right Digital Twin with Model Engineering[J]. Journal of Manufacturing Systems, 2021, 59: 151-164.
2	Boschert S, Rosen R. Digital Twin-the Simulation Aspect[M]. Cham: Springer International Publishing, 2016: 59-74.
3	Bielefeldt B, Hochhalter J, Hartl D. Computationally Efficient Analysis of SMA Sensory Particles Embedded in Complex Aerostructures Using a Substructure Approach[C]//ASME 2015 Conference on Smart Materials, Adaptive Structures and Intelligent Systems. New York, USA: ASME, 2015: V001T02A007.
4	Liu Shimin, Bao Jinsong, Lu Yuqian, et al. Digital Twin Modeling Method Based on Biomimicry for Machining Aerospace Components[J]. Journal of Manufacturing Systems, 2021, 58, Part B: 180-195.
5	张霖. 关于数字孪生的冷思考及其背后的建模和仿真技术[J]. 系统仿真学报, 2020, 32(4): 1-10.
	Zhang Lin. Cold Thinking About the Digital Twin and the Modeling and Simulation Techniques Behind It[J]. Journal of System Simulation, 2020, 32(4): 1-10.
6	张霖, 陆涵. 从建模仿真看数字孪生[J]. 系统仿真学报, 2021, 33(5): 995-1007.
	Zhang Lin, Lu Han. Discussing Digital Twin From of Modeling and Simulation[J]. Journal of System Simulation, 2021, 33(5): 995-1007.
7	Tuegel E J. The Airframe Digital Twin: Some Challenges to Realization[C]//53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference. 20th AIAA/ASME/AHS Adaptive Structures Conference 14th AIAA, 2012: 1812.
8	Huang Jingwei, Gheorghe A, Handley H, et al. Towards Digital Engineering: the Advent of Digital Systems Engineering[J]. International Journal of System of Systems Engineering, 2020, 10(3): 234-261.
9	张霖, 王昆玉, 赖李媛君, 等. 基于建模仿真的体系工程[J]. 系统仿真学报, 2022, 34(2): 179-190.
	Zhang Lin, Wang Kunyu, Yuanjun Laili, et al. Modeling & Simulation Based System of Systems Engineering[J]. Journal of System Simulation, 2022, 34(2): 179-190.
10	Balci O. Verification Validation and Accreditation of Simulation Models[C]//Proceedings of the 29th Conference on Winter simulation. USA: IEEE Computer Society, 1997: 135-141.
11	Lehmann A. Verification and Validation (V&V) of Models and Simulations (M&S)-Past, Present and Future[R]. NATO-Lecture Series on Application of the GM-VV the Generic Methodology for Verification & Validation of Models, Simulations and Data, 2014.
12	Department of Defense, Untied States of America. Documentation of Verification, Validation, and Accreditation (VV&A) for Models and Simulations: MIL-S [S]. Washington: Department of Defense, Untied States of America, 2008.
13	IEEE. IEEE Recommended Practice for Verification, Validation, and Accreditation of a Federationan Overlay to the High Level Architecture Federation Development and Execution Process: 1516.4-2007 [S]. Piscataway, NJ, USA: IEEE, 2007.
14	SGGV. Generic Methodology for Verification and Validation (GM-VV), Volume 1.1, Volume 1.2, Volume 1.3 [S]. USA: Simulation Interoperability Standardization Organization (SISO), 2012-2013.
15	唐见兵. 作战仿真系统可信性研究[D]. 长沙: 国防科技大学, 2009.
	Tang Jianbing. Research on the Credibility of Warfare Simulation System[D]. Changsha: National University of Defense Technology, 2009.
16	廖瑛, 邓方林, 梁加红, 等. 系统建模与仿真的校核、验证与确认(VV&A)技术[M]. 长沙: 国防科技大学出版社, 2006.
	Liao Ying, Deng Fanglin, Liang Jiahong, et al. The Verification, Validation and Accreditation Technologies on System Modeling and Simulation[M]. Changsha: National University of Defense Technology Press, 2006.
17	王鹏. 虚实结合的武器装备试验方法的若干技术研究[D]. 长沙: 国防科技大学, 2018.
	Wang Peng. Research on Some Technologies of Virtual and Real Integrated Equipment Test[D]. Changsha: National University of Defense Technology, 2018.
18	Yuanjun Laili, Zhang Lin, Luo Yongliang. A Pattern-based Validation Method for the Credibility Evaluation of Simulation Models[J]. Simulation, 2020, 6(2): 151-167.
19	刘飞, 马萍, 杨明, 等. 复杂仿真系统可信度量化研究[J]. 哈尔滨工业大学学报, 2007, 39(1): 1-3.
	Liu Fei, Ma Ping, Yang Ming, et al. Research on Credibility Quantification of Complex Simulation Systems[J]. Journal of Harbin Institute of Technology, 2007, 39(1): 1-3.
20	李伟, 周玉臣, 林圣琳, 等. 仿真模型验证方法综述[J]. 系统仿真学报, 2019, 31(7): 1249-1256.
	Li Wei, Zhou Yuchen, Lin Shenglin, et al. Review of Simulation Model Validation Methods[J]. Journal of System Simulation, 2019, 31(7): 1249-1256.
21	Schruben L W. Establishing the Credibility of Simulations[J]. Simulation, 1980, 34(3): 101-105.
22	Beydoun G, Low G, Bogg P. Suitability Assessment Framework of Agent-based Software Architectures[J]. Information and Software Technology, 2013, 55(4): 673-689.
23	Buchmann C M, Grossmann K, Schwarz N. How Agent Heterogeneity, Model Structure and Input Data Determine the Performance of an Empirical ABM-a Real-world Case Study on Residential Mobility[J]. Environmental Modelling & Software, 2016, 75: 77-93.
24	Acar E. Effect of Error Metrics on Optimum Weight Factor Selection for Ensemble of Metamodels[J]. Expert Systems With Applications, 2015, 42(5): 2703-2709.
25	Huang Guangbin, Zhu Qinyu, Siew C K. Extreme Learning Machine: Theory and Applications[J]. Neurocomputing, 2006, 70(1/3): 489-501.
26	Li Ni, Dong Liwei, Zhao Luming, et al. A Credibility Evaluation Method for Complex Simulation Systems Based on Interactive Network Analysis[J]. Simulation Modelling Practice and Theory, 2021, 110: 102289.
27	Ferson S, Oberkampf W L. Validation of Imprecise Probability Models[J]. International Journal of Reliability and Safety, 2009, 3(1/3): 3-22.
28	Li Wei, Chen Wei, Jiang Zhen, et al. New Validation Metrics for Models with Multiple Correlated Responses[J]. Reliability Engineering & System Safety, 2014, 127: 1-11.
29	Chan W K V. Interaction Metric of Emergent Behaviors in Agent-based Simulation[C]//Proceedings of the 2011 Winter Simulation Conference (WSC). Piscataway, NJ, USA: IEEE, 2011: 357-368.
30	Dornheim H, Brazauskas V. Robust-efficient Credibility Models with Heavy-tailed Claims: A Mixed Linear Models Perspective[J]. Insurance: Mathematics and Economics, 2011, 48(1): 72-84.
31	Liang Jiaojiao, Bai Yuebin, Bi Chongguang, et al. Adaptive Routing Based on Bayesian Network and Fuzzy Decision Algorithm in Delay-tolerant Network[C]//2013 IEEE 10th International Conference on High Performance Computing and Communications & 2013 IEEE International Conference on Embedded and Ubiquitous Computing. Piscataway, NJ, USA: IEEE, 2013: 690-697.
32	Dezfuli H, Kelly D, Smith C, et al. Bayesian Inference for NASA Probabilistic Risk and Reliability Analysis[EB/OL]. (2009-06-01)[2022-10-29]. .
33	张淑丽, 叶满昌. 导弹武器系统仿真可信度评估方法研究[J]. 计算机仿真, 2006, 23(5): 48-52.
	Zhang Shuli, Ye Manchang. Simulation Credibility Evaluation Method for Missile Weapon System[J]. Computer Simulation, 2006, 23(5): 48-52.
34	林圣琳. 面向复杂仿真的评估与优化方法研究[D]. 哈尔滨: 哈尔滨工业大学, 2020.
	Lin Shenglin. Complex Simulation-oriented Assessment and Optimization Methods[D]. Harbin: Harbin Institute of Technology, 2020.
35	刘翠翠. 建模与仿真的VV&A方法研究[D]. 哈尔滨: 哈尔滨工程大学, 2012.
	Liu Cuicui. Research on VV&A Methods of the Modeling and Simulation[D]. Harbin: Harbin Engineering University, 2012.
36	Ramin H, Krishnan E N, Annadurai G, et al. Transient Sensor Errors and Their Impact on Fixed-bed Regenerator (FBR) Testing Standards[J]. Science and Technology for the Built Environment, 2021, 27(5): 656-678.
37	Ren Yongjun, Qi Jian, Liu Yepeng, et al. Integrity Verification Mechanism of Sensor Data Based on Bilinear Map Accumulator[J]. ACM Transactions on Internet Technology, 2021, 21(1): 5.
38	Townend S, Venters P, Lau C C, et al. Trusted Digital Spaces Through Timely Reliable and Personalized Provenance[C]//15th IEEE International Symposium on Object/Component/Service-oriented Real-time Distributed Computing Workshops, 2012: 136-141.
39	汪洁. THz波大气传输衰减特性研究[D]. 南京: 南京信息工程大学, 2017.
	Wang Jie. The Research on Atmospheric Transmission and Attenuation Characteristics of THz Wave[D]. Nanjing: Nanjing University of Information Science & Technology, 2017.
40	文静, 袁家斌, 李宁伟. 面向大数据节点的动态可信度评估模型[J]. 江苏科技大学学报(自然科学版), 2019, 33(6): 75-81.
	Wen Jing, Yuan Jiabin, Li Ningwei. Dynamic Credibility Evaluation Model Based on Big Data Nodes[J]. Journal of Jiangsu University of Science and Technology(Natural Science Edition), 2019, 33(6): 75-81.
41	李嘉, 洪珂一, 熊琭. 关于网络服务数据传输可信度评估仿真[J]. 计算机仿真, 2018, 35(2): 330-333, 401.
	Li Jia, Hong Keyi, Xiong Lu. Simulation of Reliability Evaluation of Network Service Data Transmission[J]. Computer Simulation, 2018, 35(2): 330-333, 401.
42	杜文振, 陈海明, 李栋, 等. 基于综合链路质量评估的传感器网络重传机制研究[J]. 高技术通讯, 2016, 26(8): 750-760.
	Du Wenzhen, Chen Haiming, Li Dong, et al. Research on WSNs' Retransmission Strategy Based on Comprehensive Assessment of Link Quality[J]. Chinese High Technology Letters, 2016, 26(8): 750-760.
43	Zhang Lin, Liu Ying, Yuanjun Laili, et al. Model Maturity Towards Modeling and Simulation: Concepts, Index System Framework and Evaluation Method[J]. International Journal of Modeling, Simulation, and Scientific Computing, 2020, 11(3): 2040001.

考察对象	含义
能力成熟度	包含团队协作性、管理规范性、团队权威性等
模型成熟度	包含可重用性、安全性、健壮性、可持续性等

[1]	Mingyuan Liu, Jiaxiang Xie, Hao Wu, Jianlin Fu, Guofu Ding. Research on Workshop Logic Modeling and Simulation Based on Finite State Machine [J]. Journal of System Simulation, 2023, 35(4): 853-861.
[2]	Fan Yang, Ping Ma, Wei Li, Ming Yang. Research on Digital Twin Credibility Assessment Process and Index [J]. Journal of System Simulation, 2023, 35(2): 350-358.
[3]	Guowei Lu, Xueqiang Tao, Deguang Duan, Hao Li, Zerui Zhang, En Chen. Research on Modeling and Simulation of Application Efficiency of Tactical Medical Equipment [J]. Journal of System Simulation, 2023, 35(1): 190-201.
[4]	Wei Li, Huan Zhang, Ping Ma, Ming Yang. Research on Credibility Assessment of Cloud Simulation System [J]. Journal of System Simulation, 2022, 34(4): 679-687.
[5]	Guoqiang Shi, Zewei Liu, Tingyu Lin, Zhao Xu, Xingyi Yang, Liqin Guo, Zhengxuan Jia. Open Cloud Architecture Design for Complex Product Modeling and Simulation System [J]. Journal of System Simulation, 2022, 34(3): 442-451.
[6]	Feng Li, Ying Wei. Product Decisions in Presence of Social Learning and Reference Point Effect [J]. Journal of System Simulation, 2022, 34(2): 234-246.
[7]	Lianyi Zhang, Xisheng Shen, Duzheng Qing, Han Zhang, Min Zhou, Xifu Wang. Research on Complex Combat Network Dynamic Evolution Based on Information Entropy [J]. Journal of System Simulation, 2022, 34(12): 2605-2618.
[8]	Zhiguang Wang, Baiting Liu, Xiaolei Wang, Tao Liu, Zhaowei Yang. Reliability Parameter Optimization Complex System Simulation Based on R-VIKOR Method [J]. Journal of System Simulation, 2022, 34(12): 2670-2679.
[9]	Gongguo Xu, Libing Cai, Peibing Du, Yu Liu. Research on Modeling and Simulation of Optimization Deployment for Cooperative Localization by Multiple Detection Sensors in Complex Environment [J]. Journal of System Simulation, 2022, 34(10): 2171-2180.
[10]	Fan Changjia, Du Yanqiu, Liang Di, Hu Kai, Huang Jiayan. Modeling and Simulation of Emergency Medical Resources Allocation in Shanghai during COVID-19 [J]. Journal of System Simulation, 2022, 34(1): 93-103.
[11]	Si Guangya, Wang Yanzheng. Challenges and Reflection on Next-generation Large-scale Computer Wargame System [J]. Journal of System Simulation, 2021, 33(9): 2010-2016.
[12]	Cao Qi, Xiang Qun, Wang Wenzheng. Research on Component-based Modeling of Simulation Entity for Logistics and Equipment Support [J]. Journal of System Simulation, 2021, 33(6): 1233-1240.
[13]	Li Wenxiang, Li Ye, Dong Jieshuang, Li Yiming. Development Paths of New Energy Vehicles Incorporating CO₂ Emissions Trading Scheme [J]. Journal of System Simulation, 2021, 33(6): 1451-1465.
[14]	Zheng Lingsha, Jiang Bing, Zhao Zhe, Yang Zhaoxu. Research on Time Performance Simulation and Analysis Technology of Aviation Complex Embedded System [J]. Journal of System Simulation, 2021, 33(6): 1489-1499.
[15]	Zhang Lin, Lu Han. Discussing Digital Twin from of Modeling and Simulation [J]. Journal of System Simulation, 2021, 33(5): 995-1007.