Digital Testing and Evaluation: Current Status, Challenges, and Prospects

doi:10.16182/j.issn1004731x.joss.25-0347

Abstract

Abstract:

Digital testing and evaluation (DTE) represents a novel paradigm in the evolution of testing and evaluation methodologies within the digital era. It is achieved through the integration of multiple digital theories and technologies to conduct testing and evaluations in the digital domain. This paper analyzed the characteristics of test objects across different historical periods, reviewed the core features of testing and evaluation techniques in each stage, and unveiled the paradigm shifts within the testing and evaluation technology system. Building upon this foundation, it explored the new demands placed on testing by test objects in the information age, clarifying the essence of DTE in the current context. Furthermore, focusing on the "design-modeling-implementation-application" full-process technical system of DTE, the paper identified crucial directions for technological breakthroughs and highlighted the challenges faced in the transition from "physical testing and evaluation" to "intelligent testing and evaluation." It outlined prospects for its development in terms of theory, talent, standards, and industry applications.

Key words: testing and evaluation, digital testing and evaluation, test object, test design, test data, artificial intelligence

CLC Number:

TP206+.1

Sun Bo, Zheng Kai. Digital Testing and Evaluation: Current Status, Challenges, and Prospects[J]. Journal of System Simulation, 2025, 37(8): 1885-1906.

Figures/Tables 11

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Fig. 9

Fig. 10

Fig. 11

References 69

[1]	佚名. 中共中央国务院印发《数字中国建设整体布局规划》[N]. 人民日报, 2023-02-28(001).
[2]	陶飞, 高鹏飞, 张辰源, 等. 数字试验测试验证：理论、关键技术及应用探索[J]. 机械工程学报, 2024, 60(15): 227-254.
	Tao Fei, Gao Pengfei, Zhang Chenyuan, et al. D-ETV: Digital Experiment, Testing and Verification[J]. Journal of Mechanical Engineering, 2024, 60(15): 227-254.
[3]	Stein S K. Archimedes: What Did He Do Besides Cry Eureka?[M]. Washington: Mathematical Association of America, 1999.
[4]	刘安. 淮南子译注[M]. 陈广忠, 译. 上海: 上海古籍出版社, 2016.
[5]	孙雅芬, 于孟晨, 贺菊玲, 等. 武经总要注[M]. 西安: 西安出版社, 2017.
[6]	沈括. 梦溪笔谈全译[M]. 金良年, 胡小静, 译. 上海: 上海古籍出版社, 2013.
[7]	詹姆斯·E·麦克莱伦第三, 哈罗德·多恩. 世界科学技术通史[M]. 3版. 王鸣阳, 陈多雨, 译. 上海: 上海科技教育出版社, 2020.
[8]	罗杰·奥斯本. 钢铁、蒸汽与资本: 工业革命的起源[M]. 曹磊, 译. 北京: 电子工业出版社, 2016.
	Iron Osborne L., Steam & Money : The Making of the Industrial Revolution[M]. Translated by Cao Lei. Beijing: Publishing House of Electronics Industry, 2016.
[9]	沈乃澂. 国际单位制的形成、建立和发展[J]. 物理, 1991, 20(3): 183-188.
[10]	阿什顿 T S. 工业革命[M]. 李冠杰, 译. 上海: 上海人民出版社, 2020.
	Ashton T S. The Industrial Revolution[M]. Translated by Li Guanjie. Shanghai: Shanghai People's Publishing House, 2020.
[11]	王元庆. 文明的融合: 近代科技简史[M]. 北京: 清华大学出版社, 2022.
[12]	蔡立志. 软件测试导论[M]. 北京: 清华大学出版社, 2016.
[13]	李彪, 王巍. 智能网联汽车建模与仿真技术[M]. 北京: 机械工业出版社, 2023.
[14]	张钹, 朱军, 苏航. 迈向第三代人工智能[J]. 中国科学(信息科学), 2020, 50(9): 1281-1302.
	Zhang Bo, Zhu Jun, Su Hang. Toward the Third Generation of Artificial Intelligence[J]. Scientia Sinica(Informationis), 2020, 50(9): 1281-1302.
[15]	张震, 崔志兴, 庞冲, 等. 无人装备技术发展研究[C]//中国航天电子技术研究院科学技术委员会2020年学术年会论文集. 北京: 中国航天电子技术研究院科学技术委员会, 2020: 802-807.
[16]	普杰, 沈泽华, 吴立金, 等. 无人装备仿真试验与能力评估技术研究[J]. 中国电子科学研究院学报, 2023, 18(1): 7-11, 20.
	Pu Jie, Shen Zehua, Wu Lijin, et al. Research on Simulation Test and Capability Evaluation Technology of Unmanned Equipment[J]. Journal of China Academy of Electronics and Information Technology, 2023, 18(1): 7-11, 20.
[17]	丁洪, 吴奇. 极端条件下的重大科学问题与极端条件设施发展现状及展望[J]. 中国科学(物理学力学天文学), 2014, 44(10): 1108-1115.
	Ding Hong, Wu Qi. Major Scientific Problems Under Extreme Conditions and Development of Synergetic Extreme Condition Facilities[J]. Scientia Sinica(Physica, Mechanica & Astronomica), 2014, 44(10): 1108-1115.
[18]	宋明涛. 考虑设施耐久及能耗的卡车编队优化方法研究[D]. 上海: 同济大学, 2022.
	Song Mingtao. Study on the Optimization of Truck Platoon Formation Considering Infrastructure Durability and Energy Consumption[D]. Shanghai: Tongji University, 2022.
[19]	伍毅平, 赵子龙, 倪鹏. 智能网联卡车编队领航车驾驶人驾驶能力需求综述[J]. 中国公路学报, 2024, 37(2): 219-238.
	Wu Yiping, Zhao Zilong, Ni Peng. Driving Capability Requirements for Leading Vehicle Driver of Intelligent Connected Truck Platoon: Review[J]. China Journal of Highway and Transport, 2024, 37(2): 219-238.
[20]	赵林, 张宇飞, 姚明旿, 等. 无人机集群协同技术发展与展望[J]. 无线电工程, 2021, 51(8): 823-828.
	Zhao Lin, Zhang Yufei, Yao Mingwu, et al. Development and Trend of UAV Swarm Cooperative Techniques[J]. Radio Engineering, 2021, 51(8): 823-828.
[21]	张家奎, 李晓东, 周河宇, 等. 俄乌冲突中无人艇作战运用的分析研究[J]. 数字海洋与水下攻防, 2024, 7(6): 616-622.
	Zhang Jiakui, Li Xiaodong, Zhou Heyu, et al. Analysis and Research on Operational Application of Unmanned Surface Vehicles in Russia-Ukraine Conflict[J]. Digital Ocean & Underwater Warfare, 2024, 7(6): 616-622.
[22]	罗鹏程, 傅攀峰, 周经伦. 武器装备体系作战能力评估框架[J]. 系统工程与电子技术, 2005, 27(1): 72-75.
	Luo Pengcheng, Fu Panfeng, Zhou Jinglun. Framework to Evaluate the Combat Capability of Weapons SoS[J]. Systems Engineering and Electronics, 2005, 27(1): 72-75.
[23]	杨克巍, 杨志伟, 谭跃进, 等. 面向体系贡献率的装备体系评估方法研究综述[J]. 系统工程与电子技术, 2019, 41(2): 311-321.
	Yang Kewei, Yang Zhiwei, Tan Yuejin, et al. Review of the Evaluation Methods of Equipment System of Systems Facing the Contribution Rate[J]. Systems Engineering and Electronics, 2019, 41(2): 311-321.
[24]	唐胜景, 史松伟, 张尧, 等. 智能化分布式协同作战体系发展综述[J]. 空天防御, 2019, 2(1): 6-13.
	Tang Shengjing, Shi Songwei, Zhang Yao, et al. Review on the Development of Intelligence-based Distributed Cooperative Operational System[J]. Air & Space Defense, 2019, 2(1): 6-13.
[25]	陈虎, 梁云芳, 高雷, 等. 国外舰船流场测试及可视化技术研究[J]. 舰船科学技术, 2016, 38(11): 1-7.
	Chen Hu, Liang Yunfang, Gao Lei, et al. Research Development on Ship Flow Measurement and Visualization[J]. Ship Science and Technology, 2016, 38(11): 1-7.
[26]	赵春慧, 李国强. 船舶耐波性拖曳水池试验方法研究[J]. 船舶物资与市场, 2023, 31(12): 26-28.
[27]	朱德祥, 沈泓萃, 洪方文, 等. 船模数值水池框架及其研究基础[J]. 水动力学研究与进展A辑, 2008, 23(1): 24-32.
	Zhu Dexiang, Shen Hongcui, Hong Fangwen, et al. The Framework of Ship Model Numerical Towing Tank and Research Fundament in China[J]. Journal of Hydrodynamics, 2008, 23(1): 24-32.
[28]	朱仁传, 缪国平, 向红贵, 等. 数值水池及其在船舶与海洋工程中的应用[J]. 上海造船, 2007(4): 21-23.
	Zhu Renchuan, Miao Guoping, Xiang Honggui, et al. Numerical Tank and Applications in Ship and Marine Hydrodynamics[J]. Shanghai Shipbuilding, 2007(4): 21-23.
[29]	闫喜强, 李宁波, 符澄, 等. 基于MBSE的风洞数字化研制模式构建及实现[J/OL]. 实验流体力学. (2025-03-27) [2025-06-07]. .
	Yan Xiqiang, Li Ningbo, Fu Cheng, et al. Construction and Implementation of Digital Wind Tunnel Developing Mode Based on MBSE[J/OL]. Journal of Experiments in Fluid Mechanics. (2025-03-27) [2025-06-07]. .
[30]	王翔. 虚实相生:空间站的数字化研制[J]. 航天器工程, 2024, 33(增1): 66-83.
	Wang Xiang. Mutual Promotion Between Virtualization and Reality: Digital Design and Development of Space Station[J]. Spacecraft Engineering, 2024, 33(S1): 66-83.
[31]	王国庆, 熊焕, 侯俊杰. 数字时代的航天系统工程[J]. 机械工程学报, 2024, 60(14): 206-214.
	Wang Guoqing, Xiong Huan, Hou Junjie. China Aerospace Systems Engineering Method in the Digital Age[J]. Journal of Mechanical Engineering, 2024, 60(14): 206-214.
[32]	方开泰. 均匀试验设计的理论、方法和应用-历史回顾[J]. 数理统计与管理, 2004, 23(3): 69-80.
[33]	刘瑞江, 张业旺, 闻崇炜, 等. 正交试验设计和分析方法研究[J]. 实验技术与管理, 2010, 27(9): 52-55.
	Liu Ruijiang, Zhang Yewang, Wen Chongwei, et al. Study on the Design and Analysis Methods of Orthogonal Experiment[J]. Experimental Technology and Management, 2010, 27(9): 52-55.
[34]	王聪聪, 柴倩云, 田夏, 等. 随机交叉试验方法及其在中医药临床试验中的实施[J]. 中医杂志, 2016, 57(13): 1116-1120.
	Wang Congcong, Chai Qianyun, Tian Xia, et al. Method and Implement Points of Random Cross-over Design in the Chinese Medicine Clinical Trial[J]. Journal of Traditional Chinese Medicine, 2016, 57(13): 1116-1120.
[35]	李夏明, 张健. 非重复析因试验数据分析的两个方法[J]. 数理统计与管理, 2002, 21(3): 40-47.
	Li Xiaming, Zhang Jian. Two Methods in Analyzing Data from Unreplicated Factorial Experiments[J]. Journal of Applied Statistics and Management, 2002, 21(3): 40-47.
[36]	Feng Shuo, Sun Haowei, Yan Xintao, et al. Dense Reinforcement Learning for Safety Validation of Autonomous Vehicles[J]. Nature, 2023, 615(7953): 620-627.
[37]	闫斌, 鄢挺, 杨林, 等. 混合动力车辆硬件在环测试系统研究[J]. 系统仿真学报, 2017, 29(6): 1237-1243, 1252.
	Yan Bin, Yan Ting, Yang Lin, et al. Development of Hybrid Electric Vehicle Hardware-in-the-loop Testing System[J]. Journal of System Simulation, 2017, 29(6): 1237-1243, 1252.
[38]	牛凯, 李静如, 李旭晨, 等. 电化学测试技术在锂离子电池中的应用研究[J]. 中国测试, 2020, 46(7): 90-101.
	Niu Kai, Li Jingru, Li Xuchen, et al. Research on the Applications of Electrochemical Measurement Technologies in Lithium-ion Batteries[J]. China Measurement & Testing Technology, 2020, 46(7): 90-101.
[39]	宋振宇, 周鑫. 航空发动机测试技术发展及展望[J]. 航空发动机, 2025, 51(1): 1-11.
	Song Zhenyu, Zhou Xin. Development and Prospect of Aeroengine Testing Technology[J]. Aeroengine, 2025, 51(1): 1-11.
[40]	罗勇, 尹林林, 冯黎明, 等. CFD技术在风资源评估中的应用综述与未来展望[J]. 机电产品开发与创新, 2025, 38(2): 167-170, 174.
	Luo Yong, Yin Linlin, Feng Liming, et al. A Review of CFD Technology in Wind Resource Assessment and Future Prospects[J]. Development & Innovation of Machinery & Electrical Products, 2025, 38(2): 167-170, 174.
[41]	赵峰, 李胜忠, 杨磊, 等. 基于CFD的船型优化设计研究进展综述[J]. 船舶力学, 2010, 14(7): 812-821.
	Zhao Feng, Li Shengzhong, Yang Lei, et al. An Overview on the Design Optimization of Ship Hull Based on CFD Techniques[J]. Journal of Ship Mechanics, 2010, 14(7): 812-821.
[42]	Raissi M, Perdikaris P, Karniadakis G E. Physics-informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations[J]. Journal of Computational Physics, 2019, 378: 686-707.
[43]	Vinyals O, Babuschkin I, Czarnecki W M, et al. Grandmaster Level in StarCraft II Using Multi-agent Reinforcement Learning[J]. Nature, 2019, 575(7782): 350-354.
[44]	Tang Zhentao, Shao Kun, Zhu Yuanheng, et al. A Review of Computational Intelligence for StarCraft AI[C]//2018 IEEE Symposium Series on Computational Intelligence (SSCI). Piscataway: IEEE, 2018: 1167-1173.
[45]	邓国强, 周早生, 杨秀敏. 爆炸冲击效应数值仿真中的几项关键技术[J]. 系统仿真学报, 2005, 17(5): 1059-1062, 1067.
	Deng Guoqiang, Zhou Zaosheng, Yang Xiumin. Key Technologies of Numerical Simulation on Explosion and Impact Effect[J]. Journal of System Simulation, 2005, 17(5): 1059-1062, 1067.
[46]	李淑君, 王惠泉, 赵文玉, 等. 基于COMSOL多物理场耦合仿真建模方法研究[J]. 机械工程与自动化, 2014(4): 19-20, 23.
	Li Shujun, Wang Huiquan, Zhao Wenyu, et al. Multiphysics Coupling Simulation Modeling Methods Based on COMSOL[J]. Mechanical Engineering & Automation, 2014(4): 19-20, 23.
[47]	杨立杰, 赵楠, 潘彦鹏, 等. 面向装备运用的云边端大数据应用架构设计[J]. 网络安全与数据治理, 2023, 42(增2): 150-155.
	Yang Lijie, Zhao Nan, Pan Yanpeng, et al. Big Data Application Architecture of Cloud-edge-client Design for Equipment Application[J]. Cyber Security and Data Governance, 2023, 42(S2): 150-155.
[48]	Fitzgerald J, Peter Gorm Larsen, Pierce K. Multi-modelling and Co-simulation in the Engineering of Cyber-physical Systems: Towards the Digital Twin[M]//Maurice H ter Beek, Alessandro Fantechi, Laura Semini. From Software Engineering to Formal Methods and Tools, and Back: Essays Dedicated to Stefania Gnesi on the Occasion of Her 65th Birthday, 2019: 40-55.
[49]	Al-Hammouri Ahmad T. A Comprehensive Co-simulation Platform for Cyber-physical Systems[J]. Computer Communications, 2012, 36(1): 8-19.
[50]	Helmy Mona, Sobhy Omar, ElHusseiny Farida. AI-driven Testing: Unleashing Autonomous Systems for Superior Software Quality Using Generative AI[C]//2024 International Telecommunications Conference (ITC-Egypt). Piscataway: IEEE. 2024: 1-6.
[51]	Huang Yu, Yu A, Liu L, et al. AI Driven Testing[C]//2024 IEEE International Test Conference in Asia (ITC-Asia). Piscataway: IEEE, 2024: 1.
[52]	赵祥模国家重点研发计划( 2021 YFB2501200)团队. 自动驾驶测试与评价技术研究进展[J]. 交通运输工程学报, 2023, 23(6): 10-77.
	Zhao Xiangmo's Team Supported by the National Key Research and Development Program of China (2021 YFB2501200). Research Progress in Testing and Evaluation Technologies for Autonomous Driving[J]. Journal of Traffic and Transportation Engineering, 2023, 23(6): 10-77.
[53]	刘晶郁, 马辉, 张学文, 等. 自动驾驶虚拟仿真测试技术研究进展[J]. 中国科技论文, 2021, 16(6): 571-577, 584.
	Liu Jingyu, Ma Hui, Zhang Xuewen, et al. Research and Development of Autonomous Driving Virtual Simulation Technology[J]. China Sciencepaper, 2021, 16(6): 571-577, 584.
[54]	袁先旭, 陈坚强, 杜雁霞, 等. 国家数值风洞(NNW)工程中的CFD基础科学问题研究进展[J]. 航空学报, 2021, 42(9): 23-40.
	Yuan Xianxu, Chen Jianqiang, Du Yanxia, et al. Research Progress on Fundamental CFD Issues in National Numerical Windtunnel Project[J]. Acta Aeronautica et Astronautica Sinica, 2021, 42(9): 23-40.
[55]	张念凡, 肖龙飞, 陈刚. 海洋结构物波浪砰击的数值研究综述[J]. 上海交通大学学报, 2024, 58(2): 127-140.
	Zhang Nianfan, Xiao Longfei, Chen Gang. A Review of Numerical Studies of Wave Impacts on Marine Structures[J]. Journal of Shanghai Jiao Tong University, 2024, 58(2): 127-140.
[56]	孙浩, 张旭辉, 王力, 等. 数据驱动靶场静态毁伤数字试验技术[J]. 现代应用物理, 2024, 15(3): 171-175.
	Sun Hao, Zhang Xuhui, Wang Li, et al. Digital Test Technology of Data-driven Static Damage[J]. Modern Applied Physics, 2024, 15(3): 171-175.
[57]	孙惠斌, 颜建兴, 魏小红, 等. 数字孪生驱动的航空发动机装配技术[J]. 中国机械工程, 2020, 31(7): 833-841.
	Sun Huibin, Yan Jianxing, Wei Xiaohong, et al. Digital Twin-driven Aero-engine Assembly Technology[J]. China Mechanical Engineering, 2020, 31(7): 833-841.
[58]	白浩, 周长城, 袁智勇, 等. 基于数字孪生的数字电网展望和思考[J]. 南方电网技术, 2020, 14(8): 18-24, 40.
	Bai Hao, Zhou Changcheng, Yuan Zhiyong, et al. Prospect and Thinking of Digital Power Grid Based on Digital Twin[J]. Southern Power System Technology, 2020, 14(8): 18-24, 40.
[59]	余红星, 李文杰, 柴晓明, 等. 数字反应堆发展与挑战[J]. 核动力工程, 2020, 41(4): 1-7.
	Yu Hongxing, Li Wenjie, Chai Xiaoming, et al. Digital Reactor: Development and Challenges[J]. Nuclear Power Engineering, 2020, 41(4): 1-7.
[60]	石小林, 王为. 数字空间站建设及其应用[J]. 航天器工程, 2022, 31(6): 76-85.
	Shi Xiaolin, Wang Wei. Digital Space Station and Its Application[J]. Spacecraft Engineering, 2022, 31(6): 76-85.
[61]	Olbrechts Thierry. Digital Transformation of Verification Process for Faster Aircraft Certification[EB/OL]. (2023-03-13) [2025-03-25]. .
[62]	佚名. 国务院关于印发"十四五"数字经济发展规划的通知[J]. 中华人民共和国国务院公报, 2022(3): 5-18.
[63]	佚名. 中华人民共和国国民经济和社会发展第十四个五年规划和2035年远景目标纲要[N]. 人民日报, 2021-03-13(001).
[64]	佚名. 加快数字人才培育支撑数字经济发展行动方案(2024—2026年)[J]. 工业信息安全, 2024(3): 83-85.
	Anon. Action Plan to Accelerate the Cultivation of Digital Talents to Support the Development of the Digital Economy (2024—2026) Was Released[J]. Industry Information Security, 2024(3): 83-85.
[65]	佚名. 中共中央国务院印发《教育强国建设规划纲要(2024—2035年)》[N]. 人民日报, 2025-01-20(006).
[66]	陶飞, 马昕, 张辰源, 等. 数字试验测试验证标准体系[J]. 计算机集成制造系统, 2025, 31(1): 1-19.
	Tao Fei, Ma Xin, Zhang Chenyuan, et al. A Standard System for Digital Experiment, Testing, and Validation (D-ETV)[J]. Computer Integrated Manufacturing Systems, 2025, 31(1): 1-19.
[67]	弗若斯特沙利文. 中国工业无人机行业研究报告[EB/OL]. (2020-02-14) [2025-03-25]. .
[68]	麦肯锡. 2024麦肯锡中国汽车消费者洞察[EB/OL]. [2025-03-25].
[69]	北京中研普华产业研究院. 2024—2030年中国海洋机器人行业市场发展趋势与前景展望战略分析报告[EB/OL]. [2025-03-25]. .