高通量仿真中数据存储与访问技术研究

doi:10.16182/j.issn1004731x.joss.201709019

系统仿真学报 ›› 2017, Vol. 29 ›› Issue (9): 2016-2024.doi: 10.16182/j.issn1004731x.joss.201709019

高通量仿真中数据存储与访问技术研究

王子硕¹, 翟岩龙¹, 陶文俊¹, 杨浩¹, 张晗², 卿杜政²

1.北京理工大学计算机学院,北京 100081;
2.北京仿真中心航天系统仿真重点实验室,北京 100854

收稿日期:2017-05-20 发布日期:2020-06-02
作者简介:王子硕(1993-),男,山东,硕士,研究方向为分布式并行仿真;翟岩龙(1981-),男,北京,博士,讲师,研究方向为分布式并行仿真等。
基金资助:
国家自然科学基金(61602037)

Research on Technology of Data Storage and Access in High-throughput Simulation

Wang Zishuo¹, Zhai Yanlong¹, Tao Wenjun¹, Yang Hao¹, Zhang Han², Qing Duzheng²

1. School of Computer Science, Beijing Institute of Technology, Beijing 100081, China;
2. Science and Technology on Special System Simulation Laboratory, Beijing Simulation Center, Beijing 100854, China

Received:2017-05-20 Published:2020-06-02

摘要/Abstract

摘要： 针对大数据环境下仿真应用的海量数据处理需求,参照面向大数据高通量计算的架构和技术特点,提出并定义了高通量仿真的概念和参考架构。针对高通量仿真中数据访问瓶颈问题,设计了基于分布式内存文件系统的高通量仿真数据存储和访问体系,并扩展集成了非易失性存储器进一步提高数据访问的吞吐量。通过对典型仿真应用的实验验证,表明采用分布式内存文件系统和非易失性存储器的高通量仿真存储系统能够有效提升高通量仿真数据存储与访问性能。

关键词: 大数据, 高通量仿真, 非易失性内存, 分布式文件系统

Abstract: Aiming at the massive data processing requirements of simulation application in the big data environment, the concept and reference structure of high-throughput simulation were proposed and defined in accordance with the architecture and technical characteristics of high-throughput big data computing. For the problem of data access bottleneck in high-throughput simulation, a high-throughput simulation data storage and access system was designed based on distributed memory file system, and the non-volatile memory was integrated to improve the throughput of data access. The experimental results of typical simulation applications show that the high-throughput simulation storage system with distributed memory file system and non-volatile memory can effectively improve the data storage and access performance of high-throughput simulation.

Key words: big data, high-throughput simulation, non-volatile memory, distributed file system

中图分类号:

TP391.1

王子硕, 翟岩龙, 陶文俊, 杨浩, 张晗, 卿杜政. 高通量仿真中数据存储与访问技术研究[J]. 系统仿真学报, 2017, 29(9): 2016-2024.

Wang Zishuo, Zhai Yanlong, Tao Wenjun, Yang Hao, Zhang Han, Qing Duzheng. Research on Technology of Data Storage and Access in High-throughput Simulation[J]. Journal of System Simulation, 2017, 29(9): 2016-2024.

参考文献

[1] 候彦庆. 计算机仿真技术的应用与发展趋势[J]. 信息通信, 2016 (2): 181-182.
(Hou Yanqing.The Application and Development Trend of Computer Simulation Technology[J]. Information & Communications, 2016 (2): 181-182.)
[2] 李宏亮. 基于Agent的复杂系统分布仿真 [D]. 长沙:国防科学技术大学, 2001.
(Li Hongliang.The Agent-Based Distributed Simulation for Complex Systems [D]. Changsha, China: National University of Defense Technology, 2001.)
[3] 李琼. 面向高性能计算的可扩展I/O体系结构研究与实现 [D]. 长沙: 国防科学技术大学, 2009.
(Li Qiong.Research on I/O Architecture and Implementing Technology for High Performance Computing [D]. Changsha, China: National University of Defense Technology, 2009.)
[4] May J M.Parallel I/O for high performance computing[M]. USA: Morgan Kaufmann Publisher, 2001: 178-179.
[5] 徐越. 面向高性能计算的分布式内存文件缓存系统 [D]. 武汉: 华中科技大学, 2013.
(Xu Yue.Distributed Memory File Caching System for High Performance Computing [D]. Wuhan, China: Huazhong University of Science and Technology, 2013.)
[6] 王士斌. 大数据应用技术浅析[J]. 计算机光盘软件与应用, 2014 (8): 96-97.
(Wang Shibin.Analysis of Application Technology of Large Data[J]. Computer CD Software and Applications, 2014 (8): 96-97.)
[7] Schroeder B, Gibson G A.A large-scale study of failures in high-performance computing systems[C]// International Conference on Dependable Systems and Networks. USA: IEEE, 2006: 249-258.
[8] 刘颖, 陈煜, 林林, 等. 高性能计算集群中的网络技术研究与实践[J]. 中国水利水电科学研究院学报, 2016, 14(2): 90-95.
(Liu Ying, Chen Yu, Lin Lin, et al.Research and practice of high performance computing cluster network technology[J]. Journal of China Institute of Water Resources and Hydropower Research, 2016, 14(2): 90-95.)
[9] Iwata K, Onosato M, Teramoto K, et al.A Modelling and Simulation Architecture for Virtual Manufacturing Systems[J]. CIRP Annals - Manufacturing Technology (S0007-8506), 1995, 44(1): 399-402.
[10] Liang S, Noronha R, Panda D K.Swapping to remote memory over infiniband: an approach using a high performance network block device[C]// Proc of the IEEE International Conference on Cluster Computing (Cluster2005). USA: IEEE, 2005.
[11] Grošelj B, Tropper C.The distributed simulation of clustered processes[J]. Distributed Computing (S0178-2770), 1991, 4(3): 111-121.
[12] 蔡涛, 牛德姣, 刘扬宽, 等. NVMMDS—一种面向非易失存储器的元数据管理方法[J]. 计算机研究与发展, 2013, 50(1): 69-79.
(Cai Tao, Niu Dejiao, Liu Yangkuan, et al.NVMMDS—Metadata Management Method Based on Non-Volatile Memory[J]. Journal of Computer Research and Development, 2013, 50(1): 69-79.)
[13] Manoj Sinha, Steven Hsu, Atila Alvandpour.High-Performance and Low-Voltage Sense-Amplifier Techniques for sub-90nm SRAM[C]// Proceedings of IEEE International SOC Conference. USA: IEEE, 2003.
[14] 陈怡然, 赵巍胜, 孙振宇, 等. 新型非易失性存储器[J]. 现代物理知识, 2015 (1): 41-46.
(Chen Yiran, Zhao Weisheng, Sun Zhenyu, et al.New Non-volatile Memory[J]. Modern Physics, 2015 (1): 41-46.)
[15] 何炎祥, 沈凡凡, 张军, 等. 新型非易失性存储器架构的缓存优化方法综述[J]. 计算机研究与发展, 2015, 52(6): 1225-1241.
(He Yanxiang, Shen Fanfan, Zhang Jun, et al.Cache Optimization Approaches of Emerging Non-Volatile Memory Architecture:A Survey[J]. Journal of Computer Research and Development, 2015, 52(6): 1225-1241.)

高通量仿真中数据存储与访问技术研究

Research on Technology of Data Storage and Access in High-throughput Simulation

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 14

编辑推荐

Metrics

本文评价

[1]	孙一铃, 陈谊, 单桂华, 李晓兴. 基于AR技术的多人互动地球仪系统[J]. 系统仿真学报, 2022, 34(6): 1367-1374.
[2]	胡杨, 张小咏, 肖迪. 基于手机大数据的城市建筑物人口时空估算[J]. 系统仿真学报, 2020, 32(10): 1874-1883.
[3]	黄子赫, 高尚兵, 潘志庚, 惠浩, 廖麒羽, 赵锋锋. 基于快速密度聚类的载客热点可视化分析方法[J]. 系统仿真学报, 2019, 31(7): 1429-1438.
[4]	司光亚, 王飞, 刘洋. 基于仿真大数据的体系分析方法研究[J]. 系统仿真学报, 2019, 31(3): 511-519.
[5]	孔源, 杜怡曼, 吴建平, 胡可臻. 基于浮动车大数据的电动汽车潜在用户分析[J]. 系统仿真学报, 2019, 31(3): 575-584.
[6]	肖迪, 张小咏, 胡杨. 基于手机大数据的城市功能区识别方法[J]. 系统仿真学报, 2019, 31(11): 2281-2288.
[7]	李莉, 朱永利, 宋亚奇. 泄漏电流数据的Spark-KNN并行模式识别方法[J]. 系统仿真学报, 2018, 30(4): 1473-1481.
[8]	张明智, 李义. Map Reduce框架下空间大数据的关联规则分析方法[J]. 系统仿真学报, 2018, 30(3): 840-845.
[9]	董泽, 尹二新. 基于状态跟踪的非线性工业系统全工况建模[J]. 系统仿真学报, 2018, 30(3): 846-856.
[10]	王刘旺, 朱永利, 贾亚飞. 一种多源海量局部放电信号脉冲的并行提取方法[J]. 系统仿真学报, 2017, 29(1): 57-66.
[11]	谈敦铭, 曹国廷, 郎娟芳, 杨朔. 飞行器大数据量CAD模型并行绘制[J]. 系统仿真学报, 2016, 28(9): 2049-2054.
[12]	韦容, 申希兵, 杨毅. SMT有界约束非集中自动机web服务模型检测[J]. 系统仿真学报, 2016, 28(9): 2283-2288.
[13]	万聪, 王翠荣, 王聪. 面向数据密集型工作流的QoS-aware调度算法[J]. 系统仿真学报, 2016, 28(3): 549-558.
[14]	王远, 陶烨, 袁军, 何卫. 一种基于HBase的智能电网时序大数据处理方法[J]. 系统仿真学报, 2016, 28(3): 559-568.