构建自动演进的天文大数据负载模型

doi:10.16182/j.issn1004731x.joss.201809009

系统仿真学报 ›› 2018, Vol. 30 ›› Issue (9): 3293-3305.doi: 10.16182/j.issn1004731x.joss.201809009

构建自动演进的天文大数据负载模型

王华进^1,2, 万萌³, 韩锐⁴, 任玮⁵, 张海明¹, 黎建辉^1,*

1. 中国科学院计算机网络信息中心, 北京 100190;
2. 中国科学院大学, 北京 100049;
3. 中国科学院国家天文台, 北京 100012;
4. 中国科学院计算技术研究所, 北京 100190;
5. 中国人民大学, 北京 100872

收稿日期:2016-12-08 出版日期:2018-09-10 发布日期:2019-01-08
作者简介:王华进(1987-),男,山东茌平,博士生,研究方向为大规模数据管理、负载仿真;万萌(1983-),女,山东青岛,博士,工程师,研究方向为天文数据库系统。
基金资助:
国家重点研发计划(2016YFB1000600,2016YFB0501900)

Towards the Automatic Evolution of Workload Models in Large-scale Astronomical Data Management

Wang Huajin^1,2, Wan Meng³, Han Rui⁴, Ren Wei⁵, Zhang Haiming¹, Li Jianhui^1,*

1. Computer Network Information Center, Chinese Academy of Sciences, Beijing 100190, China;
2. University of Chinese Academy of Sciences, Beijing 100049, China;
3. National Astronomical Observatories, Chinese Academy of Sciences, Beijing 100012, China;
4. Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China;
5. Renmin University of China, Beijing 100872, China

Received:2016-12-08 Online:2018-09-10 Published:2019-01-08

摘要/Abstract

摘要： 基准测试在数据管理系统的选型和优化中发挥指导作用的前提是其采用的负载模型能够：运行在目标场景中的各类系统上(移植性);反映目标场景中典型任务的特点和数据访问偏好(代表性)。当前天文大数据管理领域的新系统和新任务层出不穷,导致现有方法构建的负载模型容易失去移植性和代表性。提出了自动演进的负载建模方法：采用抽象操作保持对新型系统的移植性,通过分析负载日志保持对新型任务的代表性。通过一个系统优化案例展示了该方法的可行性。

关键词: 负载模型, 基准测试, 天文数据管理, 查询优化

Abstract: The benchmark's guiding role in system selection/optimization requires its workload model has the ability to: Run on various systems of the target application scenario (be portable); Reflect the typical tasks' characteristics and data access patterns (be representative). The emerging systems and tasks in large-scale astronomical data management field have led workload models constructed by existing methods to be prone to lose portability and representativeness. An automatic evolutionary workload modeling method has been proposed: Abstract operations are used to keep the workload model’s portability; Automatic workload log analytics are used to keep the workload model’s representativeness. The feasibility of this method is verified by a cluster optimization case.

Key words: workload model, benchmarking, astronomical data management, query optimization

中图分类号:

TP.391.9

王华进, 万萌, 韩锐, 任玮, 张海明, 黎建辉. 构建自动演进的天文大数据负载模型[J]. 系统仿真学报, 2018, 30(9): 3293-3305.

Wang Huajin, Wan Meng, Han Rui, Ren Wei, Zhang Haiming, Li Jianhui. Towards the Automatic Evolution of Workload Models in Large-scale Astronomical Data Management[J]. Journal of System Simulation, 2018, 30(9): 3293-3305.

参考文献

[1] Albareti F D, Prieto C A, Almeida A. The Thirteenth Data Release of the Sloan Digital Sky Survey: First Spectroscopic Data from the SDSS-IV Survey MApping Nearby Galaxies at Apache Point Observatory[EB/OL].2016. arXiv preprint: 1608. 02013.
[2] Ivezic Z, Tyson J A, Abel B. LSST: from Science Drivers to Reference Design and Anticipated Data Products [EB/OL].2014. arXiv preprint: 0805.2366.
[3] Wan M, Wu C, Wang J, et al.Column Store for GWAC: A High-cadence, High-density, Large-scale Astronomical Light Curve Pipeline and Distributed Shared-nothing Database[J]. Publications of the Astronomical Society of the Pacific (S0004-6280), 2016, 128(969): 114501.
[4] 万萌, 吴潮, YingZhang, 等. GWAC 海量星表数据处理的数据库系统选型研究[J]. 天文研究与技术, 2016, 13(3): 373-381.
Wan Meng, Wu Chao, Zhang Ying, et al.A Pre-research on GWAC Massive Catalog Data Storage and Processing System[J]. Astronomical Research & Technology- Publications of National Astronomical Observatories of China, 2016, 13(3): 373-381.
[5] Shaw R A.LSST Data Challenge Handbook (Version 2.0) [R]. Tucson, AZ: LSST Corp, 2012: 1-44.
[6] Wang D L, Monkewitz S M, Lim K T, et al.Qserv: A Distributed Shared-nothing Database for the LSST Catalog[C]// International Conference for High Performance Computing, Networking, Storage and Analysis,Seattle, WA: IEEE, 2011: 1-11.
[7] Mesmoudi A, Hacid M S, Toumani F.Benchmarking SQL on MapReduce Systems Using Large Astronomy Databases[J]. Distributed and Parallel Databases (S0926-8782), 2016, 34(3): 347-378.
[8] Ghazal A, Rabl T, Hu M, et al.BigBench: Towards an Industry Standard Benchmark for Big Data Analytics[C]// ACM SIGMOD International Conference on Management of Data.New York: ACM, 2013: 1197-1208.
[9] Wang L, Zhang S, Zheng C, et al.BigDataBench: A Big Data Benchmark Suite from Internet Services[C]// IEEE International Symposium on High Performance Computer Architecture. Orlando, FL:IEEE, 2014: 488-499.
[10] Han R, Jia Z, Gao W, et al. Benchmarking Big Data Systems: State-of-the-Art and Future Directions[EB/OL].2015. arXiv preprint: 1506. 01494.
[11] Chen Y, Alspaugh S, Katz R.Interactive Analytical Processing in Big Data Systems: A Cross-industry Study of MapReduce Workloads[C]// Proceedings of the VLDB Endowment.Istanbul, Turkey:2012: 1802-1813.
[12] Mishra A K, Hellerstein J L, Cirne W.Towards Characterizing Cloud Backend Workloads: Insights from Google Compute Clusters[J]. ACM SIGMETRICS Performance Evaluation Review (S0163-5999), 2010, 37(4): 34-41.
[13] Ren Z, Wan J, Shi W, et al.Workload Analysis, Implications, and Optimization on A Production Hadoop Cluster: A Case Study on Taobao[J]. IEEE Transactions on Services Computing (S1939-1374), 2014, 7(2): 307-321.
[14] Kavulya S, Tan J, Gandhi R, et al.An Analysis of Traces from A Production Mapreduce Cluster[C]// IEEE/ACM International Conference on Cluster, Cloud and Grid Computing. Victoria, Australia:IEEE, 2010: 94-103.
[15] Chen Y, Alspaugh S, Katz R. Design Insights for Mapreduce from Diverse Production Workloads[R]. Berkeley, CA: University of California, Berkeley, 2012: UCB/EECS-2012-17.
[16] 万萌. 列存储MonetDB数据库在GWAC海量天文数据管理的应用研究[D]. 北京: 中国科学院大学, 2016.
Wan Meng.An Application Research of Column Store MonetDB Database on GWAC Large-scale Astronomical Data Management[D]. Beijing: The University of Chinese Academy of Sciences, 2016.

构建自动演进的天文大数据负载模型

Towards the Automatic Evolution of Workload Models in Large-scale Astronomical Data Management

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 11

编辑推荐

Metrics

本文评价

[1]	贾立山, 李昂. 基于改进欧拉角公式的飞行仿真建模方法[J]. 系统仿真学报, 2021, 33(2): 339-345.
[2]	张舵, 刘满国, 章校, 刘平安, 何宜轩. 基于半实物仿真环境的系统误差分析研究[J]. 系统仿真学报, 2019, 31(1): 53-59.
[3]	王爱珍, 吕勋. 基于非完美调试和测试工作量的软件可靠性研究[J]. 系统仿真学报, 2018, 30(7): 2593-2599.
[4]	杨曦中, 艾剑良. 自主空战中无人机规避导弹机动策略研究[J]. 系统仿真学报, 2018, 30(5): 1957-1966.
[5]	陈坚. 船用发电柴油机调速器的数学建模和仿真研究[J]. 系统仿真学报, 2018, 30(3): 937-942.
[6]	王梅亮, 万华根, 顾键萍. 基于游戏模式的行人横穿道路技能辅助训练系统研究[J]. 系统仿真学报, 2016, 28(6): 1406-1412.
[7]	傅妍芳, 李路, 高祥, 程兵. 基于TDMA的卫星战术通信网路由协议仿真研究[J]. 系统仿真学报, 2016, 28(2): 467-475.
[8]	李春胜, 张雪. 一类具有双向时延的脑皮层电模型的分支研究[J]. 系统仿真学报, 2015, 27(12): 3063-3069.
[9]	严宗睿, 巫银花, 陈勇, 董建树. 装备作战需求探索性仿真实验方法研究[J]. 系统仿真学报, 2015, 27(8): 1888-1894.
[10]	贾立山, 王立文. 机场消防车消防泵系统实时仿真模型的构建[J]. 系统仿真学报, 2015, 27(6): 1209-1213.
[11]	张远, 蒋浩, 朱登明, 王兆其. 基于骨架匹配的人体轮廓线提取[J]. 系统仿真学报, 2017, 29(11): 2774-2781.