Towards the Automatic Evolution of Workload Models in Large-scale Astronomical Data Management

doi:10.16182/j.issn1004731x.joss.201809009

Abstract

Abstract: The benchmark's guiding role in system selection/optimization requires its workload model has the ability to: Run on various systems of the target application scenario (be portable); Reflect the typical tasks' characteristics and data access patterns (be representative). The emerging systems and tasks in large-scale astronomical data management field have led workload models constructed by existing methods to be prone to lose portability and representativeness. An automatic evolutionary workload modeling method has been proposed: Abstract operations are used to keep the workload model’s portability; Automatic workload log analytics are used to keep the workload model’s representativeness. The feasibility of this method is verified by a cluster optimization case.

Key words: workload model, benchmarking, astronomical data management, query optimization

CLC Number:

TP.391.9

Wang Huajin, Wan Meng, Han Rui, Ren Wei, Zhang Haiming, Li Jianhui. Towards the Automatic Evolution of Workload Models in Large-scale Astronomical Data Management[J]. Journal of System Simulation, 2018, 30(9): 3293-3305.

References

[1] Albareti F D, Prieto C A, Almeida A. The Thirteenth Data Release of the Sloan Digital Sky Survey: First Spectroscopic Data from the SDSS-IV Survey MApping Nearby Galaxies at Apache Point Observatory[EB/OL].2016. arXiv preprint: 1608. 02013.
[2] Ivezic Z, Tyson J A, Abel B. LSST: from Science Drivers to Reference Design and Anticipated Data Products [EB/OL].2014. arXiv preprint: 0805.2366.
[3] Wan M, Wu C, Wang J, et al.Column Store for GWAC: A High-cadence, High-density, Large-scale Astronomical Light Curve Pipeline and Distributed Shared-nothing Database[J]. Publications of the Astronomical Society of the Pacific (S0004-6280), 2016, 128(969): 114501.
[4] 万萌, 吴潮, YingZhang, 等. GWAC 海量星表数据处理的数据库系统选型研究[J]. 天文研究与技术, 2016, 13(3): 373-381.
Wan Meng, Wu Chao, Zhang Ying, et al.A Pre-research on GWAC Massive Catalog Data Storage and Processing System[J]. Astronomical Research & Technology- Publications of National Astronomical Observatories of China, 2016, 13(3): 373-381.
[5] Shaw R A.LSST Data Challenge Handbook (Version 2.0) [R]. Tucson, AZ: LSST Corp, 2012: 1-44.
[6] Wang D L, Monkewitz S M, Lim K T, et al.Qserv: A Distributed Shared-nothing Database for the LSST Catalog[C]// International Conference for High Performance Computing, Networking, Storage and Analysis,Seattle, WA: IEEE, 2011: 1-11.
[7] Mesmoudi A, Hacid M S, Toumani F.Benchmarking SQL on MapReduce Systems Using Large Astronomy Databases[J]. Distributed and Parallel Databases (S0926-8782), 2016, 34(3): 347-378.
[8] Ghazal A, Rabl T, Hu M, et al.BigBench: Towards an Industry Standard Benchmark for Big Data Analytics[C]// ACM SIGMOD International Conference on Management of Data.New York: ACM, 2013: 1197-1208.
[9] Wang L, Zhang S, Zheng C, et al.BigDataBench: A Big Data Benchmark Suite from Internet Services[C]// IEEE International Symposium on High Performance Computer Architecture. Orlando, FL:IEEE, 2014: 488-499.
[10] Han R, Jia Z, Gao W, et al. Benchmarking Big Data Systems: State-of-the-Art and Future Directions[EB/OL].2015. arXiv preprint: 1506. 01494.
[11] Chen Y, Alspaugh S, Katz R.Interactive Analytical Processing in Big Data Systems: A Cross-industry Study of MapReduce Workloads[C]// Proceedings of the VLDB Endowment.Istanbul, Turkey:2012: 1802-1813.
[12] Mishra A K, Hellerstein J L, Cirne W.Towards Characterizing Cloud Backend Workloads: Insights from Google Compute Clusters[J]. ACM SIGMETRICS Performance Evaluation Review (S0163-5999), 2010, 37(4): 34-41.
[13] Ren Z, Wan J, Shi W, et al.Workload Analysis, Implications, and Optimization on A Production Hadoop Cluster: A Case Study on Taobao[J]. IEEE Transactions on Services Computing (S1939-1374), 2014, 7(2): 307-321.
[14] Kavulya S, Tan J, Gandhi R, et al.An Analysis of Traces from A Production Mapreduce Cluster[C]// IEEE/ACM International Conference on Cluster, Cloud and Grid Computing. Victoria, Australia:IEEE, 2010: 94-103.
[15] Chen Y, Alspaugh S, Katz R. Design Insights for Mapreduce from Diverse Production Workloads[R]. Berkeley, CA: University of California, Berkeley, 2012: UCB/EECS-2012-17.
[16] 万萌. 列存储MonetDB数据库在GWAC海量天文数据管理的应用研究[D]. 北京: 中国科学院大学, 2016.
Wan Meng.An Application Research of Column Store MonetDB Database on GWAC Large-scale Astronomical Data Management[D]. Beijing: The University of Chinese Academy of Sciences, 2016.