一种基于HBase的智能电网时序大数据处理方法

系统仿真学报 ›› 2016, Vol. 28 ›› Issue (3): 559-568.

一种基于HBase的智能电网时序大数据处理方法

王远¹, 陶烨², 袁军¹, 何卫¹

1.国网电科院江苏瑞中数据股份有限公司,江苏南京 210012;
2.南京大学计算机科学与技术系,江苏南京 210046

收稿日期:2014-10-17 修回日期:2014-11-05 发布日期:2020-07-02
作者简介:王远(1980-),男,山东青岛,博士,高工,研究方向为大规模并行处理技术、海量数据管理和软件体系结构;陶烨(1981-),女,江苏常州,硕士,讲师,研究方向为并行计算与分布式系统。
基金资助:
国家863计划(2011AA05A116);国家电网科技项目(62KJ130501B1003620120)

Approach to Process Smart Grid Time-Serial Big Data Based on HBase

Wang Yuan¹, Tao Ye², Yuan Jun¹, He Wei¹

1. China Realtime Database Co., Ltd, State Grid Electric Power Institute, Nanjing 210012, China;
2. Department of Computer Science & Technology, Nanjing University, Nanjing 210046, China

Received:2014-10-17 Revised:2014-11-05 Published:2020-07-02

摘要/Abstract

摘要： 随着物联网关键技术与理论的发展,物联网应用受到了越来越多的关注。智能电网是一类典型的物联网应用,遍布全网的传感器收集及产生了大量反映关键设备运行状态的时序数据。如何利用时序数据确保智能电网的安全以及稳定运行是当前的研究热点之一。针对智能电网时序数据设备多、数据规模大、产生速度快等特点,提出了一种基于HBase的海量时序数据存储处理方法,着重介绍了如何利用策略驱动技术实现时序数据的灵活存储与处理。通过构建HBase集群,验证了该方法的有效性。

关键词: 大数据, 物联网, HBase, 时序数据, 性能优化

Abstract: With the development of critical theories and technologies in Internet of things (IOT), more and more attentions have been focused on the IOT applications. Smart Grid is one of the typical IOT applications on which a huge number of sensors have been deployed to gather and generate time-serial data to make sense of the running states of the key devices. How to apply these data to make smart grid running secure and stable is a hot research topic. By considering the fact that smart grid is characterized by a huge number of devices, a huge amount of data and the high speed data generating, an approach was proposed to store and process big time-serial data in smart grid based on the HBase. The emphasis is a policy-driven method to make the time-serial data organized in HBase on demand. The HBase cluster was deployed to verify effectiveness of the approach.

Key words: Big data, Internet of things, HBase, Time-serial data, performance optimization

中图分类号:

TP274+.2

王远, 陶烨, 袁军, 何卫. 一种基于HBase的智能电网时序大数据处理方法[J]. 系统仿真学报, 2016, 28(3): 559-568.

Wang Yuan, Tao Ye, Yuan Jun, He Wei. Approach to Process Smart Grid Time-Serial Big Data Based on HBase[J]. Journal of System Simulation, 2016, 28(3): 559-568.

参考文献

[1] HayT, Tansleys, TolleK. The Fourth Paradigm: Data-Intensive Science Discovery[M]. Redmond, Washington, USA: Microsoft Research, 2009.
[2] 宫学庆, 金澈清, 王晓玲, 等, 数据密集型科学与工程: 需求和挑战[J]. 计算机学报, 2012, 35(8): 1563-1578.
[3] 王珊, 王会举, 覃雄派, 等, 架构大数据: 挑战、现状与展望[J]. 计算机学报, 2011, 34(10): 1741-1752.
[4] 王元卓, 靳小龙, 程学旗, 等, 网络大数据: 现状与挑战[J]. 计算机学报, 2013, 36(6): 1125-1138.
[5] 覃雄派, 王会举, 杜小勇, 等. 大数据分析—RDBMS 与MapReduce 的竞争与共生[J]. 软件学报, 2012, 23(1): 32-45.
[6] 覃雄派, 王会举, 李芙蓉, 等. 数据管理技术的新格局[J]. 软件学报, 2013, 24(2): 175-197.
[7] Gray J, Liu DT, Nieto-Santisteban M, et al.Scientific data management in the coming decade[J]. ACM SIGMOD Record, 2005, 34(4): 34-41.
[8] 丁道奇. 复杂大电网安全性分析 [M]. 北京: 中国电力出版社, 2010.
[9] Apache Software Foundation. The Apache HBase Reference Guide [EB/OL]. (20140-06-10) [2014-08-12]. http://hbase.apache.org/book /book.html
[10] Prichard J.RTSQL: Extending the SQL Standard To Support Real—Time Database [D]// Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Applied Mathematical Sciences. USA: University of Rhode Island, 1995.
[11] Azer Bestavros, Kwei-Jay Lin, Sang Hyuk Son.Real-time database systems: Issues and applications[M]. Springer Science & Business Media, 2012.
[12] Pugll W, Marlow. Compiler and tool support for real—time systems[C]// Proceedings of the ACM SIGPLAN workshop on language, ACM SIGPLAN. USA: ACM, 1994.
[13] Ramamritham K.Real—time databases[J]. International Journal of Distributed and Paralled Databases (S0926-8782), 1993, 1(2): 199-226.
[14] Jeffrey Dean, Sanjay Ghemawat, MapReduce: Simplifed Data Processing on Large Clusters, Operating Systems Design and Implementation[J]. Communications of the ACM (S0001-0782), 2008, 51(1): 107-113.
[15] Apache Software Foundation. HDFS Users Guide [EB/OL]. (2014-08-01) [2014-07-20]. http:// hadoop. apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html
[16] Ghemawat S, Gobioff H, Leung S-T.The Google file system[C]// Proc. of the 19th ACM SOSP (Dec. 2003). USA: ACM, 2003: 29-43.
[17] Opentsdb.net. OpenTSDB - A Distributed, Scalable Monitoring System [EB/OL]. (2014) [2014-08-30]. http://opentsdb.net/.
[18] Chang F, Dean J, Ghemawat S, et al.Bigtable: A Distributed Storage System for Structured Data[J]. ACM Transactions on Computer Systems (TOCS)(S0734-2071), 2006, 26(2): 205-218.
[19] Paul Zikopoulos, Chris Eaton.Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data[M]. USA: McGraw-Hill Osborne Media, 2011.
[20] N Dimiduk, A Khurana, MH Ryan, et al. HBase in Action[M]. USA: O'Reilly Media, 2012.
[21] Yifeng Jiang. HBase Cookbook Administration.Birmingham [M]. UK: PACKT Publishing, 2012.