Parallel Pattern Recognition of Leak Current Data Using Spark-KNN

doi:10.16182/j.issn1004731x.joss.201804032

Abstract

Abstract: With the rapid development of smart grid, the status monitoring data of power grid equipment increase exponentially and gradually form the big data. Traditional computing architectures are no longer to meet the demand of computing performance. This paper explores how Spark and Cloud computing can accelerate performance of missive insulator leak current data pattern recognition. The Parallel KNN (k-Nearest Neighbor) algorithm is designed and implemented by using Spark and Aliyun E-MapReduce cloud computing platform. The results from experiments show that the performance of Spark-KNN is 2.97 times of MapReduce-KNN and gains acceleration of 8.8 times. The experimental results confirm that Spark is more suitable for real time data processing tasks than MapReduce.

Key words: power grid equipment, online monitoring, big data, Spark

CLC Number:

TM764

Li Li, Zhu Yongli, Song Yaqi. Parallel Pattern Recognition of Leak Current Data Using Spark-KNN[J]. Journal of System Simulation, 2018, 30(4): 1473-1481.

References

[1] 周国亮, 朱永利, 王桂兰, 等. 实时大数据处理技术在状态监测领域中的应用[J]. 电工技术学报, 2014, 29(增1): 432-437.
Zhou Guoliang, Zhu Yongli, Wang Guilan, et al.Real- Time Big Data Processing Technology Application in the Field of State Monitoring[J]. TRANSACTIONS OF CHINA ELECTROTECHNICAL SOCIETY, 2014, 29(S1): 432-437.
[2] Dean J, Ghemawat S.MapReduce: simplified data processing on large clusters[J]. Communications of the ACM(S0001-0782), 2008, 51(1): 107-113.
[3] Zaharia M, Chowdhury M, Das T, et al.Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing[C]//Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, USENIX Association, 2012: 2.
[4] Zaharia M, Chowdhury M, Das T, et al.Fast and interactive analytics over Hadoop data with Spark[J]. USENIX; login, 2012, 37(4): 45-51.
[5] Cover T, Hart, P.Nearest neighbor pattern classification[J]. IEEE Trans. Inf. Theory, 1967, 30(1): 21-27.
[6] 曲朝阳, 朱莉, 张士林. 基于Hadoop的广域测量系统数据处理[J]. 电力系统自动化, 2014, 37(4): 92-97.
QU Zhaoyang, ZHU Li, ZHANG Shilin.Data Processing of Hadoop-based Wide Area Measurement System[J]. Automation of Electric Power Systems, 2014, 37(4): 92-97.
[7] Christophe Bisciglia. The smart grid: Hadoop at the tennessee valley authority(TVA)[EB/OL] http://www. cloudera.com/blog/2009/06/smart-grid-hadoop-tennesseevalley-authority-tva/
[8] 齐林海, 艾明浩. 一种基于云计算的电压暂降并行计算方法[J]. 中国电机工程学报, 2014, 34(31):5493-5499.
QI Linhai, AI Minghao.A Voltage Sag Parallel Calculation Method Based on Cloud Computing[J]. Proceedings of the CSEE, 2014, 34(31):5493-5499.
[9] 张逸, 杨洪耕, 叶茂清. 基于分布式文件系统的海量电能质量监测数据管理方案[J]. 电力系统自动化, 2014, 38(2): 102-108.
Zhang Yi, YANG Honggeng, YE Maoqing.A Data Management Scheme for Massive Power Quality Monitoring Data Based on Distributed File System[J]. Automation of Electric Power Systems, 2014, 38(2): 102-108.
[10] Shvachko K, Kuang H, Radia S, et al.The hadoop distributed file system[C]//2010 IEEE 26th Symposium on Mass Storage Systems and Technologies. IEEE, 2010: 1-10.
[11] Wang B, Huang S, Qiu J, et al.Parallel online sequential extreme learning machine based on MapReduce[J]. NeuroComputing(S0925-2312), 2015, 149: 224-232.
[12] 王德文, 孙志伟. 电力用户侧大数据分析与并行负荷预测[J]. 中国电机工程学报, 2015 , 35(3): 527-537.
WANG Dewen, SUN Zhiwei.Big Data Analysis and Parallel Load Forecasting of Electric Power User Side[J]. Proceedings of the CSEE, 2015 , 35(3): 527-537.
[13] 曲广龙, 杨洪耕, 张逸. 采用Map-Reduce模型的海量电能质量数据交换格式文件快速解析方案[J]. 电网技术, 2014, 38(6): 1705-1711.
QU Guanglong, YANG Honggeng, ZHANG Yi.A Fast Parallel Parsing Scheme for Massive PQDIF Files With Map-Reduce Model[J]. Power System Technology, 2014, 38(6): 1705-1711.
[14] Fadika Z, Govindaraju M, Canon R, et al.Evaluating hadoop for data-intensive scientific operations[C]//2012 IEEE 5th International Conference on Cloud Computing (CLOUD)[C]. IEEE, 2012: 67-74.
[15] 王德文, 刘晓建. 基于MapReduce的电力设备并行故障诊断方法[J]. 电力自动化设备, 2014, 34(10): 116-120.
WANG Dewen, LIU Xiaojian.Parallel fault diagnosis based on MapReduce for electric power equipments[J]. Electric Power Automation Equipment, 2014, 34(10): 116-120.
[16] 王远, 陶烨, 袁军, 等. 一种基于HBase的智能电网时序大数据处理方法[J]. 系统仿真学报, 2016, 28(3): 559-568.
Wang Yuan, Tao Ye, Yuan Jun, et al.Approach to Process Smart Grid Time-Serial Big Data Base on HBase[J]. Journal of System Simulation, 2016, 28(3): 559-568.
[17] 邵士雯. 超高压线路绝缘子覆冰泄漏电流特性研究[D]. 保定: 华北电力大学, 2012.
Shao Shiwen.Research on Characteristics of Leakage Current of EHV Transmission Line leed Insulator Strings [D]. Baoding: North China Electric Power University, 2012.