泄漏电流数据的Spark-KNN并行模式识别方法

doi:10.16182/j.issn1004731x.joss.201804032

系统仿真学报 ›› 2018, Vol. 30 ›› Issue (4): 1473-1481.doi: 10.16182/j.issn1004731x.joss.201804032

泄漏电流数据的Spark-KNN并行模式识别方法

李莉, 朱永利, 宋亚奇

华北电力大学控制与计算机工程学院,河北保定 071003

收稿日期:2016-05-11 修回日期:2016-07-15 出版日期:2018-04-08 发布日期:2019-01-04
作者简介:李莉(1980-),女,重庆,硕士,讲师,研究方向为现代信号处理方法在电力系统故障诊断等方面的应用。
基金资助:
国家自然科学基金(51677072),河北省自然科学基金(A2016502001),中央高校基本科研业务费专项资金(2018MS074)

Parallel Pattern Recognition of Leak Current Data Using Spark-KNN

Li Li, Zhu Yongli, Song Yaqi

School of Control and Computer Engineering, North China Electric Power University, Baoding 071003, China

Received:2016-05-11 Revised:2016-07-15 Online:2018-04-08 Published:2019-01-04

摘要/Abstract

摘要： 随着智能电网的快速发展,电网设备状态监测数据呈指数级增长,逐渐构成电网设备状态监测大数据。传统计算架构已无法满足计算性能需求。结合Spark大数据处理技术和阿里云E-MapReduce云计算平台,提出电网设备状态监测大数据并行模式识别方法,旨在提升电网设备在线监测系统对短时间内骤增的报警监测数据快速批量分析的能力。设计了基于Spark的并行化k最近邻分类算法(k-Nearest Neighbor,KNN)Spark-KNN,实现了海量绝缘子泄漏电流数据的并行模式识别。实验结果表明,Spark-KNN的平均性能是Hadoop MapReduce实现的2.97倍,获得了最高8.8倍的加速比,更适合执行电力设备监测大数据的实时处理任务。

关键词: 电网设备, 在线监测, 大数据, Spark

Abstract: With the rapid development of smart grid, the status monitoring data of power grid equipment increase exponentially and gradually form the big data. Traditional computing architectures are no longer to meet the demand of computing performance. This paper explores how Spark and Cloud computing can accelerate performance of missive insulator leak current data pattern recognition. The Parallel KNN (k-Nearest Neighbor) algorithm is designed and implemented by using Spark and Aliyun E-MapReduce cloud computing platform. The results from experiments show that the performance of Spark-KNN is 2.97 times of MapReduce-KNN and gains acceleration of 8.8 times. The experimental results confirm that Spark is more suitable for real time data processing tasks than MapReduce.

Key words: power grid equipment, online monitoring, big data, Spark

中图分类号:

TM764

李莉, 朱永利, 宋亚奇. 泄漏电流数据的Spark-KNN并行模式识别方法[J]. 系统仿真学报, 2018, 30(4): 1473-1481.

Li Li, Zhu Yongli, Song Yaqi. Parallel Pattern Recognition of Leak Current Data Using Spark-KNN[J]. Journal of System Simulation, 2018, 30(4): 1473-1481.

参考文献

[1] 周国亮, 朱永利, 王桂兰, 等. 实时大数据处理技术在状态监测领域中的应用[J]. 电工技术学报, 2014, 29(增1): 432-437.
Zhou Guoliang, Zhu Yongli, Wang Guilan, et al.Real- Time Big Data Processing Technology Application in the Field of State Monitoring[J]. TRANSACTIONS OF CHINA ELECTROTECHNICAL SOCIETY, 2014, 29(S1): 432-437.
[2] Dean J, Ghemawat S.MapReduce: simplified data processing on large clusters[J]. Communications of the ACM(S0001-0782), 2008, 51(1): 107-113.
[3] Zaharia M, Chowdhury M, Das T, et al.Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing[C]//Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, USENIX Association, 2012: 2.
[4] Zaharia M, Chowdhury M, Das T, et al.Fast and interactive analytics over Hadoop data with Spark[J]. USENIX; login, 2012, 37(4): 45-51.
[5] Cover T, Hart, P.Nearest neighbor pattern classification[J]. IEEE Trans. Inf. Theory, 1967, 30(1): 21-27.
[6] 曲朝阳, 朱莉, 张士林. 基于Hadoop的广域测量系统数据处理[J]. 电力系统自动化, 2014, 37(4): 92-97.
QU Zhaoyang, ZHU Li, ZHANG Shilin.Data Processing of Hadoop-based Wide Area Measurement System[J]. Automation of Electric Power Systems, 2014, 37(4): 92-97.
[7] Christophe Bisciglia. The smart grid: Hadoop at the tennessee valley authority(TVA)[EB/OL] http://www. cloudera.com/blog/2009/06/smart-grid-hadoop-tennesseevalley-authority-tva/
[8] 齐林海, 艾明浩. 一种基于云计算的电压暂降并行计算方法[J]. 中国电机工程学报, 2014, 34(31):5493-5499.
QI Linhai, AI Minghao.A Voltage Sag Parallel Calculation Method Based on Cloud Computing[J]. Proceedings of the CSEE, 2014, 34(31):5493-5499.
[9] 张逸, 杨洪耕, 叶茂清. 基于分布式文件系统的海量电能质量监测数据管理方案[J]. 电力系统自动化, 2014, 38(2): 102-108.
Zhang Yi, YANG Honggeng, YE Maoqing.A Data Management Scheme for Massive Power Quality Monitoring Data Based on Distributed File System[J]. Automation of Electric Power Systems, 2014, 38(2): 102-108.
[10] Shvachko K, Kuang H, Radia S, et al.The hadoop distributed file system[C]//2010 IEEE 26th Symposium on Mass Storage Systems and Technologies. IEEE, 2010: 1-10.
[11] Wang B, Huang S, Qiu J, et al.Parallel online sequential extreme learning machine based on MapReduce[J]. NeuroComputing(S0925-2312), 2015, 149: 224-232.
[12] 王德文, 孙志伟. 电力用户侧大数据分析与并行负荷预测[J]. 中国电机工程学报, 2015 , 35(3): 527-537.
WANG Dewen, SUN Zhiwei.Big Data Analysis and Parallel Load Forecasting of Electric Power User Side[J]. Proceedings of the CSEE, 2015 , 35(3): 527-537.
[13] 曲广龙, 杨洪耕, 张逸. 采用Map-Reduce模型的海量电能质量数据交换格式文件快速解析方案[J]. 电网技术, 2014, 38(6): 1705-1711.
QU Guanglong, YANG Honggeng, ZHANG Yi.A Fast Parallel Parsing Scheme for Massive PQDIF Files With Map-Reduce Model[J]. Power System Technology, 2014, 38(6): 1705-1711.
[14] Fadika Z, Govindaraju M, Canon R, et al.Evaluating hadoop for data-intensive scientific operations[C]//2012 IEEE 5th International Conference on Cloud Computing (CLOUD)[C]. IEEE, 2012: 67-74.
[15] 王德文, 刘晓建. 基于MapReduce的电力设备并行故障诊断方法[J]. 电力自动化设备, 2014, 34(10): 116-120.
WANG Dewen, LIU Xiaojian.Parallel fault diagnosis based on MapReduce for electric power equipments[J]. Electric Power Automation Equipment, 2014, 34(10): 116-120.
[16] 王远, 陶烨, 袁军, 等. 一种基于HBase的智能电网时序大数据处理方法[J]. 系统仿真学报, 2016, 28(3): 559-568.
Wang Yuan, Tao Ye, Yuan Jun, et al.Approach to Process Smart Grid Time-Serial Big Data Base on HBase[J]. Journal of System Simulation, 2016, 28(3): 559-568.
[17] 邵士雯. 超高压线路绝缘子覆冰泄漏电流特性研究[D]. 保定: 华北电力大学, 2012.
Shao Shiwen.Research on Characteristics of Leakage Current of EHV Transmission Line leed Insulator Strings [D]. Baoding: North China Electric Power University, 2012.

泄漏电流数据的Spark-KNN并行模式识别方法

Parallel Pattern Recognition of Leak Current Data Using Spark-KNN

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	孙一铃, 陈谊, 单桂华, 李晓兴. 基于AR技术的多人互动地球仪系统[J]. 系统仿真学报, 2022, 34(6): 1367-1374.
[2]	刘东江, 黎建辉. 基于Spark的并行图聚类算法研究[J]. 系统仿真学报, 2020, 32(6): 1038-1050.
[3]	胡杨, 张小咏, 肖迪. 基于手机大数据的城市建筑物人口时空估算[J]. 系统仿真学报, 2020, 32(10): 1874-1883.
[4]	黄子赫, 高尚兵, 潘志庚, 惠浩, 廖麒羽, 赵锋锋. 基于快速密度聚类的载客热点可视化分析方法[J]. 系统仿真学报, 2019, 31(7): 1429-1438.
[5]	司光亚, 王飞, 刘洋. 基于仿真大数据的体系分析方法研究[J]. 系统仿真学报, 2019, 31(3): 511-519.
[6]	孔源, 杜怡曼, 吴建平, 胡可臻. 基于浮动车大数据的电动汽车潜在用户分析[J]. 系统仿真学报, 2019, 31(3): 575-584.
[7]	肖迪, 张小咏, 胡杨. 基于手机大数据的城市功能区识别方法[J]. 系统仿真学报, 2019, 31(11): 2281-2288.
[8]	张明智, 李义. Map Reduce框架下空间大数据的关联规则分析方法[J]. 系统仿真学报, 2018, 30(3): 840-845.
[9]	董泽, 尹二新. 基于状态跟踪的非线性工业系统全工况建模[J]. 系统仿真学报, 2018, 30(3): 846-856.
[10]	王子硕, 翟岩龙, 陶文俊, 杨浩, 张晗, 卿杜政. 高通量仿真中数据存储与访问技术研究[J]. 系统仿真学报, 2017, 29(9): 2016-2024.
[11]	邓丽, 余玥, 庞洪霖, 费敏锐. 一种基于并行聚类的温室天窗开度预测方法[J]. 系统仿真学报, 2017, 29(10): 2459-2467.
[12]	王刘旺, 朱永利, 贾亚飞. 一种多源海量局部放电信号脉冲的并行提取方法[J]. 系统仿真学报, 2017, 29(1): 57-66.
[13]	谈敦铭, 曹国廷, 郎娟芳, 杨朔. 飞行器大数据量CAD模型并行绘制[J]. 系统仿真学报, 2016, 28(9): 2049-2054.
[14]	韦容, 申希兵, 杨毅. SMT有界约束非集中自动机web服务模型检测[J]. 系统仿真学报, 2016, 28(9): 2283-2288.
[15]	万聪, 王翠荣, 王聪. 面向数据密集型工作流的QoS-aware调度算法[J]. 系统仿真学报, 2016, 28(3): 549-558.