分布式聚类在农场环境数据异常检测中的应用

doi:10.16182/j.issn1004731x.joss.201712014

系统仿真学报 ›› 2017, Vol. 29 ›› Issue (12): 3035-3041.doi: 10.16182/j.issn1004731x.joss.201712014

分布式聚类在农场环境数据异常检测中的应用

邓丽^1,2, 庞洪霖^1,2, 王灵^1,2, 费敏锐^1,2

1.上海大学机电工程与自动化学院,上海 200072;
2.上海市电站自动化技术重点实验室,上海 200072

收稿日期:2015-10-16 发布日期:2020-06-06
作者简介:邓丽(1978-),女,安徽,博士后,副教授,研究方向为机器学习、智能优化算法等;庞洪霖(1991-),男,江苏,硕士生,研究方向为机器学习与分布式计算。
基金资助:
上海市科委重点项目(14DZ1206302)

Application of Distributed Clustering in Anomaly Detection of Farm Environment Data

Deng Li^1,2, Pang Honglin^1,2, Ling Wang^1,2, Minrui Fei^1,2

1. School of Mechatronics Engineering and Automation, Shanghai University, Shanghai 200072, China;
2. Shanghai Key Laboratory of Power Station Automation Technology, Shanghai 200072, China

Received:2015-10-16 Published:2020-06-06

摘要/Abstract

摘要： 为了处理大量分布式存储的农场环境数据,为作物增产提供异常环境参考并制定预防策略,本文结合农场环境数据的特点,在Hadoop平台中实现了对农场环境数据的Dirichlet过程混合模型聚类,并提出了基于聚类分析的农场环境异常检测方法。在MapReduce框架下,Map阶段完成样本点到模型的分配;Reduce阶段对模型与类簇个数进行更新。通过实验验证了分布式Dirichlet聚类的性能,分析结果表明该方法可以应用于大量农场环境数据的异常检测。

关键词: Dirichlet过程混合模型, 分布式聚类, 农场环境数据, 异常检测

Abstract: The massive farm environment data stored in the distributed system should be dealt with so as to provide abnormal environment reference and make preventive strategies for crop yield. Considering the characteristics of the farm environment data, the Dirichlet Process Mixture Model (DPMM) clustering is implemented with the farm environment data on Hadoop and the anomaly detection method of the farm environment is proposed based on clustering analysis. Under the framework of MapReduce, Map stage implements the distribution of the sample points to the models; Reduce stage completes the update of models and the number of clusters. The performance has been verified by experiments. The results of clustering and the index of suitable environment for tomato are compared to implement the anomaly detection. The analysis results show that the method can be applied to anomaly detection of large number of farm environment data.

Key words: DPMM, distributed clustering, farm environment data, anomaly detection

中图分类号:

TP338.8

邓丽, 庞洪霖, 王灵, 费敏锐. 分布式聚类在农场环境数据异常检测中的应用[J]. 系统仿真学报, 2017, 29(12): 3035-3041.

Deng Li, Pang Honglin, Ling Wang, Minrui Fei. Application of Distributed Clustering in Anomaly Detection of Farm Environment Data[J]. Journal of System Simulation, 2017, 29(12): 3035-3041.

参考文献

[1] M. G Ananthara, Dr. T. Arunkumar, Ms. R. Hemavathy. CRY - An improved Crop Yield Prediction model using Bee Hive Clustering Approach for Agricultural data sets[C]//International Conference on Pattern Recognition, Informatics and Mobile Engineering, 2013. India: IEEE, 2013.
[2] 时雷, 马新明. 基于物联网的小麦生长环境数据采集与数据挖掘技术研究[D]. 河南: 河南农业大学, 2013: 61-64.
Shi Lei, Ma Xinming.Research on Data Collection and Data Mining of Wheat Growth Environment based on the Internet of Things[D]. Henan: Henan Agricultural University, 2013:61-64.
[3] 潘渊洋, 李光辉, 徐勇军. 基于DBSCAN的环境传感器网络异常数据检测方法[J]. 计算机应用与软件 2012, 29(11): 69-73.
Pan Yuanyang, Li Guanghui, Xu Yongjun.Abnormal Data Detection Method for Environment Wireless Sensor Networks Based on DBSCAN[J].Computer Applications and Software, 2012, 29(11): 69-73.
[4] 唐成华, 刘鹏程, 汤申生, 等.基于特征选择的模糊聚类异常入侵行为检测[J]. 计算机研究与发展, 2015, 52(3): 718-728.
Tang Chenghua, Liu Pengcheng, Tang Shensheng, et al.Anomaly Intrusion Behavior Detection Based on Fuzzy Clustering and Features Selection[J].Journal of Computer Research and Development, 2015, 52(3): 718-728.
[5] Fox E B, Choi D S, Willsky A S.Nonparametric Bayesian methods for large scale multi-target tracking[C]//Proceedings of the 40th Asilomar Conference on Signals Systems and Computers. Pacific Grove, USA: IEEE, 2006.
[6] Orbanz P, Buhmann J M.Nonparametric Bayesian image segmentation[J]. International Journal of Computer Vision (S0920-5691), 2008, 77(13): 25-45.
[7] 刘禅. 温室番茄生长期水分诊断研究[D]. 北京:中科院研究生院, 2011: 31-34.
Liu Chan.Research on Water Diagnosis of Greenhouse Tomato during Growth Period[D]. Beijing: Graduate University of Chinese Academy of Sciences, 2011: 31-34
[8] 何明胜. 基于GPU的Dirichlet算法并行计算设计与实现[D]. 北京: 北京邮电大学, 2012: 17-19.
He Mingsheng.Parallel Computing Design and Implementation for Dirichlet Algorithm Based on GPU[D]. Beijing: Beijing University of Posts and Telecommunications, 2012: 17-19.
[9] 周建英, 王飞跃, 曾大军. 分层Dirichlet过程及其应用综述[J]. 自动化学报, 2011, 37(4): 389-407.
Zhou Jianying, Wang Feiyue, Zeng Dajun.Hierarchical Dirichlet Processes and Their Applications: A Survey[J]. ACTA AUTOMATICA SINICA, 2011, 37(4): 389-407.
[10] Nicole L.B, Lishu L. Clustering digital forensic string search output[J]. Digital Investigation (S1742-2876). 2014, 11(4): 314-322.
[11] Lai Y, Zhongzhi S, Li D.Xu, et al. DH-TRIE Frequent Pattern Mining on Hadoop using JPA[C]. International Conference on Granular Computing, 2011. Taiwan: IEEE, 2011.
[12] Yonghong X, Aziguli W, Yantao W, et al.Implementation of Time Series Data Clustering Based on SVD for Stock Data Analysis on Hadoop Platform[C]//9th Conference on Industrial Electronics and Applications, 2011. China: IEEE, 2014.
[13] Wentao F, Nizar B.Infinite Dirichlet Mixture Model and Its Application via Variational Bayes[C]//10th International Conference on Machine Learning and Applications, 2011. USA: IEEE, 2011.
[14] 杨来, 史忠植, 梁帆, 等. 基于Hadoop云平台的并行数据挖掘方法[J]. 系统仿真学报 2013, 25(5): 936-944.
Yang Lai, Shi Zhongzhi, Liang Fan, et al.Parallel Approach in Data Mining Based on Hadoop Cloud Platform[J]. Journal of System Simulation, 2013, 25(5): 936-944.
[15] 林兰芬, 于鹏华, 李泽洋. 基于聚类的农产品流通物联网感知数据时空可视化技术[J]. 农业工程学报, 2015, 31(3): 228-235.
Lin Lanfen, Yu Penghua, Li Zeyang.Using clustering to visualize spatial-temporal internet of things data in process of agricultural product circulation[J]. Transactions of the Chinese Society of Agricultural Engineering, 2015, 31(3): 228-235.

分布式聚类在农场环境数据异常检测中的应用

Application of Distributed Clustering in Anomaly Detection of Farm Environment Data

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 5

编辑推荐

Metrics

本文评价

[1]	秦婉亭, 老松杨, 汤俊, 卢聪. 基于变分自编码器的飓风轨迹异常检测方法[J]. 系统仿真学报, 2021, 33(9): 2191-2201.
[2]	董书琴, 张斌. 面向不平衡数据的网络流量异常检测方法[J]. 系统仿真学报, 2021, 33(3): 679-689.
[3]	刘轩, 李向阳, 何芳, 赵建伟, 张峰干. 基于空谱特征融合的高光谱RX异常检测方法[J]. 系统仿真学报, 2021, 33(12): 2891-2900.
[4]	宋晓瑞, 邹玲, 吴玲达, 徐万朋. 基于背景重建的高光谱图像异常检测[J]. 系统仿真学报, 2020, 32(7): 1287-1293.
[5]	邓丽, 余玥, 庞洪霖, 费敏锐. 一种基于并行聚类的温室天窗开度预测方法[J]. 系统仿真学报, 2017, 29(10): 2459-2467.