Application of Distributed Clustering in Anomaly Detection of Farm Environment Data

doi:10.16182/j.issn1004731x.joss.201712014

Abstract

Abstract: The massive farm environment data stored in the distributed system should be dealt with so as to provide abnormal environment reference and make preventive strategies for crop yield. Considering the characteristics of the farm environment data, the Dirichlet Process Mixture Model (DPMM) clustering is implemented with the farm environment data on Hadoop and the anomaly detection method of the farm environment is proposed based on clustering analysis. Under the framework of MapReduce, Map stage implements the distribution of the sample points to the models; Reduce stage completes the update of models and the number of clusters. The performance has been verified by experiments. The results of clustering and the index of suitable environment for tomato are compared to implement the anomaly detection. The analysis results show that the method can be applied to anomaly detection of large number of farm environment data.

Key words: DPMM, distributed clustering, farm environment data, anomaly detection

CLC Number:

TP338.8

Deng Li, Pang Honglin, Ling Wang, Minrui Fei. Application of Distributed Clustering in Anomaly Detection of Farm Environment Data[J]. Journal of System Simulation, 2017, 29(12): 3035-3041.

References

[1] M. G Ananthara, Dr. T. Arunkumar, Ms. R. Hemavathy. CRY - An improved Crop Yield Prediction model using Bee Hive Clustering Approach for Agricultural data sets[C]//International Conference on Pattern Recognition, Informatics and Mobile Engineering, 2013. India: IEEE, 2013.
[2] 时雷, 马新明. 基于物联网的小麦生长环境数据采集与数据挖掘技术研究[D]. 河南: 河南农业大学, 2013: 61-64.
Shi Lei, Ma Xinming.Research on Data Collection and Data Mining of Wheat Growth Environment based on the Internet of Things[D]. Henan: Henan Agricultural University, 2013:61-64.
[3] 潘渊洋, 李光辉, 徐勇军. 基于DBSCAN的环境传感器网络异常数据检测方法[J]. 计算机应用与软件 2012, 29(11): 69-73.
Pan Yuanyang, Li Guanghui, Xu Yongjun.Abnormal Data Detection Method for Environment Wireless Sensor Networks Based on DBSCAN[J].Computer Applications and Software, 2012, 29(11): 69-73.
[4] 唐成华, 刘鹏程, 汤申生, 等.基于特征选择的模糊聚类异常入侵行为检测[J]. 计算机研究与发展, 2015, 52(3): 718-728.
Tang Chenghua, Liu Pengcheng, Tang Shensheng, et al.Anomaly Intrusion Behavior Detection Based on Fuzzy Clustering and Features Selection[J].Journal of Computer Research and Development, 2015, 52(3): 718-728.
[5] Fox E B, Choi D S, Willsky A S.Nonparametric Bayesian methods for large scale multi-target tracking[C]//Proceedings of the 40th Asilomar Conference on Signals Systems and Computers. Pacific Grove, USA: IEEE, 2006.
[6] Orbanz P, Buhmann J M.Nonparametric Bayesian image segmentation[J]. International Journal of Computer Vision (S0920-5691), 2008, 77(13): 25-45.
[7] 刘禅. 温室番茄生长期水分诊断研究[D]. 北京:中科院研究生院, 2011: 31-34.
Liu Chan.Research on Water Diagnosis of Greenhouse Tomato during Growth Period[D]. Beijing: Graduate University of Chinese Academy of Sciences, 2011: 31-34
[8] 何明胜. 基于GPU的Dirichlet算法并行计算设计与实现[D]. 北京: 北京邮电大学, 2012: 17-19.
He Mingsheng.Parallel Computing Design and Implementation for Dirichlet Algorithm Based on GPU[D]. Beijing: Beijing University of Posts and Telecommunications, 2012: 17-19.
[9] 周建英, 王飞跃, 曾大军. 分层Dirichlet过程及其应用综述[J]. 自动化学报, 2011, 37(4): 389-407.
Zhou Jianying, Wang Feiyue, Zeng Dajun.Hierarchical Dirichlet Processes and Their Applications: A Survey[J]. ACTA AUTOMATICA SINICA, 2011, 37(4): 389-407.
[10] Nicole L.B, Lishu L. Clustering digital forensic string search output[J]. Digital Investigation (S1742-2876). 2014, 11(4): 314-322.
[11] Lai Y, Zhongzhi S, Li D.Xu, et al. DH-TRIE Frequent Pattern Mining on Hadoop using JPA[C]. International Conference on Granular Computing, 2011. Taiwan: IEEE, 2011.
[12] Yonghong X, Aziguli W, Yantao W, et al.Implementation of Time Series Data Clustering Based on SVD for Stock Data Analysis on Hadoop Platform[C]//9th Conference on Industrial Electronics and Applications, 2011. China: IEEE, 2014.
[13] Wentao F, Nizar B.Infinite Dirichlet Mixture Model and Its Application via Variational Bayes[C]//10th International Conference on Machine Learning and Applications, 2011. USA: IEEE, 2011.
[14] 杨来, 史忠植, 梁帆, 等. 基于Hadoop云平台的并行数据挖掘方法[J]. 系统仿真学报 2013, 25(5): 936-944.
Yang Lai, Shi Zhongzhi, Liang Fan, et al.Parallel Approach in Data Mining Based on Hadoop Cloud Platform[J]. Journal of System Simulation, 2013, 25(5): 936-944.
[15] 林兰芬, 于鹏华, 李泽洋. 基于聚类的农产品流通物联网感知数据时空可视化技术[J]. 农业工程学报, 2015, 31(3): 228-235.
Lin Lanfen, Yu Penghua, Li Zeyang.Using clustering to visualize spatial-temporal internet of things data in process of agricultural product circulation[J]. Transactions of the Chinese Society of Agricultural Engineering, 2015, 31(3): 228-235.