The Scheduling Algorithm of Cloud Job Based on Hopfield Neural Network

doi:10.16182/j.issn1004731x.joss.19-FZ0323

Abstract

Abstract: Focusing on the low efficiency of cloud job scheduling and the insufficient utility of resource, a job scheduling algorithm based on Hopfield Neural Network is proposed. In order to improve the resource scheduling ability of the system, The resource characteristics which influence the cloud job scheduling are shown. The mathematical model of resource constraints is established, and the Hopfield energy function is designed and optimized. The average utilization rate of 9 nodes is analyzed by using the standard test cases, and the performance and resource utilization of the proposed strategy are compared with three typical algorithms. The results show that the average efficiency of the cloud job scheduling based on the algorithm is improved significantly.

Key words: Hadoop, cloud scheduling algorithm, Hopfield neural network, MapReduce, optimization algorithm

CLC Number:

TP391.9

Guo Yudong, Zuo Jinping. The Scheduling Algorithm of Cloud Job Based on Hopfield Neural Network[J]. Journal of System Simulation, 2019, 31(12): 2859-2867.

References

[1] Vaibhav P, Poonam S.How Heterogeneity Affects the Design of Hadoop MapReduce Schedulers: A State-of-the-Art Survey and Challenges[J]. Big Data (S2167-6461), 2018, 6(2): 72-95.
[2] Jafarnejad G E, Masoud R A, Nasih Q N.Load-balancing algorithms in cloud computing: A survey[J]. Journal of Network and Computer Applications (S1084- 8045), 2017, 88: 50-71.
[3] Nima K S, Jafari N N.MapReduce and its Applications, Challenges, and Architecture: a Comprehensive Review and Directions for Future Research[J]. Journal of Grid Computing (S1570-7873), 2017, 15(3): 295-321.
[4] Shivnath B.Towards automatic optimization of MapReduce programs[C]. Proceedings of the 1st ACM Symposium on Cloud Computing (SoCC 10), Indianapolis, USA, Jun 10- 11, 2010. New York, NY, USA: ACM, 2010: 137-142.
[5] Herodotou H, Lim H, Luo Gang, et al.Starfish: a self-tuning system for big data analytics[C]. Proceedings of the 5th Biennial Conference on Innovative Data Systems Research (CIDR 11), Asilomar, USA, Jan 9-12. 2011: 261-272.
[6] Herodotos H, Shivnath B.Profiling, what- if analysis, and cost-based optimization of MapReduce programs[C]. Proceedings of the 36th International Conference on Very Large Data Bases (VLDB 10), Singapore, Sep 13-17. 2010: 1111-1122.
[7] Tian C, Zhou H J, He Y Q, et al.A dynamic MapReduce scheduler for heterogeneous workloads[C]. Proceedings of the 8th International Conference on Grid and Cooperative Computing (GCC 09), Aug 27- 29, Lanzhou, China. 2009: 218-224.
[8] Jahani E, Cafarella M J, Re C.Automatic optimization for MapReduce programs[C]. Proceedings of the 37th International Conference on Very Large Data Bases (VLDB 11), Seattle, USA, Aug 29-Sep 3.2011: 385-396.
[9] Ahmad F, Chakradhar S T, Raghunathan A, et al.Tarazu: optimizing MapReduce on heterogeneous clusters[C]. Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 12), London, UK, May 3-7. 2012: 61-74.
[10] Polo J, Carrera D, Becerra Y, et al.Performance management of accelerated MapReduce workloads in heterogeneous clusters[C]. Proceedings of the 2010 39^th International Conference on Parallel Processing (ICPP 10), San Diego, USA, Sep 13- 16, 2010. Washington, DC. USA: IEEE Computer Society, 2010: 653-662.
[11] Fadika Z, Dede E, Hartog J, et al.MARLA: MapReduce for heterogeneous clusters[C]. Proceedings of the 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 12), Ottawa, Canada, May 13-16, 2012. Washington, DC. USA: IEEE Computer Society, 2012: 49-56.
[12] Zhuo T, Zhou J Q, Li K L, Li R X.A MapReduce task scheduling algorithm for deadline constraints[J]. Cluster Computing (S1386-7857), 2013, 16(4): 651-662.
[13] 马莉, 唐善成, 王静, 等. 云计算环境下的动态反馈作业调度算法[J]. 西安交通大学学报, 2014, 48(7): 77-82.
Ma Li, Tang Shancheng, Wang Jing, et al.Dynamic Feedback Job Scheduling Algorithm in Cloud Computing[J]. Journal of Xi’an Jiaotong University, 2014, 48(7): 77-82.
[14] Xu X Y, Tang M L, Tian Y C.QoS-guaranteed Resource Provisioning for Cloud-Based MapReduce in Dynamical Environments[J]. Future Generation Computer System (S0167-739X), 2017, 78: 18-30.
[15] Suresh S, Gopalan N P.An optimal task selection scheme for Hadoop scheduling[J]. IERI Procedia (S2212-6678), 2014, 10: 70-75.
[16] Rasooli A, Down D G, COSHH: A classification and optimization based scheduler for heterogeneous Hadoop systems[J]. Future Generation computer Systems (S0167-739X), 2014, (36): 1-15.
[17] Zhang F, Cao J, Li K, et al.Multi-objective scheduling of many tasks in cloud platforms[J]. Future Generation Computer Systems (S0167-739X), 2014, 37: 309-320.
[18] 夏家莉, 陈辉, 杨兵. 一种动态优先级实时任务调度算法[J]. 计算机学报, 2012, 35(12): 2685-2695.
Xia Jiali, Chen Hui, Yang Bin.A Real-Time Tasks Scheduling Algorithm Based on Dynamic Priority[J]. Chinese Journal of Computers, 2012, 35(12): 2685-2695.
[19] 宋杰, 徐澍, 郭朝鹏, 等. 一种优化MapReduce系统能耗的任务分发算法[J]. 计算机学报, 2016, 39(2): 323-338.
Song Jie, Xu Shu, Guo Chaopeng, et al.A Task Distribution Algorithm for Energy Consumption Optimization of Map Reduce System[J]. Chinese Journal of computers, 2016, 39(2): 323-338.
[20] Hopfield J J, Tank D W.Neural computation of decision in optimization problems[J]. Biol. Cybern (S0340-1200), 1985, 52: 141-152.
[21] 王万良, 吴启迪, 徐新黎. 基于Hopfield神经网络的作业车间生产调度方法[J]. 自动化学报, 2002, 28(5): 838-844.
Wang Wanliang, Wu Qidi, Xu Xinli.Hopfield Neural Network Approach for Job-Shop Scheduling Problems[J]. Acta Automatica Sinica, 2002, 28(5): 838-844.
[22] 王秀利, 吴惕华. 一种求解多处理器作业调度的Hopfield神经网络方法[J]. 系统工程与电子技术, 2002, 24(8): 13-16.
Wang Xiuli, Wu Tihua.Scheduling Mutiprocessor Job Using Hopfield Neural Network[J]. Systems Engineering and Electronics, 2002, 24(8): 13-16.
[23] 徐昇, 业宁, 朱发, 等. Hopfield网络在视差空间上的立体匹配求解[J]. 计算机研究与发展, 2013, 50(5): 1021-1029.
Xu Sheng, Ye Ning, Zhu Fa, et al.Solving the Stereo Matching in Disparity Image Space Using the Hopfield Neural Network[J]. Journal of Computer Research and Development, 2013, 50(5): 1021-1029.