Curvature-based BP Algorithm Optimization and Its Application in FNN

doi:10.16182/j.issn1004731x.joss.17-0461

Abstract

Abstract: In order to improve the optimization efficiency of BP algorithm affected by the selection of step size, a step size optimization BP algorithm based on curvature information is proposed and applied to the training process of FNN (Fuzzy Neural Network). Reference to Newton's method, The gradient of the cost function and the curvature information in the direction are calculated to determine the direction and magnitude of the parameter adjustment in each iteration. This method only needs to consider the two order information of the gradient direction, so it does not need the storage and processing of Hessian matrix. The effectiveness and efficiency of the proposed method are verified by a numerical simulation and data simulation of blast furnace ironmaking process.

Key words: Step size optimization, BP algorithm, Fuzzy neural network, Learning speed

CLC Number:

TP183

Xiong Weili, Sun Wenxin, Shi Xudong. Curvature-based BP Algorithm Optimization and Its Application in FNN[J]. Journal of System Simulation, 2020, 32(1): 1-8.

References

[1] 王海军. 带自适应步长神经网络学习算法的收敛性分析[D]. 大连: 大连理工大学, 2016.
Wang Haijun.Convergence Analysis of Neural Network Learning Algorithm with Adaptive Learning Rate[D]. Dalian: Dalian University of Technology, 2016.
[2] LeCun Y A, Bottou L, Orr G B, et al. Efficient backprop[M]. Neural networks: Tricks of the trade. Springer Berlin Heidelberg, 2012: 9-48.
[3] Nesterov Y.Gradient methods for minimizing composite functions[J]. Mathematical Programming (S0025-5610), 2013, 140(1): 125-161.
[4] 房振勇, 游文虎, 冯汝鹏. 改进BP算法在模糊神经网络中的应用[J]. 北京航空航天大学学报, 2007, 33(11): 1321-1324.
Fang Zhenyong, You Wenhu, Feng Rupeng.Application of improved BP algorithm in fuzzy neural networks[J]. Journal of Beijing University of Aeronautics and Astronautics, 2007, 33(11): 1321-1324.
[5] 高雪鹏, 丛爽. BP网络改进算法的性能对比研究[J]. 控制与决策, 2001, 16(2): 167-171.
Gao Xuepeng, Cong Shuang.Comparative Study on Fast Learning Algorithms of BP Networks[J]. Control and Decision, 2001, 16(2): 167-171.
[6] Hadgu A T, Nigam A, Diaz-Aviles E.Large-scale learning with AdaGrad on Spark[C]// IEEE International Conference on Big Data. Beijing, China: IEEE, 2015: 2828-2830.
[7] Goodfellow I, Bengio Y, Courville A.Deep learning[M]. Massachusetts, USA: MIT Press, 2016.
[8] Mukkamala M C, Hein M.Variants of RMSProp and Adagrad with logarithmic regret bounds[C]// Proceedings of the 34th International Conference on Machine Learning, PMLR 70. Sydney, Australia: ICML, 2017: 2545-2553.
[9] Wang L X.A course in fuzzy systems and control[M]. Englewood, USA: Prentice-Hall, Inc. 1996.
[10] Hecht-Nielsen.Theory of the backpropagation neural network[C]// International Joint Conference on Neural Networks. Honolulu, HI, USA: IEEE, 2002(1): 593-605.
[11] 董海鹰, 魏占宏, 杨玺, 等. 基于自适应模糊神经网络的风速软测量[J]. 电力系统及其自动化学报, 2013, 25(1): 60-65.
Dong Haiying, Wei Zhanhong, Yang Xi, et al.Wind Speed Soft Sensor Based on Adaptive Fuzzy Neural Network[J]. Proceedings of the CSU-EPSA, 2013, 25(1): 60-65.
[12] Pratama M, Er M J, Li X, et al.Data driven modeling based on dynamic parsimonious fuzzy neural network[J]. Neurocomputing (S0925-2312), 2013, 110: 18-28.
[13] Zhang M, Liu X.A soft sensor based on adaptive fuzzy neural network and support vector regression for industrial melt index prediction[J]. Chemometrics & Intelligent Laboratory Systems (S0169-7439), 2013, 126: 83-90.
[14] 李国勇, 杨丽娟. 神经·模糊·预测控制及其MATLAB实现. [M]. 3版. 北京: 电子工业出版社, 2013: 257-260.
Li Guoyong, Yang Lijuan. Neural Netowrk.Fuzzy. Predictive Control and its MATLAB implementation [M]. 3 Edition. Beijing: Publishing House of Electronics Industry, 2013: 257-260.
[15] 丛爽. 几种模糊神经网络系统关系的对比研究[J]. 信息与控制, 2001, 30(6): 486-491.
Cong Shuang.Comparative Research on Relationships between Several Fuzzy-Neural Network Systems[J]. Information and Control, 2001, 30(6): 486-491.
[16] 李炯城, 肖恒辉, 李桂愉. 高效的线搜索寻优方法[J]. 计算机工程与应用, 2013, 49(3): 53-56.
Li Jiongcheng, Xiao Henghui, Li Guiyu.Efficient line search optimization method[J]. Computer Engineering and Applications, 2013, 49(3): 53-56.
[17] 蒋朝辉, 董梦林, 桂卫华, 等. 基于Bootstrap的高炉铁水硅含量二维预报[J]. 自动化学报, 2015, 42(5): 715-723.
Jiang Zhaohui, Dong Menglin, Gui Weihua, et al.Two-dimensional Prediction for Silicon Content of Hot Metal of Blast Furnace Based on Bootstrap[J]. Acta Automatica Sinica, 2015, 42(5): 715-723.
[18] Mika S, Lkopf B, Smola A, et al.Kernel PCA and de-noising in feature spaces[C]// Conference on Advances in Neural Information Processing Systems. Marriott, Denver, USA: MIT Press, 1999: 536-542.