Convergence analysis of online gradient method for BP neural networks

Neural Networks

Volumn 24, Issue 1, 2011, Pages 91-98

  Wu, Wei ^a   Wang, Jian ^a,b   Cheng, Mingsong ^a   Li, Zhengxue ^a  

^a DALIAN UNIVERSITY OF TECHNOLOGY   (China)

^b CHINA UNIVERSITY OF PETROLEUM   (China)

Author keywords

Backpropagation learning; Neural networks; Online gradient method; Strong convergence; Weak convergence

Indexed keywords

ACTIVATION FUNCTIONS; BACK-PROPAGATION NEURAL NETWORKS; BACKPROPAGATION LEARNING; BP NEURAL NETWORKS; CONVERGENCE ANALYSIS; CONVERGENCE RESULTS; ERROR FUNCTION; FIXED POINTS; GRADIENT LEARNING METHODS; HIDDEN LAYERS; HIDDEN NEURONS; LEARNING METHODS; LEARNING RATES; ONLINE GRADIENT METHOD; P-TYPE; POLYNOMIAL FUNCTIONS; SIGMOID FUNCTION; STOCHASTIC LEARNING; STOCHASTIC ORDER; STRONG CONVERGENCE; TRAINING SETS; WEAK AND STRONG CONVERGENCE; WEAK CONVERGENCE;

BACKPROPAGATION; GRADIENT METHODS; LEARNING SYSTEMS; STOCHASTIC SYSTEMS;

NEURAL NETWORKS;

ANALYTICAL ERROR; ARTICLE; ARTIFICIAL NEURAL NETWORK; BACK PROPAGATION; CONVERGENCE ANALYSIS; LEARNING; ONLINE SYSTEM; PRIORITY JOURNAL; STATISTICAL ANALYSIS; STOCHASTIC MODEL;

ALGORITHMS; COMPUTER SIMULATION; HUMANS; NEURAL NETWORKS (COMPUTER); ONLINE SYSTEMS; SIGNAL PROCESSING, COMPUTER-ASSISTED; STOCHASTIC PROCESSES; SYSTEMS THEORY;

EID: 78649659993     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neunet.2010.09.007     Document Type: Article

References (25)

1. Bertsekas D.P., Tsitsiklis J.N. Neuro-dynamic programming 1996, Athena Scientific.
2. Chakraborty D., Pal N.R. A novel training scheme for multilayered perceptrons to realize proper generalization and incremental learning. IEEE Transactions on Neural Networks 2003, 14:1-14.
3. Fine T.L., Mukherjee S. Parameter convergence and learning curves for neural networks. Neural Computation 1999, 11:747-769.
4. Finnoff W. Diffusion approximations for the constant learning rate backpropagation algorithm and resistance to local minima. Neural Computation 1994, 6:285-295.
5. Heskes T., Wiegerinck W. A theoretical comparison of batch-mode, on-line, cyclic, and almost-cyclic learning. IEEE Transactions on Neural Networks 1996, 7:919-925.
6. LeCun Y. Une procedure d'apprentissage pour reseau à seuil asymmetrique. A la Frontieredel'Intelligence Ariticielle des Sciences de la Connaissance des Neurosciences 1985, 85:599-604.
7. Li Z.X., Ding X.S. Prediction of stock market by bp neural networks with technical indexes as input. Numerical Mathematics: A Journal of Chinese Universities 2005, 27:373-377.
8. Li Z.X., Wu W., Tian Y.L. Convergence of an online gradient method for feedforward neural networks with stochastic inputs. Journal of Computational and Applied Mathematics 2004, 163:165-176.
9. Liang Y.C., Feng D.P., Lee H.P., Lim S.P., Lee K.H. Successive approximation training algorithm for feedforward neural networks. Neurocomputing 2002, 42:11-322.
10. Mangasarian O.L., Solodov M.V. Serial and parallel backpropagation convergence via nonmonotone perturbed minimization. Optimization Methods and Software 1994, 4:117-134.
11. Nakama T. Theoretical analysis of batch and on-line training for gradient descent learning in neural networks. Neurocomputing 2009, 73:151-159.
12. Parker, D. B. (1982). Learning-logic, invention report. Stanford University, Stanford, Calif.
13. Rumelhart D.E., Hinton G.E., Williams R.J. Learning representations by back-propagation errors. Nature 1986, 323:533-536.
14. Shao H.M., Wu W., Liu W.B. Convergence of BP algorithm for training MLP with linear output. Numerical Mathematics: A Journal of Chinese Universities (English Series) 2007, 16:193-202.
15. Tadic, V., & Stankovic, S. (2000). Learning in neural networks by normalized stochastic gradient algorithm: local convergence. In Proceedings of the 5th seminar neural networks application electronic engineering.
16. Terence D.S. Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Networks 1989, 2:459-473.
17. Werbos, P. J. (1974). Beyond regression: new tools for prediction and analysis in the behavioral sciences. Ph.D. thesis. Harvard University, Cambridge, MA.
18. Wilson D.R., Martinez T.R. The general inefficiency of batch training for gradient descent learning. Neural Networks 2003, 16:1429-1451.
19. Wu W., Xu Y.S. Deterministic convergence of an on-line gradient method for neural networks. Journal of Computational and Applied Mathematics 2002, 144:335-347.
20. Wu W., Feng G.R., Li Z.X., Xu Y.S. Deterministic convergence of an online gradient method for BP neural networks. IEEE Transactions on Neural Networks 2005, 16:533-540.
21. Wu W., Feng G.R., Li X. Training multilayer perceptrons via minimization of sum of ridge functions. Advances in Computational Mathematics 2002, 17:331-347.
22. Wu W., Shao Z.Q. Convergence of online gradient methods for continuous perceptrons with linearly separable training patterns. Applied Mathematics Letters 2003, 16:999-1002.
23. Wu, W., Shao, H. M., & Qu, D. (2005). Strong convergence of gradient methods for BP networks training. In Proceedings of 2005 international conference on neural networks and brains (pp. 332-334).
24. Xu Z.B., Zhang R., Jin W.F. When on-line BP training converges. IEEE Transactions on Neural Networks 2009, 20:1529-1539.
25. Zhang H.S., Wu W., Liu F., Yao M.C. Boundedness and convergence of online gradient method with penalty for feedforward neural networks. IEEE Transactions on Neural Networks 2009, 20:1050-1054.