Deterministic convergence of conjugate gradient method for feedforward neural networks

Neurocomputing

Volumn 74, Issue 14-15, 2011, Pages 2368-2376

  Wang, Jian ^a,b,c   Wu, Wei ^a   Zurada, Jacek M ^b  

^a DALIAN UNIVERSITY OF TECHNOLOGY   (China)

^b University of Louisville   (United States)

^c CHINA UNIVERSITY OF PETROLEUM   (China)

Author keywords

Backpropagation; Conjugate gradient; Deterministic convergence; Feedforward neural networks

Indexed keywords

BACK PROPAGATION NEURAL NETWORKS; BATCH MODES; CONJUGATE GRADIENT; CONVERGENCE PROPERTIES; CONVERGENCE RESULTS; DETERMINISTIC CONVERGENCE; ERROR FUNCTION; FAST CONVERGENCE; FEED-FORWARD; FIXED POINTS; LEARNING MODE; LEARNING RATES; LOW MEMORY; NUMERICAL EXAMPLE; NUMERICAL EXPERIMENTS; THREE-LAYER; WEAK AND STRONG CONVERGENCE;

BACKPROPAGATION; CONVERGENCE OF NUMERICAL METHODS; FEEDFORWARD NEURAL NETWORKS; LEARNING ALGORITHMS; NETWORK LAYERS; NUMERICAL ANALYSIS;

CONJUGATE GRADIENT METHOD;

ARTICLE; ARTIFICIAL NEURAL NETWORK; CONTROLLED STUDY; INTERMETHOD COMPARISON; LEARNING ALGORITHM; MATHEMATICAL ANALYSIS; MATHEMATICAL COMPUTING; PRIORITY JOURNAL;

EID: 79957959758     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2011.03.016     Document Type: Article

References (41)

1. Rumelhart D.E., Hinton G.E., Williams R.J. Learning representations by back-propagating errors. Nature 1986, 323:533-536.
2. Heskes T., Wiegerinck W. A theoretical comparison of batch-mode, on-line, cyclic, and almost-cyclic learning. IEEE Transactions on Neural Networks 1996, 7:919-925.
3. Wilson D.R., Martinez T.R. The general inefficiency of batch training for gradient descent learning. Neural Networks 2003, 16:1429-1451.
4. Xu Z.B., Zhang R., Jing W.F. When does online BP training converge?. IEEE Transactions on Neural Networks 2009, 20:1529-1539.
5. Wu W., Wang J., Cheng M.S., Li Z.X. Convergence analysis of online gradient method for BP neural networks. Neural Networks 2011, 24:91-98.
6. J. Wang, J. Yang, W. Wu, Convergence of cyclic and almost-cyclic learning with momentum for feedforward neural networks, IEEE Transactions on Neural Networks, submitted for publication.
7. Park D.C., Elsharkawi M.A., Marks R.J., Atlas L.E., Damborg M.J. Electric-load forecasting using an artificial neural network. IEEE Transactions on Power Systems 1991, 6:442-449.
8. Saini L.M., Soni M.K. Artificial neural network-based peak load forecasting using conjugate gradient methods. IEEE Transactions on Power Systems 2002, 17:907-912.
9. Lu C.N., Wu H.T., Vemuri S. Neural network based short-term load forecasting. IEEE Transactions on Power Systems 1993, 8:336-342.
10. Papalexopoulos A.D., Hao S.Y., Peng T.M. An implementation of a neural-network-based load forecasting-model for the EMS. IEEE Transactions on Power Systems 1994, 9:1956-1962.
11. Goodband J.H., Haas O.C.L., Mills J.A. A comparison of neural network approaches for on-line prediction in IGRT. Medical Physics 2008, 35:1113-1122.
12. Hagan M.T., Demuth H.B., Beale M.H. Neural Network Design 1996, PWS Publisher, Boston.
13. Nocedal J., Wright S.J. Numerical Optimization 2006, Springer, New York.
14. M.R. Hestenes, E.L. Stiefel, Method of Conjugate Gradients for Solving Linear Systems, National Bureau of Standards, Washington, 1952.
15. Fletcher R., Reeves C.M. Function minimization by conjugate gradients. The Computer Journal 1964, 7:149-154.
16. Dai Y.H., Yuan Y.X. Nonlinear Conjugate Gradient Methods 2000, Shanghai Scientific, Technical Publishers, Shanghai.
17. Gonzalez A., Dorronsoro J.R. Natural conjugate gradient training of multilayer perceptrons. Neurocomputing 2008, 71:2499-2506.
18. Polak E., Ribiere G. Note sur la convergence de directions conjugates. Revue Francaise d'Informatique et de Recherche Operationnelle 1969, 16:94-112.
19. Polyak B.T. The conjugate gradient method in extremal problems. USSR Computational Mathematics and Mathematical Physics 1969, 9:94-112.
20. Shi Z.J., Shen J. Convergence of the Polak-Ribiere-Polyak conjugate gradient method. Nonlinear Analysis-Theory 2007, 66:1428-1441.
21. Jiang M.H., Gielen G., Zhang B., Luo Z.S. Fast learning algorithms for feedforward neural networks. Applied Intelligence 2003, 18:37-54.
22. A. Cichocki, B. Orsier, A. Back, S.I. Amari, On-line adaptive algorithms in non-stationary environments using a modified conjugate gradient approach, in: Neural Networks for Signal Processing [1997] VII. Proceedings of the 1997 IEEE Workshop, 1997, pp. 316-325.
23. Shen X.Z., Shi X.Z., Meng G. Online algorithm of blind source separation based on conjugate gradient method. Circuits, Systems, and Signal Processing 2006, 25:381-388.
24. Liu Q.S., Dang C.Y., Cao J.D. A novel recurrent neural network with one neuron and finite-time convergence k-winners-take-all operation. IEEE Transactions on Neural Networks 2010, 21:1140-1148.
25. Liu Q., Cao J., Chen G. A novel recurrent neural network with finite-time convergence for linear programming. Neural Computation 2010, 22:2962-2978.
26. Huang G.B., Zhu Q.Y., Siew C.K. Extreme learning machine: theory and applications. Neurocomputing 2006, 70:489-501.
27. Han F., Huang D.S. Improved extreme learning machine for function approximation by encoding a priori information. Neurocomputing 2006, 69:2369-2373.
28. Huang G.B., Ding X., Zhou H. Optimization method based extreme learning machine for classification. Neurocomputing 2010, 74:155-163.
29. Xu D.P., Zhang H.S., Liu L.J. Convergence analysis of three classes of split-complex gradient algorithms for complex-valued recurrent neural networks. Neural Computation 2010, 22:2655-2677.
30. Sanger T.D. Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Networks 1989, 2:459-473.
31. Finnoff W. Diffusion approximations for the constant learning rate backpropagation algorithm and resistance to local minima. Neural Computation 1994, 6:285-295.
32. Chakraborty D., Pal N.R. A novel training scheme for multilayered perceptrons to realize proper generalization and incremental learning. IEEE Transactions on Neural Networks 2003, 14:1-14.
33. Zhang H.S., Wu W., Liu F., Yao M.C. Boundedness and convergence of online gradient method with penalty for feedforward neural networks. IEEE Transactions on Neural Networks 2009, 20:1050-1054.
34. Wu W., Feng G.R., Li Z.X., Xu Y.S. Deterministic convergence of an online gradient method for BP neural networks. IEEE Transactions on Neural Networks 2005, 16:533-540.
35. Li Z.X., Wu W., Tian Y.L. Convergence of an online gradient method for feedforward neural networks with stochastic inputs. Journal of Computational and Applied Mathematics 2004, 163:165-176.
36. Grippo L., Lucidi S. A globally convergent version of the Polak-Ribiere conjugate gradient method. Mathematics Programming 1997, 78:375-391.
37. Khoda K.M., Liu Y., Storey C. Contributed papers generalized Polak-Ribiere algorithm. Journal of Optimization Theory and Applications 1992, 75:345-354.
38. Nocedal J. Theory of algorithms for unconstrained optimization. Acta Numerica 1992, 1:199-242.
39. Shi Z.J. Restricted PR conjugate gradient method and its global convergence. Advances in Mathematics 2002, 1:47-55.
40. Shi Z.J., Guo J.H. A new algorithm of nonlinear conjugate gradient method with strong convergence. Computational and Applied Mathematics 2008, 27:93-106.
41. Gorman R.P., Sejnowski T.J. Analysis of hidden units in a layered network trained to classify sonar targets. Neural Networks 1988, 1:75-89.