Boundedness and convergence of online gradient method with penalty and momentum

Neurocomputing

Volumn 74, Issue 5, 2011, Pages 765-770

  Shao, Hongmei ^a   Zheng, Gaofeng ^b  

^a CHINA UNIVERSITY OF PETROLEUM   (China)

^b JANA Solutions Inc   (Japan)

Author keywords

Boundedness; Convergence; Feedforward neural network; Momentum; Online gradient method; Penalty

Indexed keywords

BOUNDEDNESS; CONVERGENCE; ERROR FUNCTION; MONOTONICITY; ONLINE GRADIENT METHOD; PENALTY; PENALTY TERM; SUFFICIENT CONDITIONS; TRAINING PROCESS; TWO LAYERS; UNIFORMLY BOUNDED; WEAK AND STRONG CONVERGENCE;

GRADIENT METHODS; NETWORK LAYERS;

FEEDFORWARD NEURAL NETWORKS;

ARTICLE; ARTIFICIAL NEURAL NETWORK; ERROR; LEARNING ALGORITHM; ONLINE GRADIENT METHOD; PRIORITY JOURNAL; THEORETICAL MODEL;

EID: 78650719006     PISSN: 09252312     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neucom.2010.10.005     Document Type: Article

References (17)

1. Rumelhart D.E., McClelland J.L. Parallel Distributed Processing-Explorations in the Microstructure of Cognition 1986, PDP Research Group, MIT Press, MA.
2. Haykin S. Neural Networks: A Comprehensive Foundation 2001, Tsinghua University Press, Prentice Hall, Beijing. second ed.
3. Wu W., Xu Y.S. Deterministic convergence of an online gradient method for neural networks. Journal of Computational and Applied Mathematics 2002, 144:335-347.
4. Wu W., Feng G.R., Li X. Training multilayer perceptrons via minimization of sum of ridge functions. Advances in Computational Mathematics 2002, 17:331-347.
5. Torii M., Hagan M.T. Stability of steepest descent with momentum for quadratic functions. IEEE Transactions on Neural Networks 2002, 13(3):752-756.
6. Bhaya A., Kaszkurewicz E. Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method. Neural Networks 2004, 17:65-71.
7. Zeng Z.G. Analysis of global convergence and learning parameters of the back-propagation algorithm for quadratic functions. Lecture Notes in Computer Science 2007, vol. 4682:7-13.
8. Zhang N.M., Wu W., Zheng G.F. Convergence of gradient method with momentum for two-layer feedforward neural networks. IEEE Transactions on Neural Networks 2006, 17(2):522-525.
9. Zhang N.M. Deterministic convergence of an online gradient method with momentum. Lecture Notes in Computer Science 2007, vol. 4113:94-105.
10. Zhang N.M. An online gradient method with momentum for two-layer feedforward neural networks. Applied Mathematics and Computation 2009, 212:488-498.
11. Bartlett P.L. For valid generalization, the size of the weights is more important than the size of the network. Advances in Neural Information Processing Systems 1997, 9:134-140.
12. Geman S., Bienenstock E., Doursat R. Neural networks and the bias/variance dilemma. Neural Computation 1992, 4:1-58.
13. Hinton G.E. Connectionist learning procedures. Artificial Intelligence 1989, 40:185-234.
14. Reed R. Pruning algorithms-a survey. IEEE Transactions on Neural Networks 1997, 8:185-204.
15. Setiono R. A penalty-function approach for pruning feedforward neural networks. Neural Networks 1997, 9:185-204.
16. Zhang H.S., Wu W. Boundedness and convergence of online gradient method with penalty for linear output feedforward neural networks. Neural Process Letters 2009, 29:205-212.
17. Yuan Y.X., Sun W.Y. Optimization Theory and Methods 2001, Science Press, Beijing.