Convergence of an online gradient method with inner-product penalty and adaptive momentum

Neurocomputing

Volumn 77, Issue 1, 2012, Pages 243-252

  Shao, Hongmei ^a   Xu, Dongpo ^b   Zheng, Gaofeng ^c   Liu, Lijun ^d  

^a CHINA UNIVERSITY OF PETROLEUM   (China)

^b HARBIN ENGINEERING UNIVERSITY   (China)

^c RAKUTEN INC   (Japan)

^d DALIAN NATIONALITIES UNIVERSITY   (China)

Author keywords

Adaptive momentum; Convergence; Feedforward neural network; Inner product penalty; Online gradient method

Indexed keywords

ADAPTIVE MOMENTUM; CONVERGENCE; CONVERGENCE THEOREM; GABOR FUNCTION; INNER PRODUCT; ONLINE GRADIENT METHOD; SUFFICIENT CONDITIONS; THREE-LAYER NEURAL NETWORKS; TRAINING SAMPLE; TWO LAYERS; TWO-SPIRAL PROBLEM; WEAK AND STRONG CONVERGENCE;

GRADIENT METHODS; NETWORK LAYERS;

FEEDFORWARD NEURAL NETWORKS;

ARTICLE; ARTIFICIAL NEURAL NETWORK; CONTROLLED STUDY; FEEDFORWARD NEURAL NETWORK; LEARNING ALGORITHM; MATHEMATICAL ANALYSIS; MATHEMATICAL COMPUTING; ONLINE ANALYTICAL PROCESSING; ONLINE GRADIENT METHOD; ONLINE SYSTEM; PRIORITY JOURNAL; STOCHASTIC MODEL; THEORETICAL STUDY; TRAINING;

EID: 80955130842     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2011.09.003     Document Type: Article

References (31)

1. Rumelhart D.E., McClelland J.L. Parallel Distributed Processing-Explorations in the Microstructure of Cognition 1986, the PDP Research Group, MIT Press, MA.
2. Haykin S. Neural Networks: A Comprehensive Foundation 2001, Tsinghua University Press, Prentice Hall, Beijing. 2nd ed.
3. Wu W., Xu Y.S. Deterministic convergence of an online gradient method for neural networks. J. Comput. Appl. Math. 2002, 144:335-347.
4. Wu W., Feng G.R., Li X. Training multilayer perceptrons via minimization of sum of ridge functions. Adv. Comput. Math. 2002, 17:331-347.
5. P.L. Bartlett, For valid generalization, the size of the weights is more important than the size of the network, in: Advances in Neural Information Processing Systems, vol. 9, 1997, pp. 134-140.
6. Geman S., Bienenstock E., Doursat R. Neural networks and the bias/variance dilemma. Neural Comput. 1992, 4:1-58.
7. Hinton G.E. Connectionist learning procedures. Artif. Intell. 1989, 40:185-234.
8. Reed R. Pruning algorithms-a survey. IEEE Trans. Neural Networks 1997, 8:185-204.
9. Setiono R. A penalty-function approach for pruning feedforward neural networks. Neural Networks 1997, 9:185-204.
10. Ishikawa M. Structural learning with forgetting. Neural Networks 1996, 9(3):509-521.
11. Zhang H.S., Wu W. Boundedness and convergence of online gradient method with penalty for linear output feedforward neural networks. Neural Process. Lett. 2009, 29:205-212.
12. Kong J., Wu W. Online gradient methods with a punishing term for neural networks. Northeast. Math. 2001, 173:371-378.
13. Zhang L.Q., Wu W. Online gradient methods with a penalty term for neural networks with large training set. J. Nonlinear Dyn. Sci. Technol. 2004, 11:53-58.
14. Li Z.X., Wu W., Tian Y.L. Convergence of an online gradient method for feedforword neural networks with stochastic inputs. J. Comput. Appl. Math. 2004, 163(1):165-176.
15. Shao H.M., Wu W., Li F. Convergence of online gradient method with a penalty term for feedforward neural networks with stochastic inputs. Numer. Math. J. Chin. Universities Eng. Ser. 2005, 14(10):87-96.
16. Chan L.W., Fallside F. An adaptive training algorithm for backpropagation networks. Comput. Speech Lang. 1987, 2:205-218.
17. Qiu G., Varley M.R., Terrell T.J. Accelerated training of backpropagation networks by using adaptive momentum step. IEE Electron. Lett. 1992, 28(4):377-379.
18. Istook E., Martinez T. Improved backpropagation learning in neural networks with windowed momentum. Int. J. Neural Syst. 2002, 12(3-4):303-318.
19. Torii M., Hagan M.T. Stability of steepest descent with momentum for quadratic functions. IEEE Trans. Neural Networks 2002, 13(3):752-756.
20. Bhaya A., Kaszkurewicz E. Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method. Neural Networks 2004, 17:65-71.
21. Z.G. Zeng, Analysis of Global Convergence and Learning Parameters of the Back-propagation Algorithm for Quadratic Functions, Lecture Notes in Computer Science, vol. 4682, 2007, pp. 7-13.
22. Zhang N.M., Wu W., Zheng G.F. Convergence of gradient method with momentum for two-layer feedforward neural networks. IEEE Trans. Neural Networks 2006, 17(2):522-525.
23. N.M. Zhang, Deterministic Convergence of an Online Gradient Method with Momentum, Lecture Notes in Computer Science, vol. 4113, 2006, pp. 94-105.
24. Zhang N.M. An online gradient method with momentum for two-layer feedforward neural networks. Appl. Math. Comput. 2009, 212:488-498.
25. Wu W., Feng G.R., Li Z.X., Xu Y.S. Convergence of an online gradient method for BP neural networks. IEEE Trans. Neural Networks 2005, 16(3):533-540.
26. W. Wu, H.M. Shao, Z.X. Li, Convergence of Batch BP Algorithm with Penalty for FNN Training, Lecture Notes in Computer Science, vol. 4232, 2006, pp. 562-569.
27. Wu W., et al. Convergence of gradient method with momentum for back-propagation neural networks. J. Comput. Math. 2008, 26(4):613-623.
28. Xu D.P., Zhang H.S., Liu L.J. Convergence analysis of three classes of split-complex gradient algorithms for complex-valued recurrent neural networks. Neural Comput. 2010, 22(10):2655-2677.
29. Yuan Y.X., Sun W.Y. Optimization Theory and Methods 2001, Science Press, Beijing.
30. Wu W., et al. A modified gradient-based neuro-fuzzy learning algorithm and its convergence. Inf. Sci. 2010, 180(9):1630-1642.
31. Kathirvalavakumar T., JeyaseeliSubavathi S. Neighborhood based modified backpropagation algorithm using adaptive learning parameters for training feedforward neural networks. Neurocomputing 2009, 72:3915-3921.