Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

Neural Networks

Volumn 17, Issue 1, 2004, Pages 65-71

  Bhaya, Amit ^a   Kaszkurewicz, Eugenius ^a  

^a FEDERAL UNIVERSITY OF RIO DE JANEIRO   (Brazil)

Author keywords

Backpropagation; Bilinear system; Conjugate gradient algorithm; Continuous optimization; Control Liapunov function; Convergence; Momentum; Steepest descent

Indexed keywords

BACKPROPAGATION; FRICTION; LYAPUNOV METHODS; METHOD OF MOMENTS; PARAMETER ESTIMATION; QUADRATIC PROGRAMMING;

LEARNING RATES;

NEURAL NETWORKS;

ACCELERATION; ALGORITHM; ANALYTIC METHOD; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; COMPARATIVE STUDY; COMPUTER SIMULATION; FEEDBACK SYSTEM; FRICTION; LEARNING; MOLECULAR DYNAMICS; NONLINEAR SYSTEM; PRIORITY JOURNAL;

EID: 0346881152     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0893-6080(03)00170-9     Document Type: Article

References (23)

1. Alber Y.I. Continuous processes of the Newton type. Differential Equations. 7:(11):1971;1461-1471.
2. Attouch H., Goudou X., Redont P. The heavy ball with friction method, I. The continuous dynamical system: global exploration of the global minima of a real-valued function by asymptotic analysis of a dissipative dynamical system. Communications in Contemporary Mathematics. 2:(1):2000;1-34.
3. Bhaya, A., & Kaszkurewicz, E (2002). Iterative methods as dynamical systems with feedback control. Technical report NACAD/BK/01-02, Department of Electrical Engineering, Federal University of Rio de Janeiro, COPPE/UFRJ.
4. Chu M.T. On the continuous realization of iterative processes. SIAM Review. 30:(3):1988;375-387.
5. Chu M.T. Matrix differential equations: a continuous realization process for linear algebra problems. Nonlinear Analysis Theory, Methods and Applications. 18:(12):1992;1125-1146.
6. Datta B.N., Rincon F. Feedback stabilization of a second-order system: a nonmodal approach. Linear Algebra and its Applications. 188:1993;135-161.
7. Goodwin G.C., Graebe S.F., Salgado M.E. Control system design. 2001;Prentice-Hall, Upper Saddle River, NJ.
8. Greenbaum A. Iterative methods for solving linear systems. 1997;SIAM, Philadelphia.
9. Hagiwara M., Sato A. Analysis of momentum term in back-propagation. IEICE Transactions on Information Systems. E78-D:(8):1995;1080-1086.
10. Haykin S. Neural networks. 2nd ed. 1999;Prentice-Hall, Upper Saddle River, NJ.
11. Kamarthi S.V., Pittner S. Accelerating neural network training using weight extrapolations. Neural Networks. 12:(9):1999;1285-1299.
12. Nocedal J., Wright S.J. Numerical optimization. Springer series in operations research. 1999;Springer, New York.
13. Phansalkar V.V., Sastry P.S. Analysis of the back-propagation algorithm with momentum. IEEE Transactions on Neural Networks. 5:(3):1994;505-506.
14. Polyak B.T. Some methods of speeding up the convergence of iterative methods. USSR Computational Mathematics and Mathematical Physics. 4:(5):1964;1-17.
15. Polyak B.T. Introduction to optimization. 1987;Optimization Software, New York.
16. Qian N. On the momentum term in gradient descent learning algorithms. Neural Networks. 12:(1):1999;145-151.
17. Quinn J.P. Stabilization of bilinear systems by quadratic feedback controls. Journal of Mathematical Analysis and its Applications. 75:(1):1980;66-80.
18. Ryan E.P., Buckingham N.J. On asymptotically stabilizing feedback control of bilinear systems. IEEE Transactions on Automatic Control. AC-28:(8):1983;863-864.
19. Rybashov M.V. Method of differential equations in the problem of finding the extremum of a function using analog computers. Automation and Remote Control. 30:(5):1969;181-194.
20. Torii M., Hagan M.T. Stability of steepest descent with momentum for quadratic functions. IEEE Transactions on Neural Networks. 13:(3):2002;752-756.
21. Tsypkin Y.Z. Adaptation and learning in automatic systems. Mathematics in science and engineering. Vol. 73:1971;Academic Press, New York. first published in Russian under the title Adaptatsia i obuchenie v avtomaticheskikh sistemakh, 1968, Moscow: Nauka.
22. Yu X.-H., Chen G.-A. Efficient backpropagation learning using optimal learning rate and momentum. Neural Networks. 10:(3):1997;517-527.
23. Yu X.-H., Chen G.-A., Cheng S.-X. Dynamic learning rate optimization of the backpropagation algorithm. IEEE Transactions on Neural Networks. 6:(3):1995;669-677.