The chaotic nature of faster gradient descent methods

Journal of Scientific Computing

Volumn 51, Issue 3, 2012, Pages 560-581

  Van Den Doel, Kees ^a   Ascher, Uri ^a  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

Chaos; Dynamical systems; Gradient descent; Iterative methods

Indexed keywords

CHAOS THEORY; CHAOTIC SYSTEMS; DYNAMICAL SYSTEMS; GRADIENT METHODS; ITERATIVE METHODS; LINEAR SYSTEMS; NUMBER THEORY;

CHAOTIC DYNAMICAL SYSTEMS; CONDITION NUMBERS; GRADIENT DESCENT; GRADIENT DESCENT METHOD; NORMALIZED RESIDUALS; NUMBER OF ITERATIONS; PRACTICAL METHOD; RESIDUAL VECTORS;

STEEPEST DESCENT METHOD;

EID: 84902276730     PISSN: 08857474     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10915-011-9521-3     Document Type: Article

References (30)

1. Akaike, H.: On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method. Ann. Inst. Stat. Math., Tokyo 11, 1-16 (1959)
2. Ascher, U., Huang, H., van den Doel, K.: Artificial time integration. BIT Numer. Math. 47, 3-25 (2007)
3. Ascher, U., van den Doel, K., Huang, H., Svaiter, B.: Gradient descent and fast artificial time integration. Modél. Math. Anal. Numér. 43, 689-708 (2009)
4. Barzilai, J., Borwein, J.: Two point step size gradient methods. IMA J. Numer. Anal. 8, 141-148 (1988)
5. Bhaya, A., Bliman, P.-A., Pazos, F.: Control-theoretic design of iterative methods for symmetric linear systems of equations. In: 48th IEEE Conf. on Decision and Control, pp. 2347-2380. IEEE Press, New York (2009).
6. Dai, Y., Fletcher, R.: On the asymptotic behaviour of some new gradient methods. Math. Program. 103, 541-559 (2005)
7. Dai, Y., Fletcher, R.: Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming. Numer. Math. 100, 21-47 (2005)
8. Dai, Y.H., Liao, L.Z.: R-linear convergence of the Barzilai and Borwein gradient method. IMA J. Numer. Anal. 22, 1-10 (2002)
9. Dai, Y., Yuan, Y.: Alternate minimization gradient method. IMA J. Numer. Anal. 23, 377-393 (2003)
10. Dai, Y., Hager, W., Schittkowsky, K., Zhang, H.: A cyclic Barzilai-Borwein method for unconstrained optimization. IMA J. Numer. Anal. 26, 604-627 (2006)
11. Figueiredo, M., Nowak, R., Wright, S.: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Special Topics Signal Process. 1, 586-598 (2007)
12. Fletcher, R.: On the Barzilai-Borwein method. In: Qi, L., Teo, K., Yang, X. (eds.) Optimization and Control with Applications. Series in Applied Optimization, vol. 96, pp. 235-256. Kluwer, Dordrecht (2005)
13. Friedlander, A., Martinez, J., Molina, B., Raydan, M.: Gradient method with retard and generalizations. SIAM J. Numer. Anal. 36, 275-289 (1999)
14. Glunt, W., Hayden, T.L., Raydan, M.: Molecular conformations from distance matrices. J. Comput. Chem. 14, 114-120 (1993)
15. Golub, G., Ye, Q.: Inexact preconditioned conjugate gradient method with inner-outer iteration. SIAM J. Sci. Comput. 21, 1305-1320 (2000)
16. Golub, G.H., van Loan, C.F.: Matrix Computations. Johns Hopkins University Press, Baltimore (1988)
17. Goupillaud, P., Grossmann, A., Morlet, H.: Cycle-octave and related transforms in seismic signal analysis. Geoexploration 23, 85-102 (1984)
18. Haber, E., Ascher, U.: Preconditioned all-At-one methods for large, sparse parameter estimation problems. Inverse Probl. 17, 1847-1864 (2001)
19. Huang, H., Ascher, U.: Faster gradient descent and the efficient recovery of images. Math. Program. (2011).
20. Li, T.Y., Yorke, J.A.: Period three implies chaos. Am. Math. Mon. 82, 985-992 (1975)
21. Lyapunov, A.M.: The General Problem of the Stability of Motion. Taylor and Francis, London (1992)
22. Nagy, J., Palmer, K.: Steepest descent, CG and iterative regularization of ill-posed problems. BIT Numer. Math. 43, 1003-1017 (2003)
23. Nocedal, J., Wright, S.: Numerical Optimization. Springer, New York (1999)
24. Nocedal, J., Sartenaer, A., Zhu, C.: On the behavior of the gradient norm in the steepest descent method. Comput. Optim. Appl. 22, 5-35 (2002)
25. Pronzato, L., Wynn, H., Zhigljavsky, A.: Dynamical Search: Applications of Dynamical Systems in Search and Optimization. Chapman & Hall/CRC, Boca Raton (2000)
26. Raydan, M., Svaiter, B.: Relaxed steepest descent and Cauchy-Barzilai-Borwein method. Comput. Optim. Appl. 21, 155-167 (2002)
27. Shimada, I., Nagashima, T.: A numerical approach to ergodic problems of dissipative dynamical systems. Prog. Theor. Phys. 61(6), 1605-1616 (1979)
28. Stuart, A.M., Humphries, A.R.: Dynamical Systems and Numerical Analysis. Cambridge University Press, Cambridge (1996)
29. Trottenberg, U., Oosterlee, C., Schuller, A.: Multigrid. Academic Press, San Diego (2001)
30. Vogel, C.: Computational Methods for Inverse Problem. SIAM, Philadelphia (2002)