Convergence analysis of the Levenberg-Marquardt method

Optimization Methods and Software

Volumn 22, Issue 4, 2007, Pages 659-678

  Luo, Xin Long ^a   Liao, Li Zhi ^b   Tam, Hon Wah ^b  

^a BEIJING UNIVERSITY OF POSTS AND TELECOMMUNICATIONS   (China)

^b HONG KONG BAPTIST UNIVERSITY   (Hong Kong)

Author keywords

Levenberg Marquardt method; Ordinary differential equations; Quasi Newton method; Trust region method; Unconstrained optimization

Indexed keywords

CONSTRAINED OPTIMIZATION; ORDINARY DIFFERENTIAL EQUATIONS; PROBLEM SOLVING;

LEVENBERG MARQUARDT METHOD; QUASI NEWTON METHOD; TRUST REGION METHOD; UNCONSTRAINED OPTIMIZATION;

CONVERGENCE OF NUMERICAL METHODS;

EID: 34147222084     PISSN: 10556788     EISSN: 10294937     Source Type: Journal    
DOI: 10.1080/10556780601079233     Document Type: Article

References (36)

1. Byrd, R.H., Nocedal, J. and Yuan, Y. 1987, Global convergence of a class of quasi-Newton methods on convex problems. SIAM Journal of Numerical Analysis, 24, 1171-1190.
2. Byrd, R.H. and Nocedal, J. 1989, A tool for the analysis of quasi-Newton methods with application to unconstrained minimization. SIAM Journal on Numerical Analysis, 26, 727-739.
3. Conn, A.R., Gould, N. Toint, Ph.L., 2000, Trust-Region Methods (Philadelphia, USA: SIAM).
4. Dennis, J.E. and Moré, J.J., 1974, A characterization of superlinear convergence and its application to quasi-Newton methods. Mathematics of Computation, 28, 549-560.
5. Dennis, J.E. and Moré, J.J., 1977, Quasi-Newton methods, motivation and theory. SIAM Review, 19, 46-89.
6. Dennis, J.E. and Schnabel, R.B., 1983, Numerical Methods for Unconstrained Optimization and Nonlinear Equations (Englewood Cliffs, NJ: Prentice-Hall, Inc.).
7. Fan, J.Y. and Yuan, Y.X. 2001, On the convergence of a new Levenberg-Marquardt method. Technical Report, AMSS, Chinese Academy of Sciences.
8. Fletcher, R., 1987, Practical Methods of Optimization (New York: John Wiley and Sons).
9. Gay, D.M., 1981, Computing optimal locally constrained steps. SIAM Journal on Scientific and Statistical Computing, 2, 186-197.
10. Goldfeld, S.M., Quandt, R.E. and Trotter, H.F., 1966, Maximisation by quadratic hill-climbing. Econometrica, 34, 541-551.
11. Hairer, E., Nørsett, S.P. and Wanner, G., 1993, Solving Ordinary Differential Equations I, Nonstiff Problems (2nd edn) (Berlin: Springer-Verlag).
12. Han, Q.M., Liao, L.-Z., Qi, H.D. and Qi, L.Q., 2001, Stability analysis of gradient-based neural networks for optimization problems. Journal of Global Optimization, 19, 363-381.
13. He, B .S., 2000, Solving trust region problem in large scale optimization. Journal of Computational Mathematics, 18, 1-12.
14. Higham, D.J., 1999, Trust region algorithms and timestep selection. SIAM Journal on Numerical Analysis, 37, 194-210.
15. Humphries, A.R. and Stuart, A.M., 1994, Runge-Kutta methods for dissipative and gradient dynamical systems. SIAM Journal of Numerical Analysis, 31, 1452-1485.
16. Higham, D.J. and Stuart, A.M., 1998, Analysis of the dynamics of local error control via a piecewise continuous residual. BIT, 38, 44-57.
17. Kelley, C.T., 1999, Iterative Methods for Optimization (Philadelphia, USA: SIAM).
18. Kelley, C.T. and Keyes, D.E., 1998, Convergence analysis of pseudo-transient continuation. SIAM Journal on Numerical Analysis, 35, 508-523.
19. Levenberg, K., 1944, A method for the solution of certain problems in least squares. Quarterly of Applied Mathematics, 2, 164-168.
20. Luo, X.L., 2005, Singly Diagonally Implicit Runge-Kutta Methods Combining Line Search Techniques for Unconstrained Optimization. Journal of Computational Mathematics, 23, 153-164.
21. Luo, X.L., Kelley, C.T., Liao, L.-Z. and Tam, H.W., 2006, Combining trust region techniques and Rosenbrock methods for gradient systems, submitted for publication.
22. Marquardt, D., 1963, An algorithm for least-squares estimation of nonlinear parameters. SIAM Journal on Applied Mathematics, 11, 431-441.
23. Moré, J.J. and Sorensen, D.C. 1983, Computing a trust region step. SIAM Journal on Scientific and Statistical Computing, 4, 553-572.
24. Nocedal, J. and Wright, S.J., 1999, Numerical Optimization (Springer-Verlag).
25. Nocedal, J. and Yuan, Y., 1998, Combining trust region and line search techniques. In: Y. Yuan (Ed.) Advances in Nonlinear Programming (Kluwer), pp. 153-175.
26. Pearson, J.D., 1969, Variable metric methods of minimization. The Computer Journal, 12, 171-178.
27. Powell, M.J.D., 1975, Convergence properties of a class of minimization algorithms. In: O.L. Mangasarian, R.R. Meyer and S.M. Robinson (Eds) Nonlinear Programming 2 (New York: Academic Press), pp. 1-27.
28. Powell, M.J.D., 1976, Some global convergence properties of a variable metric algorithm for minimization without exact line searches. In: R.W. Cottle and C.E. Lemeke, (Eds), Nonlinear Programming, SIAM-AMS Proceedings (SIAM publications) 9, pp. 53-72.
29. Powell, M.J.D., 1982, ED., Nonlinear Optimization 1981: Proceedings of the NATO Advanced Research.Institute (New York: Academic Press).
30. Powell, M.J.D., 1984, On the global convergence of trust region algorithm for unconstrained optimization. Mathematical Programming, 29, 297-303.
31. Schropp, J., 1995, Using dynamical systems methods to solving minimization problems. Applied Numerical Mathematics, 18, 321-335.
32. Schnabel, R.B. and Eskow, E., 1999, A revised modified Cholesky factorization algorithm. SIAM Journal on Optimization, 9, 1135-1148.
33. Steihaug, T., 1983, The conjugate gradient method and trust regions in large scale optimization. SIAM Journal on Numerical Analysis, 20, 626-637.
34. Stuart, A.M. and Humphries, A.R., 1994, Model problems in numerical stability theory for initial value problems. SIAM Review, 36, 226-257.
35. Yamashita, N. and Fukushima, M., 2001, On the rate of convergence of the Levenberg-Marquardt method. Computing (Supplement 15), 239-249.
36. Yuan, Y. and Sun, W., 1997, Optimization Theories and Methods (in Chinese) (Beijing: Science Press).