Variable metric methods for unconstrained optimization and nonlinear least squares

Journal of Computational and Applied Mathematics

Volumn 124, Issue 1-2, 2000, Pages 61-95

  Lukšan, Ladislav ^a   Spedicato, Emilio ^b  

^a INSTITUTE OF COMPUTER SCIENCE   (Czech Republic)

^b UNIVERSITY OF BERGAMO   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

CONVERGENCE OF NUMERICAL METHODS; LEAST SQUARES APPROXIMATIONS; PROBLEM SOLVING;

NONLINEAR LEAST SQUARES; QUASI-NEWTON METHODS; SPARSE PROBLEMS; UNCONSTRAINED OPTIMIZATION;

MATHEMATICAL PROGRAMMING;

EID: 0034544838     PISSN: 03770427     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0377-0427(00)00420-9     Document Type: Article

References (99)

1. J. Abaffy E. Spedicato ABS Projection Algorithms, Mathematical Techniques for Linear and Nonlinear Equations 1989 Ellis Horwood Chichester
2. N. Adachi On variable metric algorithm J. Optim. Theory Appl. 7 1971 391 409
3. M. Al-Baali Highly efficient Broyden methods of minimization with variable parameter Optim. Methods Software 1 1992 301 310
4. M. Al-Baali R. Fletcher Variational methods for nonlinear least squares J. Optim. Theory Appl. 36 1985 405 421
5. O. Axelsson Iterative Solution Methods 1996 Cambridge University Press Cambridge
6. J.T. Betts Solving the nonlinear least squares problem: application of a general method J. Optim. Theory Appl. 18 1976 469 483
7. M.C. Biggs Minimization algorithms making use of nonquadratic properties of the objective function J. Inst. Math. Appl. 8 1971 315 327
8. M.C. Biggs A note on minimization algorithms which make use of non-quadratic properties of the objective function J. Inst. Math. Appl. 12 1973 337 338
9. M.C. Biggs The estimation of the Hessian matrix in nonlinear least squares problems with nonzero residuals Math. Programming 12 1977 67 80
10. C.G. Broyden A class of methods for solving nonlinear simultaneous equations Math. Comp. 19 1965 577 593
11. C.G. Broyden, The convergence of a class of double rank minimization algorithms, Part 1 – general considerations, Part 2 – the new algorithm, J. Inst. Math. Appl. 6 (1970) 76–90, 222–231.
12. A. Buckley A combined conjugate-gradient quasi-Newton minimization algorithm Math. Programming 15 1978 200 210
13. A. Buckley A. LeNir QN-like variable storage conjugate gradients Math. Programming 27 1983 155 175
14. R.H. Byrd, D.C. Liu, J. Nocedal, On the behavior of Broyden's class of quasi-Newton methods, Report No. NAM 01, Dept. of Electrical Engn. and Computer Science, Northwestern University, Evanston, 1990.
15. R.H. Byrd J. Nocedal R.B. Schnabel Representation of quasi-Newton matrices and their use in limited memory methods Math. Programming 63 1994 129 156
16. R.H. Byrd J. Nocedal Y.X. Yuan Global convergence of a class of quasi-Newton methods on convex problems SIAM J. Numer. Anal. 24 1987 1171 1190
17. M. Coleman J.J. Moré Estimation of sparse Hessian matrices and graph coloring problems Math. Programming 42 1988 245 270
18. M. Contreras R.A. Tapia Sizing the BFGS and DFP updates: a numerical study J. Optim. Theory Appl. 78 1993 93 108
19. W.C. Davidon, Variable metric method for minimisation, A.E.C. Research and Development Report ANL-5990, 1959.
20. W.C. Davidon Optimally conditioned optimization algorithms without line searches Math. Programming 9 1975 1 30
21. R.S. Dembo T. Steihaug Truncated-Newton algorithms for large-scale unconstrained minimization Math. Programming 26 1983 190 212
22. J.E. Dennis Some computational techniques for the nonlinear least squares problem G.D. Byrne C.A. Hall Numerical Solution of Nonlinear Algebraic Equations 1974 Academic Press London
23. J.E. Dennis D. Gay R.E. Welsch An adaptive nonlinear least squares algorithm ACM Trans. Math. Software 7 1981 348 368
24. J.E. Dennis H.J. Martinez R.A. Tapia Convergence theory for the structured BFGS secant method with application to nonlinear least squares J. Optim. Theory Appl. 61 1989 161 177
25. J.E. Dennis, H.H.W. Mei, An unconstrained optimization algorithm which uses function and gradient values, Report No. TR-75-246, Dept. of Computer Science, Cornell University, 1975.
26. J.E. Dennis J.J. Moré A characterization of superlinear convergence and its application to quasi-Newton methods Math. Comp. 28 1974 549 560
27. J.E. Dennis, R.B. Schnabel, Least change secant updates for quasi-Newton methods, Report No. TR78-344, Dept. of Computer Sci., Cornell University, Ithaca, 1978.
28. J.E. Dennis R.B. Schnabel Numerical Methods for Unconstrained Optimization and Nonlinear Equations 1983 Prentice-Hall Englewood Cliffs, NJ
29. J.E. Dennis R.B. Schnabel A view of unconstrained optimization G.L. Nemhauser A.H.G. Rinnooy Kan M.J. Todd Optimization 1989 North-Holland Amsterdam
30. L.C.W. Dixon Quasi-Newton algorithms generate identical points Math. Programming 2 1972 383 387
31. R. Fletcher A new approach to variable metric algorithms Comput. J. 13 1970 317 322
32. R. Fletcher Practical Methods of Optimization 1987 Wiley New York
33. R. Fletcher Low storage methods for unconstrained optimization E.L. Algower K. Georg Computational Solution of Nonlinear Systems of Equations, Lectures in Applied Mathematics, Vol. 26 1990 AMS Publications Providence, RI
34. R. Fletcher A new variational result for quasi-Newton formulae SIAM J. Optim. 1 1991 18 21
35. R. Fletcher An optimal positive definite update for sparse Hessian matrices SIAM J. Optim. 5 1995 192 218
36. R. Fletcher M.J.D. Powell A rapidly convergent descent method for minimization Comput. J. 6 1963 163 168
37. R. Fletcher C.M. Reeves Function minimization by conjugate gradients Comput. J. 7 1964 149 154
38. R. Fletcher C. Xu Hybrid methods for nonlinear least squares IMA J. Numer. Anal. 7 1987 371 389
39. J.A. Ford I.A. Moghrabi Alternative parameter choices for multi-step quasi-Newton methods Optim. Methods Software 2 1993 357 370
40. J.A. Ford I.A. Moghrabi Multi-step quasi-Newton methods for optimization J. Comput. Appl. Math. 50 1994 305 323
41. J.A. Ford, I.A. Moghrabi, Minimum curvature multi-step quasi-Newton methods for unconstrained optimization, Report No. CSM-201, Department of Computer Science, University of Essex, Colchester, 1995.
42. J. Greenstadt Variations on variable metric methods Math. Comput. 24 1970 1 18
43. J.C. Gilbert C. Lemarechal Some numerical experiments with variable-storage quasi-Newton algorithms Math. Programming 45 1989 407 435
44. P.E. Gill, M.W. Leonard, Limited-memory reduced-Hessian methods for large-scale unconstrained optimization, Report NA 97-1, Dept. of Mathematics, University of California, San Diego, La Jolla, 1997.
45. P.E. Gill W. Murray M.A. Saunders Methods for computing and modifying LDV factors of a matrix Math. Comput. 29 1975 1051 1077
46. D. Goldfarb A family of variable metric algorithms derived by variational means Math. Comput. 24 1970 23 26
47. A. Griewank The global convergence of partitioned BFGS on problems with convex decompositions and Lipschitzian gradients Math. Programming 50 1991 141 175
48. A. Griewank P.L. Toint Partitioned variable metric updates for large-scale structured optimization problems Numer. Math. 39 1982 119 137
49. A. Griewank P.L. Toint Local convergence analysis for partitioned quasi-Newton updates Numer. Math. 39 1982 429 448
50. A. Griewank, P.L. Toint, Numerical experiments with partially separable optimization problems, in: D.F. Griffits (Ed.), Numerical Analysis, Proc. Dundee 1983, Lecture Notes in Mathematics, Vol. 1066, Springer, Berlin, 1984, pp. 203–220.
51. M.R. Hestenes C.M. Stiefel Methods of conjugate gradient for solving linear systems J. Res. NBS 49 1964 409 436
52. S. Hoshino A formulation of variable metric methods J. Inst. Math. Appl. 10 1972 394 403
53. H.Y. Huang Unified approach to quadratically convergent algorithms for function minimization J. Optim. Theory Appl. 5 1970 405 423
54. J. Huschens On the use of product structure in secant methods for nonlinear least squares SIAM J. Optim. 4 1994 108 129
55. D.C. Liu J. Nocedal On the limited memory BFGS method for large-scale optimization Math. Programming 45 1989 503 528
56. L. Lukšan Computational experience with improved variable metric methods for unconstrained minimization Kybernetika 26 1990 415 431
57. L. Lukšan Computational experience with known variable metric updates J. Optim. Theory Appl. 83 1994 27 47
58. L. Lukšan, J. Vlček, Simple scaling for variable metric updates, Report No. 611, Institute of Computer Science, Academy of Sciences of the Czech Republic, Prague 1995.
59. L. Lukšan Hybrid methods for large sparse nonlinear least squares J. Optim. Theory Appl. 89 1996 575 595
60. L. Lukšan, Numerical methods for unconstrained optimization, Report DMSIA 12/97, University of Bergamo, 1997.
61. L. Lukšan, M. Tůma, M. Šiška, J. Vlček, N. Ramešová, Interactive system for universal functional optimization (UFO) — Version 1998, Report No. 766, Institute of Computer Science, Academy of Sciences of the Czech Republic, Prague, 1998.
62. L. Lukšan, J.Vlček, Subroutines for testing large sparse and partially separable unconstrained and equality constrained optimization problems, Report No. 767, Institute of Computer Science, Academy of Sciences of the Czech Republic, Prague, 1999.
63. L. Lukšan J. Vlček Globally convergent variable metric method for convex nonsmooth unconstrained minimization J. Optim. Theory Appl. 102 1999 593 613
64. J.M. Martinez A quasi-Newton method with modification of one column per iteration Computing 33 1984 353 362
65. J.J. Moré B.S. Garbow K.E. Hillstrom Testing unconstrained optimization software ACM Trans. Math. Software 7 1981 17 41
66. J.J. Moré, D.C. Sorensen, Computing a trust region step, Report ANL-81-83, Argonne National Laboratory, 1981.
67. J. Nocedal Updating quasi-Newton matrices with limited storage Math. Comp. 35 1980 773 782
68. J. Nocedal Y. Yuan Analysis of a self-scaling quasi-Newton method Math. Programming 61 1993 19 37
69. S.S. Oren, D.G. Luenberger, Self scaling variable metric (SSVM) algorithms, Part 1 — criteria and sufficient condition for scaling a class of algorithms, Part 2 — implementation and experiments, Management Sci. 20 (1974) 845–862, 863–874.
70. S.S. Oren E. Spedicato Optimal conditioning of self scaling variable metric algorithms Math. Programming 10 1976 70 90
71. M.R. Osborne, L.P. Sun, A new approach to the symmetric rank-one updating algorithm. Report No. NMO/01, School of Mathematics, Australian National University, Canberra, 1988.
72. A. Perry A modified conjugate gradient algorithm Oper. Res. 26 1978 1073 1078
73. E. Polak G. Ribiére Note sur la convergence des méthodes de directions conjugées Rev. Française Inform. Mech. Oper. 16-R1 1969 35 43
74. M.J.D. Powell A new algorithm for unconstrained optimization J.B. Rosen O.L. Mangasarian K. Ritter Nonlinear Programming 1970 Academic Press London
75. M.J.D. Powell On the global convergence of trust region algorithms for unconstrained minimization Math. Programming 29 1984 297 303
76. D.G. Pu W.W. Tian A class of modified Broyden algorithms J. Comput. Math. 12 1994 366 379
77. D.F. Shanno Conditioning of quasi-Newton methods for function minimization Math. Comp. 24 1970 647 656
78. D.F. Shanno K.J. Phua Matrix conditioning and nonlinear optimization Math. Programming 14 1978 144 160
79. D. Siegel, Implementing and modifying Broyden class updates for large scale optimization, Report DAMTP NA12, Department of Applied Mathematics and Theoretical Physics, University of Cambridge, 1992.
80. E. Spedicato Stability of Huang's update for the conjugate gradient method J. Optim. Theory Appl. 11 1973 469 479
81. E. Spedicato On condition numbers of matrices in rank-two minimization algorithms L.C.W. Dixon G.P. Szego Towards Global Optimization 1975 North-Holland Amsterdam
82. E. Spedicato A bound on the condition number of rank-two corrections and applications to the variable metric method Calcolo 12 1975 185 199
83. E. Spedicato, Parameter estimation and least squares, in: F. Archetti, M. Cugiani (Eds.), Numerical Techniques for Stochastic Systems, North-Holland, Amsterdam, 1980.
84. E. Spedicato A class of rank-one positive definite quasi-Newton updates for unconstrained minimization Math. Operationsforsch. Statist. Ser. Optim. 14 1983 61 70
85. E. Spedicato A class of sparse symmetric quasi-Newton updates Ricerca Operativa 22 1992 63 70
86. E. Spedicato, N.Y. Deng, Z. Li, On sparse quasi-Newton quasi-diagonally dominant updates, Report DMSIA 1/96, University of Bergamo, 1996.
87. E. Spedicato Z. Xia Finding general solutions of the quasi-Newton equation via the ABS approach Optim. Methods Software 1 1992 243 252
88. E. Spedicato J. Zhao Explicit general solution of the Quasi-Newton equation with sparsity and symmetry Optim. Methods Software 2 1993 311 319
89. T. Steihaug Local and superlinear convergence for truncated iterated projections methods Math. Programming 27 1983 176 190
90. T. Steihaug The conjugate gradient method and trust regions in large-scale optimization SIAM J. Numer. Anal. 20 1983 626 637
91. P.L. Toint On sparse and symmetric matrix updating subject to a linear equation Math. Comp. 31 1977 954 961
92. P.L. Toint Global convergence of the partitioned BFGS algorithm for convex partially separable optimization Math. Programming 36 1986 290 306
93. P.L. Toint On large scale nonlinear least squares calculations SIAM J. Sci. Statis. Comput. 8 1987 416 435
94. M. Tůma, Sparse fractioned variable metric updates, Report No. 497, Institute of Computer and Information Sciences, Czechoslovak Academy of Sciences, Prague 1991.
95. H. Wolkowicz Measures for rank-one updates Math. Oper. Res. 19 1994 815 830
96. H. Wolkowicz An all-inclusive efficient region of updates for least change secant methods SIAM J. Optim. 5 1996 172 191
97. H. Yabe T. Takahashi Factorized quasi-Newton methods for nonlinear least squares problems Math. Programming 51 1991 75 100
98. Y. Yuan Non-quasi-Newton updates for unconstrained optimization J. Comput. Math. 13 1995 95 107
99. J. Zhang, N.Y. Deng, L. Chen, A new quasi-Newton equation and related methods for unconstrained optimization, Report MA-96-05, City University of Hong Kong, 1996.