On the oracle complexity of first-order and derivative-free algorithms for smooth nonconvex minimization

SIAM Journal on Optimization

Volumn 22, Issue 1, 2012, Pages 66-86

  Cartis, Coralia ^a   Gould, Nicholas I M ^b   Toint, Philippe L ^c  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

^b RUTHERFORD APPLETON LABORATORY   (United Kingdom)

^c University of Namur   (Belgium)

Author keywords

Derivative free optimization; Finite differences; First order methods; Nonconvex optimization; Oracle complexity; Worst case analysis

Indexed keywords

DERIVATIVE-FREE OPTIMIZATION; FINITE DIFFERENCE; FIRST-ORDER; NONCONVEX OPTIMIZATION; ORACLE COMPLEXITY; WORST-CASE ANALYSIS;

OPTIMIZATION;

ADAPTIVE ALGORITHMS;

EID: 84861621132     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/100812276     Document Type: Article

References (27)

1. A. Agarwal, P. L. Bartlett, P. Ravikummar, and M. J. Wainwright (2009), Informationtheoretic lower bounds on the oracle complexity of convex optimization, in Proceedings of the 23rd Annual Conference on Neural Information Processing Systems.
2. C. Cartis, N. I. M. Gould, and Ph. L. Toint (2010), On the complexity of steepest descent, Newton's and regularized Newton's methods for nonconvex unconstrained optimization problems, SIAM J. Optim., 20, pp. 2833-2852.
3. C. Cartis, N. I. M. Gould, and Ph. L. Toint (2011a), Adaptive cubic regularisation methods for unconstrained optimization. Part I: Motivation, convergence and numerical results, Math. Program., 127, pp. 245-295.
4. C. Cartis, N. I. M. Gould, and Ph. L. Toint (2011b), Optimal Newton-Type Methods for Nonconvex Smooth Optimization, ERGO Technical Report 11-09, School of Mathematics, University of Edinburgh.
5. C. Cartis, N. I. M. Gould, and Ph. L. Toint (2011c), Adaptive cubic regularisation methods for unconstrained optimization. Part II: Worst-case function-evaluation complexity, Math. Program., 130, pp. 295-319.
6. C. Cartis, N. I. M.Gould, andPh. L. Toint (2012), Complexity bounds for second-order optimality in unconstrained optimization, J. Complexity, 28, pp. 93-108.
7. A. R. Conn, N. I. M. Gould, and Ph. L. Toint (2000), Trust-Region Methods, MPS/SIAM Ser. Optim. 1, SIAM, Philadelphia.
8. A. R. Conn, K. Scheinberg, and L. N. Vicente (2009), Introduction to Derivative-Free Optimization, MPS/SIAM Ser. Optim. 8, SIAM, Philadelphia.
9. J. E. Dennis and R. B. Schnabel (1983), Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice-Hall, Englewood Cliffs, NJ. Reprinted as Classics Appl. Math. 16, SIAM, Philadelphia, 1996.
10. G. Fasano, J. Nocedal, and J.-L. Morales (2009), On the geometry phase in model-based algorithms for derivative-free optimization, Optim. Methods Softw., 24, pp. 145-154.
11. D. Goldfarb and Ph. L. Toint (1984), Optimal estimation of Jacobian and Hessian matrices that arise in finite difference calculations, Math. Comp., 43, pp. 69-88.
12. S. Gratton, A. Sartenaer, and Ph. L. Toint (2008), Recursive trust-region methods for multiscale nonlinear optimization, SIAM J. Optim., 19, pp. 414-444.
13. A. Griewank (1981), The Modification of Newton's Method for Unconstrained Optimization by Bounding Cubic Terms, Technical Report NA/12, Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK.
14. A. Griewank and Ph. L. Toint (1982), On the unconstrained optimization of partially separable functions, in Nonlinear Optimization 1981, M. J. D. Powell, ed., Academic Press, London, pp. 301-312.
15. A. S. Nemirovski (1994), Efficient Methods in Convex Programming, lecture notes, available online from http://www2.isye.gatech.edu/~nemirovs/OPTI LectureNotes.pdf.
16. A. S. Nemirovski and D. B. Yudin (1983), Problem Complexity and Method Efficiency in Optimization, J. Wiley and Sons, Chichester, UK.
17. Yu. Nesterov (2004), Introductory Lectures on Convex Optimization, Applied Optimization, Kluwer Academic, Dordrecht, The Netherlands.
18. Yu. Nesterov (2008), Accelerating the cubic regularization of Newton's method on convex problems, Math. Program., 112, pp. 159-181.
19. Yu. Nesterov and B. T. Polyak (2006), Cubic regularization of Newton method and its global performance, Math. Program., 108, pp. 177-205.
20. J. Nocedal and S. J. Wright, Numerical Optimization (1999), Springer Ser. Oper. Res., Springer-Verlag, Heidelberg, Berlin, New York.
21. M. J. D. Powell and Ph. L. Toint (1979), On the estimation of sparse Hessian matrices, SIAM J. Numer. Anal., 16, pp. 1060-1074.
22. K. Scheinberg and Ph. L. Toint (2010), Self-correcting geometry in model-based algorithms for derivative-free unconstrained optimization, SIAM J. Optim., 20, pp. 3512-3532.
23. S. A. Vavasis (1992a), Approximation algorithms for indefinite quadratic programming, Math. Program., 57, pp. 279-311.
24. S. A. Vavasis (1992b), Nonlinear Optimization: Complexity Issues, International Series of Monographs on Computer Science, Oxford University Press, Oxford, UK.
25. S. A. Vavasis (1993), Black-box complexity of local minimization, SIAM J. Optim., 3, pp. 60-80.
26. L. N. Vicente (2010), Worst Case Complexity of Direct Search, Technical report, Preprint 10-17, Department of Mathematics, University of Coimbra, Coimbra, Portugal (revised 2011).
27. M. Weiser, P. Deuflhard, and B. Erdmann (2007), Affine conjugate adaptive Newton methods for nonlinear elastomechanics, Optim. Methods Softw., 22, pp. 413-431.