Convergence of trust-region methods based on probabilistic models

SIAM Journal on Optimization

Volumn 24, Issue 3, 2014, Pages 1238-1264

  Bandeira, A S ^a   Scheinberg, K ^b   Vicente, L N ^c  

^a PRINCETON UNIVERSITY   (United States)

^b Lehigh University   (United States)

^c UNIVERSITY OF COIMBRA   (Portugal)

Author keywords

Derivativefree optimization; Global convergence; Probabilistic models; Trust region methods; Unconstrained optimizat ion

Indexed keywords

GRADIENT METHODS; IMAGE REGISTRATION; OPTIMIZATION; STOCHASTIC SYSTEMS;

DERIVATIVE-FREE OPTIMIZATION; GLOBAL CONVER-GENCE; NONLINEAR FUNCTIONS; PROBABILISTIC MODELS; STANDARD OPTIMIZATION; STOCHASTIC GRADIENT METHODS; STOCHASTIC OPTIMIZATION APPROACH; TRUST-REGION METHODS;

STOCHASTIC MODELS;

EID: 84910601452     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/130915984     Document Type: Article

References (35)

1. C. AUDET and J. E. DENNIS JR., Mesh adaptive direct search algorithms for constrained optimization, SIAM J. Optim., 17(2006), pp. 188-217.
2. U. AYAZ and H. RAUHUT, Nonuniform sparse recovery with subgaussian matrices, Electron. Trans. Numer. Anal., 41(2014), pp. 167-178.
3. A. S. BANDEIRA, K. SCHEINBERG, and L. N. VICENTE, Computation of sparse low degree interpolating polynomials and their application to derivative-free optimization, Math. Program., 134(2012), pp. 223-257.
4. S. C. BILLUPS, J. LARSON, and P. GRAF, Derivative-free optimization of expensive functions with computational error using weighted regression, SIAM J. Optim., 23(2013), pp. 27-53.
5. R. BYRD, G. M. CHIN, J. NOCEDAL, and Y. WU, Sample size selection in optimization methods for machine learning, Math. Program., 134(2012), pp. 127-155.
6. E. J. CANDÈS, Compressive sampling, Proceedings of the International Congress of Mathematicians, Vol. III. Madrid 2006, European Mathematical Society, Zurich, 2006, pp. 1433-1452.
7. Z. CHEN and J. J. DONGARRA, Condition numbers of Gaussian random matrices, SIAM J. Matrix Anal. Appl., 27(2005), pp. 603-620.
8. A. R. CONN, N. I. M. GOULD, and PH. L. TOINT, Trust-Region Methods, MPS-SIAM Ser. Optim. 1, SIAM, Philadelphia, 2000.
9. A. R. CONN, K. SCHEINBERG, and PH. L. TOINT, On the convergence of derivative-free methods for unconstrained optimization, in Approximation Theory and Optimization, Tributes to M. J. D. Powell, M. D. Buhmann, and A. Iserles, eds., Cambridge University Press, Cambridge, 1997, pp. 83-108.
10. A. R. CONN, K. SCHEINBERG, and PH. L. TOINT, Recent progress in unconstrained nonlinear optimization without derivatives, Math. Program., 79(1997), pp. 397-414.
11. A. R. CONN, K. SCHEINBERG, and PH. L. TOINT, A derivative free optimization algorithm in practice, in Proceedings of the 7th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, St. Louis, Missouri, 1998, American Institute of Aeronautics and Astronautics, Restor, VA, 1998, pp. 129-139.
12. A. R. CONN, K. SCHEINBERG, and L. N. VICENTE, Geometry of interpolation sets in derivative free optimization, Math. Program., 111(2008), pp. 141-172.
13. A. R. CONN, K. SCHEINBERG, and L. N. VICENTE, Geometry of sample sets in derivative free optimization: Polynomial regression and underdetermined interpolation, IMA J. Numer. Anal., 28(2008), pp. 721-748.
14. A. R. CONN, K. SCHEINBERG, and L. N. VICENTE, Global convergence of general derivativefree trust-region algorithms to first- and second-order critical points, SIAM J. Optim., 20(2009), pp. 387-415.
15. A. R. CONN, K. SCHEINBERG, and L. N. VICENTE, Introduction to Derivative-Free Optimization, MPS-SIAM Ser. Optim. 8, SIAM, Philadelphia, 2009.
16. R. DURRETT, Probability: Theory and Examples, Camb. Ser. Stat. Probab. Math., 4th ed., Cambridge University Press, Cambridge, 2010.
17. A. EDELMAN, Eigenvalues and condition numbers of random matrices, SIAM J. Matrix Anal. Appl., 9(1988), pp. 543-560.
18. G. FASANO, J. L. MORALES, and J. NOCEDAL, On the geometry phase in model-based algorithms for derivative-free optimization, Optim. Methods Softw., 24(2009), pp. 145-154.
19. S. GHADIMI and G. LAN, Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming, SIAM J. Optim., 23(2013), pp. 2341-2368.
20. T. G. KOLDA, R. M. LEWIS, and V. TORCZON, Optimization by direct search: New perspectives on some classical and modern methods, SIAM Rev., 45(2003), pp. 385-482.
21. J. MATYAS, Random optimization, Autom. Remote Control, 26(1965), pp. 246-253.
22. J. J. MORÉ and S. M. WILD, Benchmarking derivative-free optimization algorithms, SIAM J. Optim., 20(2009), pp. 172-191.
23. Y. NESTEROV, Random Gradient-Free Minimization of Convex Functions, Technical report 2011/1, CORE, Discussion Paper, Université Catholique de Louvain, Louvain, Belgium, 2011.
24. YU. NESTEROV, Efficiency of coordinate descent methods on huge-scale optimization problems, SIAM J. Optim., 22(2012), pp. 341-362.
25. B. T. POLYAK, Introduction to Optimization, Optimization Software, New York, 1987.
26. M. J. D. POWELL, A direct search optimization method that models the objective and constraint functions by linear interpolation, in Advances in Optimization and Numerical Analysis, Proceedings of the Sixth Workshop on Optimization and Numerical Analysis, Oaxaca, Mexico, S. Gomez and J.-P. Hennart, eds., Math. Appl. 275, Kluwer Academic, Dordrecht, 1994, pp. 51-67.
27. M. J. D. POWELL, On the Lagrange functions of quadratic models that are defined by interpolation, Optim. Methods Softw., 16(2001), pp. 289-309.
28. M. J. D. POWELL, On trust region methods for unconstrained minimization without derivatives, Math. Program., 97(2003), pp. 605-623.
29. M. J. D. POWELL, Least Frobenius norm updating of quadratic models that satisfy interpolation conditions, Math. Program., 100(2004), pp. 183-215.
30. H. RAUHUT, Compressive sensing and structured random matrices, in Theoretical Foundations and Numerical Methods for Sparse Recovery, M. Fornasier, ed., Radon Series Comp. Appl. Math., De Gruyter, Berlin, 2010, pp. 1-92.
31. K. SCHEINBERG and PH. L. TOINT, Self-correcting geometry in model-based algorithms for derivative-free unconstrained optimization, SIAM J. Optim., 20(2010), pp. 3512-3532.
32. V. TORCZON, On the convergence of pattern search algorithms, SIAM J. Optim., 7(1997), pp. 1-25.
33. L. N. VICENTE and A. L. CUSTÓDIO, Analysis of direct searches for discontinuous functions, Math. Program., 133(2012), pp. 299-325.
34. L. N. VICENTE, Worst case complexity of direct search, EURO J. Comput. Optim., 1(2013), pp. 143-153.
35. S. M. WILD, MNH: A derivative-free optimization algorithm using minimal norm Hessians, in Tenth Copper Mountain Conference on Iterative Methods, 2008, Copper Mountain, CO, 2008.