Stochastic first- and zeroth-order methods for nonconvex stochastic programming

SIAM Journal on Optimization

Volumn 23, Issue 4, 2013, Pages 2341-2368

  Ghadimi, Saeed ^a   Lan, Guanghui ^a  

^a University of Florida   (United States)

Author keywords

Nonconvex optimization; Simulation based optimization; Stochastic approximation; Stochastic programming

Indexed keywords

NONCONVEX OPTIMIZATION; NONCONVEX STOCHASTIC PROGRAMMING; NONLINEAR PROGRAMMING PROBLEM; OPTIMAL RATE OF CONVERGENCE; SIMULATION-BASED OPTIMIZATIONS; STATIONARY POINTS; STOCHASTIC APPROXIMATIONS; STOCHASTIC GRADIENT;

ALGORITHMS; APPROXIMATION THEORY; STOCHASTIC PROGRAMMING; STOCHASTIC SYSTEMS;

OPTIMIZATION;

EID: 84892854517     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/120880811     Document Type: Article

References (41)

1. S. Andradóttir, A review of simulation optimization techniques, in Proceedings of the Winter Simulation Conference, 1998, pp. 151-158.
2. C. Cartis, N. I.M. Gould, and Ph. L. Toint, On the complexity of steepest descent, Newton's and regularized Newton's methods for nonconvex unconstrained optimization, SIAM J. Optim., 20 (2010), pp. 2833-2852.
3. C. Cartis, N. I. M. Gould, and Ph. L. Toint, On the oracle complexity of first-order and derivative-free algorithms for smooth nonconvex minimization, SIAM J. Optim., 22 (2012), pp. 66-86.
4. K. L. Chung, On a stochastic approximation method, Ann. Math. Statist., 25 (1954), pp. 463-483.
5. A. R. Conn, K. Scheinberg, and L. N. Vicente, Introduction to Derivative-Free Optimization, SIAM, Philadelphia, 2009.
6. J. C. Duchi, P. L. Bartlett, and M. J. Wainwright, Randomized smoothing for stochastic optimization, SIAM J. Optim., 22 (2012), pp. 674-701.
7. M. Fu, Optimization for simulation: Theory vs. practice, INFORMS J. Comput., 14 (2002), pp. 192-215.
8. M. C. Fu, Gradient estimation, in Simulation, S. G. Henderson and B. L. Nelson, eds., Handbooks in Operations Research and Management Science 13 Elsevier, New York, pp. 575-616.
9. R. Garmanjani and L. N. Vicente, Smoothing and worst-case complexity for direct-search methods in nonsmooth optimization, IMA J. Numer. Anal., 33 (2013), pp. 1008-1028.
10. S. Ghadimi and G. Lan, Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, II: Shrinking procedures and optimal algorithms, SIAM J. Optim., 23 (2013), pp. 2061-2089.
11. S. Ghadimi and G. Lan, Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, I: A generic algorithmic framework, SIAM J. Optim., 22 (2012), pp. 1469-1492.
12. S. Ghadimi and G. Lan, Stochastic First-and Zeroth-Order Methods for Nonconvex Stochastic Programming, Technical report, Department of Industrial and Systems Engineering, University of Florida, Gainesville, 2012.
13. P. Glasserman, Gradient Estimation via Perturbation Analysis, Kluwer Academic Publishers, Boston, 2003.
14. A. Juditsky, A. Nazin, A. B. Tsybakov, and N. Vayatis, Recursive aggregation of estimators via the mirror descent algorithm with average, Prob. Inf. Transm., 41 (2005), pp. 368-384.
15. A. Juditsky and A. S. Nemirovski, Large Deviations of Vector-Valued Martingales in 2-Smooth Normed Spaces, www2.isye.gatech.edu/~nemirovs/ LargeDevSubmitted.pdf.
16. A. Juditsky, P. Rigollet, and A. B. Tsybakov, Learning by mirror averaging, Ann. Statist., 36 (2008), pp. 2183-2206.
17. A. J. Kleywegt, A. Shapiro, and T. Homem de Mello, The sample average approximation method for stochastic discrete optimization, SIAM J. Optim., 12 (2001), pp. 479-502.
18. G. Lan, An optimal method for stochastic composite optimization, Math. Program., 133 (2012), pp. 365-397.
19. G. Lan, A. S. Nemirovski, and A. Shapiro, Validation analysis of mirror descent stochastic approximation method, Math. Program., 134 (2012), pp. 425-458.
20. P. L Écuyer, A unified view of the IPA, SF, and LR gradient estimation techniques, Management Sci., 36 (1990), pp. 1364-1383.
21. J. Mairal, F. Bach, J. Ponce, and G. Sapiro, Online dictionary learning for sparse coding, in Proceedings of ICML, 2009, pp. 689-696.
22. L. Mason, J. Baxter, P. Bartlett, and M. Frean, Boosting algorithms as gradient descent in function space, in Proceedings of NIPS 12, 1999, pp. 512-518.
23. A. S. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro, Robust stochastic approximation approach to stochastic programming, SIAM J. Optim., 19 (2009), pp. 1574-1609.
24. A. S. Nemirovskii and D. B. Yudin, Problem Complexity and Method Efficiency in Optimization, Wiley-Intersci. Ser. Discrete Math., John Wiley, New York, 1983.
25. 2), Dokl. Akad. Nauk. SSSR, 269 (1983), pp. 543-547.
26. Y. E. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course, Kluwer Academic Publishers, Boston, 2004.
27. Y. E. Nesterov, Primal-dual subgradient methods for convex problems, Math. Program., 120 (2006), pp. 221-259.
28. Y. E. Nesterov, Random Gradient-Free Minimization of Convex Functions, Technical report, Center for Operations Research and Econometrics (CORE), Catholic University of Louvain, 2010.
29. Y. E. Nesterov and J. P. Vial, Confidence level solutions for stochastic programming, J. IFAC, 44 (2008), pp. 1559-1568.
30. J. Nocedal and S. J. Wright, Numerical Optimization, Springer-Verlag, New York, 1999.
31. B. T. Polyak, New stochastic approximation type procedures, Avtomat. i Telemekh., 7 (1990), pp. 98-107.
32. B. T. Polyak and A. B. Juditsky, Acceleration of stochastic approximation by averaging, SIAM J. Control Optim., 30 (1992), pp. 838-855.
33. H. Robbins and S. Monro, A stochastic approximation method, Ann. Math. Statist., 22 (1951), pp. 400-407.
34. R. T. Rockafellar and R. J.-B. Wets, Variational Analysis, Grundlehren Math. Wiss., Springer-Verlag, New York, 1998.
35. R. Y. Rubinstein and A. Shapiro, Discrete Event Systems: Sensitivity Analysis and Stochastic Optimization by the Score Function Method, John Wiley, New York, 1993.
36. A. Sartenaer, S. Gratton and Ph. L. Toint, Recursive trust-region methods for multiscale nonlinear optimization, SIAM J. Optim., 19 (2008), pp. 414-444.
37. J. Sacks, Asymptotic distribution of stochastic approximation, Ann. Math. Statist., 29 (1958), pp. 373-409.
38. A. Shapiro, Monte Carlo sampling methods, in Stochastic Programming, A. Ruszczyński and A. Shapiro, eds., North-Holland, Amsterdam, 2003.
39. J. C. Spall, Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control, John Wiley, Hoboken, NJ, 2003.
40. L. N. Vicente, Worst case complexity of direct search, EURO J. Comput. Optim., 1 (2013), pp. 143-153.
41. F. Yousefian, A. Nedic, and U. V. Shanbhag, On stochastic gradient and subgradient methods with adaptive steplength sequences, Automatica, 48 (2012), pp. 56-67.