Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, II: Shrinking procedures and optimal algorithms

SIAM Journal on Optimization

Volumn 23, Issue 4, 2013, Pages 2061-2089

  Ghadimi, Saeed ^a   Lan, Guanghui ^a  

^a University of Florida   (United States)

Author keywords

Complexity; Convex optimization; Large deviation; Optimal method; Stochastic approximation; Strong convexity

Indexed keywords

COMPLEXITY; LARGE DEVIATIONS; OPTIMAL METHODS; STOCHASTIC APPROXIMATIONS; STRONG CONVEXITIES;

APPROXIMATION ALGORITHMS; APPROXIMATION THEORY; CONVEX OPTIMIZATION; SHRINKAGE; STOCHASTIC SYSTEMS;

OPTIMIZATION;

EID: 84892856128     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/110848876     Document Type: Article

References (29)

1. A. Auslender and M. Teboulle, Interior projection-like methods for monotone variational inequalities, Math. Program., 104 (2005), pp. 39-68.
2. A. Auslender and M. Teboulle, Interior gradient and proximal methods for convex and conic optimization, SIAM J. Optim., 16 (2006), pp. 697-725.
3. A. Benveniste, M. Métivier, and P. Priouret, Algorithmes adaptatifs et approximations stochastiques, Masson, Paris, 1987. English translation: Adaptive Algorithms and Stochastic Approximations, Springer-Verlag, Berlin, 1990.
4. Y. Ermoliev, Stochastic quasigradient methods and their application to system optimization, Stochastics, 9 (1983), pp. 1-36.
5. A. Gaivoronski, Nonstationary stochastic programming problems, Kybernetika, 4 (1978), pp. 89-92.
6. S. Ghadimi and G. Lan, Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, I: A generic algorithmic framework, SIAM J. Optim., 22 (2012), pp. 1469-1492.
7. C. Hu, J. T. Kwok, and W. Pan, Accelerated gradient methods for stochastic optimization and online learning, in Advances in Neural Information Processing Systems, 2009, pp. 781-789.
8. A. Juditsky and A. S. Nemirovski, Large deviations of vector-valued martingales in 2-smooth normed spaces, manuscript, Georgia Institute of Technology, Atlanta, GA, 2008. Available online at www2.isye.gatech.edu/ ~nemirovs/LargeDevSubmitted.pdf.
9. A. Juditsky and Y. E. Nesterov, Primal-dual subgradient methods for minimizing uniformly convex functions, manuscript; available online at http://hal.archives-ouvertes.fr/docs/00/50/89/33/PDF/Strong-hal.pdf.
10. A. J. Kleywegt, A. Shapiro, and T. Homem-de-Mello, The sample average approximation method for stochastic discrete optimization, SIAM J. Optim., 12 (2002), pp. 479-502.
11. H. J. Kushner and G. Yin, Stochastic Approximation and Recursive Algorithms and Applications, Appl. Math. (N.Y.) 35, Springer-Verlag, New York, 2003.
12. G. Lan, An optimal method for stochastic composite optimization, Math. Program., 133 (2012), pp. 365-397.
13. G. Lan, A. S. Nemirovski, and A. Shapiro, Validation analysis of mirror descent stochastic approximation method, Math. Program., 134 (2012), pp. 425-458.
14. S. Lee and S. Wright, Manifold identification in dual averaging for regularized stochastic online learning, J. Mach. Learn. Res., 13 (2012), pp. 1705-1744.
15. Q. Lin, X. Chen, and J. Peña, A Sparsity Preserving Stochastic Gradient Method for Composite Optimization, manuscript, Carnegie Mellon University, Pittsburgh, PA, 2011.
16. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro, Robust stochastic approximation approach to stochastic programming, SIAM J. Optim., 19 (2009), pp. 1574-1609.
17. A. S. Nemirovski and D. Yudin, Problem Complexity and Method Efficiency in Optimization, Wiley-Interscience Series in Discrete Mathematics, John Wiley, New York, 1983.
18. 2), Dokl. Akad. Nauk. SSSR, 269 (1983), pp. 543-547.
19. Y. E. Nesterov, Introductory Lectures on Convex Optimization: A basic course, Kluwer Academic Publishers, Boston, 2004.
20. G. C. Pflug, Optimization of Stochastic Models, in The Interface Between Simulation and Optimization, Kluwer, Boston, 1996.
21. B. T. Polyak, New stochastic approximation type procedures, Avtomat. i Telemekh., 7 (1990), pp. 98-107.
22. B. T. Polyak and A. B. Juditsky, Acceleration of stochastic approximation by averaging, SIAM J. Control Optim., 30 (1992), pp. 838-855.
23. H. Robbins and S. Monro, A stochastic approximation method, Ann. Math. Statistics, 22 (1951), pp. 400-407.
24. A. Ruszczyński and W. Syski, A method of aggregate stochastic subgradients with on-line stepsize rules for convex stochastic programming problems, Math. Programming Stud., 28 (1986), pp. 113-131.
25. A. Shapiro, Monte Carlo sampling methods, in Stochastic Programming, A. Ruszczyński and A. Shapiro, eds., Elsevier, Amsterdam, 2003, pp. 353-425.
26. S. Smale and Y. Yao, Online learning algorithms, Found. Comput. Math, 6 (2005), pp. 145-170.
27. J. C. Spall, Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control, John Wiley & Sons, Hoboken, NJ, 2003.
28. L. Xiao, Dual averaging methods for regularized stochastic learning and online optimization, J. Mach. Learn. Res., 11 (2010), pp. 2543-2596.
29. D. B. Yudin and A.S. Nemirovskii, Computational complexity of strictly convex programming, Eknomika i Matematicheskie Metody, 3 (1977), pp. 550-569.