Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: A generic algorithmic framework

SIAM Journal on Optimization

Volumn 22, Issue 4, 2012, Pages 1469-1492

  Ghadimi, Saeed ^a   Lan, Guanghui ^a  

^a University of Florida   (United States)

Author keywords

Complexity; Convex optimization; Large deviation; Stochastic approximation; Stochastic programming

Indexed keywords

ALGORITHMIC FRAMEWORK; ASYMPTOTICALLY OPTIMAL; COMPLEXITY; COMPOSITE OPTIMIZATION; CONVERGENCE RATES; CONVEX PROBLEMS; LARGE DEVIATIONS; NUMBER OF ITERATIONS; OPTIMAL RATE; STEP SIZE; STOCHASTIC APPROXIMATION ALGORITHMS; STOCHASTIC APPROXIMATIONS;

APPROXIMATION ALGORITHMS; APPROXIMATION THEORY; CONVEX OPTIMIZATION; STOCHASTIC PROGRAMMING; STOCHASTIC SYSTEMS;

OPTIMIZATION;

EID: 84871576447     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/110848864     Document Type: Article

References (41)

1. A. Auslender and M. Teboulle, Interior gradient and proximal methods for convex and conic optimization, SIAM J. Optim., 16 (2006), pp. 697-725.
2. H. H. Bauschke, J. M. Borwein, and P. L. Combettes, Bregman monotone optimization algorithms, SIAM J. Control Optim., 42 (2003), pp. 596-636.
3. A. Ben-Tal and A. Nemirovski, Non-Euclidean restricted memory level method for large-scale convex optimization, Math. Program., 102 (2005), pp. 407-456.
4. A. Ben-Tal, T. Margalit, and A. Nemirovski, The ordered subsets mirror descent optimization method with applications to tomography, SIAM J. Optim., 12 (2001), pp. 79-108.
5. A. Benveniste, M. M'etivier, and P. Priouret, Algorithmes adaptatifs et approximations stochastiques, Masson, Paris, 1987; Adaptive Algorithms and Stochastic Approximations, Springer-Verlag, New York, 1993.
6. L. M. Bregman, The relaxation method of finding the common point convex sets and its application to the solution of problems in convex programming, USSR Comput. Math. Phys., 7 (1967), pp. 200-217.
7. K. L. Chung, On a stochastic approximation method, Ann. Math. Statist., 25 (1954), pp. 463- 483.
8. C. D. Dang and G. Lan, On the Convergence Properties of Non-Euclidean Extragradient Methods for Variational Inequalities with Generalized Monotone Operators, Department of Industrial and Systems Engineering, University of Florida, Gainesville, 2012; also available online from http://www.optimization- online.org.
9. Y. Ermoliev, Stochastic quasigradient methods and their application to system optimization, Stochastics, 9 (1983), pp. 1-36.
10. A. Gaivoronski, Nonstationary stochastic programming problems, Kybernetika, 4 (1978), pp. 89-92.
11. S. Ghadimi and G. Lan, Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, II: Shrinking procedures and optimal algorithms, SIAM J. Optim., submitted.
12. C. Hu, J. T. Kwok, and W. Pan, Accelerated gradient methods for stochastic optimization and online learning, in Proceedings of NIPS, 2009.
13. A. Juditsky, A. Nazin, A. B. Tsybakov, and N. Vayatis, Recursive aggregation of estimators via the mirror descent algorithm with average, Probl. Inf. Transm., 41 (2005), p. n.4.
14. A. Juditsky and A. Nemirovski, First-order methods for nonsmooth convex large-scale optimization, I: General purpose methods, in Optimization for Machine Learning, S. Sra, S. Nowozin, and S. J. Wright, eds., MIT Press, Cambridge, MA, 2011.
15. A. Juditsky and Y. Nesterov, Primal-Dual Subgradient Methods for Minimizing Uniformly Convex Functions, manuscript.
16. A. Juditsky, P. Rigollet, and A. B. Tsybakov, Learning by mirror averaging, Ann. Statist., 36 (2008), pp. 2183-2206.
17. K. C. Kiwiel, Proximal minimization methods with generalized Bregman functions, SIAM J. Control Optim., 35 (1997), pp. 1142-1168.
18. A. J. Kleywegt, A. Shapiro, and T. H. de Mello, The sample average approximation method for stochastic discrete optimization, SIAM J. Optim., 12 (2001), pp. 479-502.
19. H. J. Kushner and G. Yin, Stochastic Approximation and Recursive Algorithms and Applications, Appl. Math. 35, Springer-Verlag, New York, 2003.
20. G. Lan, An optimal method for stochastic composite optimization, Math. Program., to appear; DOI: 10.1007/s10107-010-0434-y.
21. G. Lan, A. Nemirovski, and A. Shapiro, Validation analysis of mirror descent stochastic approximation method, Math. Program., to appear.
22. A. S. Lewis and S. J. Wright, A Proximal Method for Composite Minimization, manuscript, Cornell University, Ithaca, NY, 2009.
23. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro, Robust stochastic approximation approach to stochastic programming, SIAM J. Optim., 19 (2009), pp. 1574-1609.
24. A. Nemirovski and D. Yudin, Problem Complexity and Method Efficiency in Optimization, Wiley-Intersci. Ser. Discrete Math. 15, John Wiley, New York, 1983.
25. Y. E. Nesterov, A method for unconstrained convex minimization problem with the rate of convergence O(1/k2), Dokl. Akad. Nauk. SSSR, 269 (1983), pp. 543-547.
26. Y. E. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course, Kluwer Academic, Norwell, MA, 2004.
27. Y. E. Nesterov, Primal-dual subgradient methods for convex problems, Math. Program., 120 (2006), pp. 221-259.
28. Y. E. Nesterov, Gradient Methods for Minimizing Composite Objective Functions, Technical report, Center for Operations Research and Econometrics (CORE), Catholic University of Louvain, Belgium, 2007.
29. G. Ch. Pflug, Optimization of stochastic models, in The Interface Between Simulation and Optimization, Kluwer, Boston, 1996.
30. B. T. Polyak, New stochastic approximation type procedures, Avtomat. i Telemekh., 7 (1990), pp. 98-107.
31. B. T. Polyak and A. B. Juditsky, Acceleration of stochastic approximation by averaging, SIAM J. Control Optim., 30 (1992), pp. 838-855.
32. H. Robbins and S. Monro, A stochastic approximation method, Ann. Math. Statist., 22 (1951), pp. 400-407.
33. A. Ruszczy'nski and W. Sysk, A method of aggregate stochastic subgradients with on-line stepsize rules for convex stochastic programming problems, Math. Program. Study, 28 (1986), pp. 113-131.
34. J. Sacks, Asymptotic distribution of stochastic approximation, Ann. Math. Statist., 29 (1958), pp. 373-409.
35. S. Shalev-Shwartz, Y. Singer, and N. Srebro, Pegasos: Primal estimated sub-gradient solver for svm, in Proceedings of ICML, 2007, pp. 807-814.
36. A. Shapiro, Monte Carlo sampling methods, in Stochastic Programming, A. Ruszczy'nski and A. Shapiro, eds., North-Holland, Amsterdam, 2003.
37. S. Smale and Y. Yao, Online learning algorithms, Found. Comput. Math, 6 (2005), pp. 145- 170.
38. J. C. Spall, Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control, John Wiley, Hoboken, NJ, 2003.
39. M. Teboulle, Convergence of proximal-like algorithms, SIAM J. Optim., 7 (1997), pp. 1069- 1083.
40. P. Tseng, On Accelerated Proximal Gradient Methods for Convex-Concave Optimization, manuscript, University of Washington, Seattle, 2008.
41. D. B. Yudin and A. S. Nemirovskii, Computational complexity of strictly convex programming, Eknomika i Matematicheskie Metody, 3 (1977), pp. 550-569.