An optimal method for stochastic composite optimization

Mathematical Programming

Volumn 133, Issue 1-2, 2012, Pages 365-397

  Lan, Guanghui ^a  

^a J Crayton Pruitt Family Department of Biomedical Engineering   (United States)

Author keywords

Complexity; Convex optimization; Large deviation; Optimal method; Quadratic penalty method; Stochastic approximation; Stochastic programming

Indexed keywords

COMPLEXITY; LARGE DEVIATIONS; OPTIMAL METHODS; PENALTY METHODS; STOCHASTIC APPROXIMATIONS;

ALGORITHMS; APPROXIMATION THEORY; CONVEX OPTIMIZATION; CONVEX PROGRAMMING; OPTIMIZATION; STOCHASTIC PROGRAMMING; STOCHASTIC SYSTEMS;

PROBLEM SOLVING;

EID: 84862273593     PISSN: 00255610     EISSN: 14364646     Source Type: Journal    
DOI: 10.1007/s10107-010-0434-y     Document Type: Article

References (51)

1. A. Auslender M. Teboulle 2006 Interior gradient and proximal methods for convex and conic optimization SIAM J. Optim. 16 697 725 2197553 1113.90118 10.1137/S1052623403427823
2. H.H. Bauschke J.M. Borwein P.L. Combettes 2003 Bregman monotone optimization algorithms SIAM J. Control Optim. 42 596 636 1982285 1049.90053 10.1137/S0363012902407120
3. Becker S,; Bobin J,; Candes E.: Nesta: A Fast and Accurate First-Order Method for Sparse Recovery, Manuscript. California Institute of Technology, Pasadena (2009)
4. A. Ben-Tal A. Nemirovski 2005 Non-euclidean restricted memory level method for large-scale convex optimization Math. Program. 102 407 456 2136222 1066.90079 10.1007/s10107-004-0553-4
5. Benveniste, A,; Métivier, M,; Priouret, P.: Algorithmes adaptatifs et approximations stochastiques. Masson, 1987. English translation: Adaptive Algorithms and Stochastic Approximations. Springer (1993)
6. Bertsekas D.: Nonlinear Programming, 2nd edn. Athena Scientific, New York (1999)
7. L.M. Bregman 1967 The relaxation method of finding the common point convex sets and its application to the solution of problems in convex programming USSR Comput. Math. Phys. 7 200 217 10.1016/0041-5553(67)90040-7
8. A. d'Aspremont 2008 Smooth optimization with approximate gradient SIAM J. Optim. 19 1171 1183 2460737 1180.90378 10.1137/060676386
9. A. d'Aspremont O. Banerjee L. El Ghaoue 2008 First-order methods for sparse covariance selection SIAM J. Matrix Anal. Appl. 30 56 66 2399568 1156.90423 10.1137/060670985
10. Y. Ermoliev 1983 Stochastic quasigradient methods and their application to system optimization Stochastics 9 1 36 703846 0512.90079 10.1080/ 17442508308833246
11. A. Gaivoronski 1978 Nonstationary stochastic programming problems Kybernetika 4 89 92
12. A. Juditsky A. Nazin A.B. Tsybakov N. Vayatis 2005 Recursive aggregation of estimators via the mirror descent algorithm with average Probl. Inf. Transm. 41 n.4 10.1007/s11122-006-0005-2
13. Juditsky, A,; Nemirovski, A,; Tauvel, C.: Solving Variational Inequalities with Stochastic Mirror-Prox Algorithm, Manuscript. Georgia Institute of Technology. Submitted to SIAM J. Control Optim Atlanta (2008)
14. A. Juditsky P. Rigollet A.B. Tsybakov 2008 Learning by mirror averaging Ann. Stat. 36 2183 2206 2458184 05368488 10.1214/07-AOS546
15. K.C. Kiwiel 1997 Proximal minimization methods with generalized bregman functions SIAM J. Control Optim. 35 1142 1168 1453294 0890.65061 10.1137/S0363012995281742
16. A.J. Kleywegt A. Shapiro T. Homem de Mello 2001 The sample average approximation method for stochastic discrete optimization SIAM J. Optim. 12 479 502 1885572 0991.90090 10.1137/S1052623499363220
17. Kushner H,; Yin J,; Shapiro G.: Approximation and Recursive Algorithms and Applications, vol. 35 of Applications of Mathematics. Springer, New York (2003)
18. Lan, G,; Lu, Z,; Monteiro, R.D.C.: Primal-dual first-order methods with O(1/ε)iteration-complexity for cone programming. Math. Program. (2009, to appear)
19. Lan, G,; Monteiro, R.D.C.: Iteration-Complexity of First-Order Penalty Methods for Convex Programming, Manuscript. School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta (June, 2008)
20. Lan G,; Monteiro, R.D.C.: Iteration-Complexity of First-Order Augmented Lagrangian Methods for Convex Programming, Manuscript. School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta (May, 2009)
21. Lan, G,; Nemirovski, A,; Shapiro, A.: Validation analysis of robust stochastic approximation method. submitted to Math. Program. (2008). http://www.optimization-online.org
22. Lewis A.S,; Wright S.J.: A Proximal Method for Composite Minimization, Manuscript. Cornell University, Ithaca (2009)
23. J. Linderoth A. Shapiro S. Wright 2006 The empirical behavior of sampling methods for stochastic programming Ann. Oper. Res. 142 215 241 2222918 1122.90391 10.1007/s10479-006-6169-8
24. Z. Lu 2009 Smooth optimization approach for sparse covariance selection SIAM J. Optim. 19 1807 1827 1179.90257 10.1137/070695915
25. Lu, Z,; Monteiro, R.D.C,; Yuan, M.: Convex Optimization Methods for Dimension Reduction and Coefficient Estimation in Multivariate Linear Regression, Manuscript. School of ISyE, Georgia Tech, Atlanta (January, 2008)
26. Z. Lu A. Nemirovski R.D.C. Monteiro 2007 Large-scale semidefinite programming via saddle point mirror-prox algorithm Math. Program. 109 211 237 2295141 1148.90009 10.1007/s10107-006-0031-2
27. W.K. Mak D.P. Morton R.K. Wood 1999 Monte carlo bounding techniques for determining solution quality in stochastic programs Oper. Res. Lett. 24 47 56 1683170 0956.90022 10.1016/S0167-6377(98)00054-6
28. Monteiro, R.D.C,; Svaiter B.F.: On the Complexity of the Hybrid Proximal Extragradient Method for the Iterates and the Ergodic Mean, Manuscript. School of ISyE, Georgia Tech, Atlanta (March, 2009)
29. A. Nemirovski 2004 Prox-method with rate of convergence o(1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems SIAM J. Optim. 15 229 251 2112984 1106.90059 10.1137/S1052623403425629
30. A. Nemirovski A. Juditsky G. Lan A. Shapiro 2009 Robust stochastic approximation approach to stochastic programming SIAM J. Optim. 19 1574 1609 2486041 1189.90109 10.1137/070704277
31. Nemirovski A,; Yudin D.: Problem Complexity and Method Efficiency in Optimization. Wiley-Interscience Series in Discrete Mathematics, vol. XV. Wiley, New York (1983)
32. 2). Doklady AN SSSR 269:543-547 (1983). Translated as Soviet Math. Docl
33. Nesterov Y.E.: Introductory Lectures on Convex Optimization: A Basic Course. Kluwer, Massachusetts (2004)
34. Y.E. Nesterov 2005 Smooth minimization of nonsmooth functions Math. Program. 103 127 152 2166537 1079.90102 10.1007/s10107-004-0552-5
35. Y.E. Nesterov 2006 Primal-dual subgradient methods for convex problems Math. Program. 120 221 259 2496434 10.1007/s10107-007-0149-x
36. Nesterov, Y.E.: Gradient Methods for Minimizing Composite Objective Functions. Technical report, Center for Operations Research and Econometrics (CORE), Catholic University of Louvain (2007, September)
37. Y.E. Nesterov 2007 Smoothing technique and its applications in semidefinite optimization Math. Program. 110 245 259 2313779 1126.90058 10.1007/s10107-006-0001-8
38. J. Peña 2008 Nash equilibria computation via smoothing techniques Optima 78 12 13
39. Pflug, G.C.: Optimization of Stochastic Models. In: The Interface Between Simulation and Optimization. Kluwer, Boston (1996)
40. B.T. Polyak 1990 New stochastic approximation type procedures Automat. i Telemekh 7 98 107 1071220
41. B.T. Polyak A.B. Juditsky 1992 Acceleration of stochastic approximation by averaging SIAM J. Control Optim. 30 838 855 1167814 0762.62022 10.1137/0330046
42. H. Robbins S. Monro 1951 A stochastic approximation method Ann. Math. Stat. 22 400 407 42668 0054.05901 10.1214/aoms/1177729586
43. Rockafellar R.T.: Convex Analysis. Princeton University Press, Princeton (1970)
44. A. Ruszczyński W. Sysk 1986 A method of aggregate stochastic subgradients with on-line stepsize rules for convex stochastic programming problems Math. Program. Study 28 113 131 0597.90064 10.1007/BFb0121128
45. Shapiro A.: Monte carlo sampling methods. In: Ruszczyński, A,; Shapiro, A. (eds) Stochastic Programming, North-Holland, Amsterdam (2003)
46. Shapiro A,; Nemirovski A.: On complexity of stochastic programming problems. In: Jeyakumar, V,; Rubinov, A.M. (eds) Continuous Optimization: Current Trends and Applications, pp. 111-144. Springer, Berlin (2005)
47. Spall J.C.: Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control. Wiley, Hoboken (2003)
48. V. Strassen 1965 The existence of probability measures with given marginals Ann. Math. Stat. 30 423 439 177430 10.1214/aoms/1177700153
49. M. Teboulle 1997 Convergence of proximal-like algorithms SIAM J. Optim. 7 1069 1083 1479615 0890.90151 10.1137/S1052623495292130
50. Tseng P.: On Accelerated Proximal Gradient Methods for Convex-Concave Optimization, Manuscript. University of Washington, Seattle (May 2008)
51. B. Verweij S. Ahmed J.A. Kleywegt G. Nemhauser A. Shapiro 2003 The sample average approximation method applied to stochastic routing problems: a computational study Comput. Optim. Appl. 24 289 333 1969156 1094.90029 10.1023/A:1021814225969