Randomized smoothing for stochastic optimization

SIAM Journal on Optimization

Volumn 22, Issue 2, 2012, Pages 674-701

  Duchi, John C ^a   Bartlett, Peter L ^b   Wainwright, Martin J ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Convex programming; Distributed optimization; Nonsmooth optimization; Smoothing; Stochastic optimization

Indexed keywords

CONVERGENCE RATES; CONVEX OPTIMIZATION PROBLEMS; DECENTRALIZED OPTIMIZATION; DISTRIBUTED OPTIMIZATION; GRADIENT ESTIMATES; HIGH PROBABILITY; NON-SMOOTH; NONSMOOTH OPTIMIZATION; SMOOTHING; SMOOTHING TECHNIQUES; STATISTICAL ESTIMATION; STOCHASTIC OPTIMIZATION ALGORITHM; STOCHASTIC OPTIMIZATIONS;

ALGORITHMS; CONVEX OPTIMIZATION; CONVEX PROGRAMMING; GRADIENT METHODS; STOCHASTIC SYSTEMS;

OPTIMIZATION;

EID: 84865694209     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/110831659     Document Type: Article

References (37)

1. A. Agarwal, P. L. Bartlett, P. Ravikumar, and M. J. Wainwright, Information-theoretic lower bounds on the oracle complexity of convex optimization, IEEE Trans. Inform. Theory, 58 (2012), pp. 3235-3249.
2. K. Azuma, Weighted sums of certain dependent random variables, Tohoku Math. J., 68 (1967), pp. 357-367.
3. A. Ben-Tal and M. Teboulle, A smoothing technique for nondifferentiable optimization problems, in Optimization, Lecture Notes in Math. 1405, Springer-Verlag, Berlin, 1989, pp. 1-11.
4. D. P. Bertsekas, Stochastic optimization problems with nondifferentiable cost functionals, J. Optim. Theory Appl., 12 (1973), pp. 218-231.
5. L. M. Bregman, The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming, USSR Comput. Math. Math. Phys., 7 (1967), pp. 200-217.
6. P. Brucker, An O(n) algorithm for quadratic knapsack problems, Oper. Res. Lett., 3 (1984), pp. 163-166.
7. V. Buldygin and Y. Kozachenko, Metric Characterization of Random Variables and Random Processes, Transl. Math. Monogr. 188, AMS, Providence, RI, 2000.
8. C. Chen and O. L. Mangasarian, A class of smoothing functions for nonlinear and mixed complementarity problems, Comput. Optim. Appl., 5 (1996), pp. 97-138. (Pubitemid 126708355)
9. O. Dekel, R. Gilad-Bachrach, O. Shamir, and L. Xiao, Optimal distributed online prediction using mini-batches, J. Mach. Learn. Res., 13 (2012), pp. 165-202.
10. J. C. Duchi, P. L. Bartlett, and M. J. Wainwright, Randomized Smoothing for Stochastic Optimization, preprint, http://arxiv.org/abs/1103.4296, 2011.
11. J. C. Duchi, S. Shalev-Shwartz, Y. Singer, and A. Tewari, Composite objective mirror descent, in Proceedings of the Twenty Third Annual Conference on Computational Learning Theory, 2010.
12. Y. M. Ermoliev, On the stochastic quasi-gradient method and stochastic quasi-Feyer sequences, Kibernetika, 2 (1969), pp. 72-83.
13. J. Hiriart-Urruty and C. Lemar'echal, Convex Analysis and Minimization Algorithms I, Springer, New York, 1996.
14. W. Hoeffding, Probability inequalities for sums of bounded random variables, J.Amer. Statist. Assoc., 58 (1963), pp. 13-30.
15. P. J. Huber, Robust Statistics, John Wiley and Sons, New York, 1981.
16. A. Juditsky, A. Nemirovski, and C. Tauvel, Solving Variational Inequalities with the Stochastic Mirror-Prox Algorithm, preprint, http://arxiv.org/abs/0809.0815, 2008.
17. V. Katkovnik and Y. Kulchitsky, Convergence of a class of random search algorithms, Automat. Remote Control, 33 (1972), pp. 1321-1326.
18. H. Lakshmanan and D. P. de Farias, Decentralized resource allocation in dynamic networks of agents, SIAM J. Optim., 19 (2008), pp. 911-940.
19. G. Lan, An optimal method for stochastic composite optimization, Math. Program., 133 (2012), pp. 365-397.
20. G. Lan, A. Nemirovski, and A. Shapiro, Validation analysis of robust stochastic approximation method, Math. Program., to appear; also available online from http://www.ise.ufl.edu/glan/files/2011/12/Validate-SA-rev-Oct28- 2010-final.pdf.
21. M. Ledoux and M. Talagrand, Probability in Banach Spaces, Springer, New York, 1991.
22. C. Lemar'echal and C. Sagastiz'abal, Practical aspects of the Moreau-Yosida regularization: Theoretical preliminaries, SIAM J. Optim., 7 (1997), pp. 367-385.
23. C. Manning, P. Raghavan, and H. Schütze, Introduction to Information Retrieval, Cambridge University Press, Cambridge, UK, 2008.
24. S. Negahban and M. J. Wainwright, Estimation of (near) low-rank matrices with noise and high-dimensional scaling, Ann. Statist., 39 (2011), pp. 1069-1097.
25. A. Nemirovski and D. Yudin, Problem complexity and method efficiency in optimization, Wiley, New York, 1983.
26. Y. Nesterov, Smooth minimization of nonsmooth functions, Math. Program., 103 (2005), pp. 127-152.
27. Y. Nesterov, Primal-dual subgradient methods for convex problems, Math. Program., 120 (2009), pp. 261-283.
28. B. T. Polyak and J. Tsypkin, Robust identification, Automatica J. IFAC, 16 (1980), pp. 53- 63.
29. H. Robbins and S. Monro, A stochastic approximation method, Ann. Math. Statist., 22 (1951), pp. 400-407.
30. R. T. Rockafellar and R. J. B. Wets, Variational Analysis, Springer, New York, 1998.
31. R. Y. Rubinstein, Simulation and the Monte Carlo Method, Wiley, New York, 1981.
32. S. Shalev-Shwartz, Y. Singer, and A. Ng, Online and batch learning of pseudo-metrics, in Proceedings of the Twenty-First International Conference on Machine Learning, 2004.
33. P. Tseng, On Accelerated Proximal Gradient Methods for Convex-Concave Optimization, http://www.math.washington.edu/∼tseng/papers/apgm.pdf, 2008.
34. A. W. van der Vaart and J. A. Wellner, Weak Convergence and Empirical Processes: With Applications to Statistics, Springer, New York, 1996.
35. L. Xiao, Dual averaging methods for regularized stochastic learning and online optimization, J. Mach. Learn. Res., 11 (2010), pp. 2543-2596.
36. E. Xing, A. Ng, M. Jordan, and S. Russell, Distance Metric Learning, with Application to Clustering with Side-Information, Adv. Neural Inf. Process. Syst. 15, MIT Press, Cambridge, MA, 2003.
37. F. Yousefian, A. Nedi'c, and U. V. Shanbhag, On stochastic gradient and subgradient methods with adaptive steplength sequences, Automatica J. IFAC, 48 (2012), pp. 56-67.