Stochastic smoothing for nonsmooth minimizations: Accelerating SGD by exploiting structure

Proceedings of the 29th International Conference on Machine Learning, ICML 2012

Volumn 1, Issue , 2012, Pages 33-40

  Ouyang, Hua ^a   Gray, Alexander ^a  

^a Georgia Institute of Technology   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CENTRAL PROBLEMS; EMPIRICAL COMPARISON; FAST RATE; LOSS FUNCTIONS; NON-SMOOTH; NOVEL ALGORITHM; OPTIMAL CONVERGENCE; STOCHASTIC ALGORITHMS; STOCHASTIC GRADIENT DESCENT; SUBGRADIENT DESCENT;

ALGORITHMS; LEARNING SYSTEMS; OPTIMIZATION;

STOCHASTIC SYSTEMS;

EID: 84867137256     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (31)

1. Agarwal, Alekh, Bartlett, Peter L., Ravikumar, P., and Wainwright, Martin J. Information-theoretic lower bounds on the oracle complexity of stochastic convex optimization. Information Theory, IEEE Trans., 2012.
2. Bach, Francis and Moulines, Eric. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In NIPS, 2011.
3. Beck, Amir and Teboulle, Marc. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci., 2(1):193-202, 2009.
4. Bottou, Leon. Stochastic gradient descent 2.0. URL http://leon.bottou. org/projects/sgd.
5. Boucheron, Stéphane, Bousquet, Olivier, and Lugosi, Gábor. Theory of classification: A survey of some recent advances. ESAIM: Probability and Statistics, 9:323-375, 2005.
6. Daubechies, I., Defrise, M., and Mol, C. De. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics, 57(11):14131457, 2004.
7. Dekel, Ofer, Gilad-Bachrach, Ran, Shamir, Ohad, and Xiao, Lin. Optimal distributed online prediction using mini-batches. arXiv, 2010. URL http://arxiv.org/abs/1012.1367.
8. Duchi, John and Singer, Yoram. Efficient online and batch learning using forward backward splitting. JMLR, (10): 2899-2934, 2009.
9. Duchi, John, Bartlett, Peter L., and Wainwright, Martin J. Randomized smoothing for stochastic optimization. arXiv, 2011. URL http://arxiv.org/abs/ 1103.4296.
10. Hastie, Trevor, Tibshirani, Robert, and Friedman, Jerome. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2nd edition, 2009.
11. Hazan, Elad and Kale, Satyen. Beyond the regret minimization barrier: an optimal algorithm for stochastic strongly-convex optimization. In COLT, 2011.
12. Hu, Chonghai, Kwok, James T., and Pan, Weike. Accelerated gradient methods for stochastic optimization and online learning. In NIPS 22, 2009.
13. Huber, Peter J. Robust estimation of a location parameter. Annals of Mathematical Statistics, 35(1):73-101, 1964.
14. Lan, G. and Ghadimi, S. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, i: a generic algorithmic framework. SIAM J. on Optimization, 2011.
15. Lan, Guanghui. An optimal method for stochastic composite optimization. Mathematical Programming, 2010. doi: DOI10.1007/s10107-010-0434-y.
16. Lions, P. L. and Mercier, B. Splitting algorithms for the sum of two nonlinear operators. SIAM J. on Numerical Analysis, 16(6):964-979, 1979.
17. Nemirovski, A. and Yudin, D. Problem Complexity and Method Efficiency in Optimization. John Wiley and Sons, 1983.
18. Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. Robust stochastic approximation approach to stochastic programming. SIAM J. on Optimization, 19(4):1574-1609, 2009.
19. Nesterov, Yurii. Introductory Lectures on Convex Optimization, A Basic Course. Kluwer Academic Publishers, 2004.
20. Nesterov, Yurii. Excessive gap technique in nonsmooth convex minimization. SIAM J. Optim., 16(1):235-249, 2005a.
21. Nesterov, Yurii. Smooth minimization of non-smooth functions. Math. Program., Ser. A, 103:127-152, 2005b.
22. Nesterov, Yurii. Gradient methods for minimizing composite objective function. Technical Report CORE DISCUSSION PAPER 2007/76, 2007a.
23. Nesterov, Yurii. Smoothing technique and its applications in semidefinite optimization. Mathematical Programming, 110(2):245-259, 2007b.
24. Polyak, Boris T. and Juditsky, Anatoli B. Acceleration of stochastic approximation by averaging. SIAM J. on Control and Optimization, 30(4):838-855, 1992.
25. Robbins, Herbert and Monro, Sutton. A stochastic approximation method. The Annals of Mathematical Statistics, 22(3):400-407, 1951.
26. Shalev-Shwartz, Shai, Singer, Yoram, and Srebro, Nathan. Pegasos: Primal estimated sub-gradient solver for svm. In ICML, 2007.
27. Shalev-Shwartz, Shai, Shamir, Ohad, Srebro, Nathan, and Sridharan, Karthik. Stochastic convex optimization. In COLT, 2009.
28. Shamir, Ohad. Making gradient descent optimal for strongly convex stochastic optimization. In OPT 2011, 2011. URL http://arxiv.org/abs/1109.5647.
29. Wright, S. J., Nowak, R. D., and Figueiredo, M. A. T. Sparse reconstruction by separable approximation. IEEE Transactions on Signal Processing, 57(7): 2479-2493, 2009.
30. Xiao, Lin. Dual averaging methods for regularized stochastic learning and online optimization. JMLR, 11:2543-2596, 2010.
31. Xu, Wei. Towards optimal one pass large scale learning with averaged stochastic gradient descent. arXiv, 2011. URL http://arxiv.org/abs/1107.2490.