Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes

30th International Conference on Machine Learning, ICML 2013

Volumn , Issue PART 1, 2013, Pages 71-79

  Shamir, Ohad ^a   Zhang, Tong ^b  

^a MICROSOFT RESEARCH   (United States)

^b RUTGERS UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

LEARNING SYSTEMS; TIME VARYING NETWORKS;

CONVERGENCE RESULTS; CONVEX OBJECTIVES; MODERN APPLICATIONS; NONSMOOTH OPTIMIZATION; OBJECTIVE FUNCTIONS; STOCHASTIC GRADIENT DESCENT; STOCHASTIC OPTIMIZATION METHODS; SUBOPTIMALITY;

OPTIMIZATION;

EID: 84897554805     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (13)

1. Agarwal, A., Bartlett, P., Ravikumar, P., and Wainwright, M. Information-theoretic lower bounds on the oracle complexity of stochastic convex optimization. IEEE Transactions on Information Theory, 58(5):3235-3249, 2012.
2. Bach, F. and Moulines, E. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In NIPS, 2011.
3. Hazan, E. and Kale, S. Beyond the regret minimization barrier: An optimal algorithm for stochastic strongly-convex optimization. In COLT, 2011.
4. Hazan, E., Agarwal, A., and Kale, S. Logarithmic regret algorithms for online convex optimization. Machine Learning, 69(2-3):169-192, 2007. (Pubitemid 47574314)
5. Kushner, H. and Yin, G. Stochastic Approximation and Recursive Algorithms and Applications. Springer, 2nd edition, 2003.
6. Lacoste-Julien, S., Schmidt, M., and Bach, F. A simpler approach to obtaining an o(l/t) convergence rate for projected stochastic subgradient descent. CoRR, abs/1212.2002, 2012.
7. Ouyang, H. and Gray, A. Stochastic smoothing for nonsmooth minimizations: Accelerating sgd by exploiting structure. In ICML, 2012.
8. Rakhlin, A., Shamir, O., and Sridharan, K. Making gradient descent optimal for strongly convex stochastic optimization. CoRR, abs/1109.5647, 2011.
9. Shalev-Shwartz, S., Shamir, O., Srebro, N., and Sridharan, K. Stochastic convex optimization. In COLT, 2009.
10. Shalev-Shwartz, S., Singer, Y., Srebro, N., and Cotter, A. Pegasos: primal estimated sub-gradient solver for svm. Mathematical Programming, 127(1):3-30, 2011.
11. Shamir, O. Is averaging needed for strongly convex stochastic gradient descent? Open problem presented at COLT, 2012.
12. Zhang, T. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In ICML, 2004.
13. Zinkevich, M. Online convex programming and generalized infinitesimal gradient ascent. In ICML, 2003.