O(logT) projections for stochastic optimization of smooth and strongly convex functions

30th International Conference on Machine Learning, ICML 2013

Volumn , Issue PART 3, 2013, Pages 2158-2166

  Zhang, Lijun ^a   Yang, Tianbao ^b   Jin, Rong ^a   He, Xiaofei ^c  

^a Michigan State University   (United States)

^b GE GLOBAL RESEARCH   (United States)

^c ZHEJIANG UNIVERSITY   (China)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; FUNCTIONS; LEARNING SYSTEMS; OPTIMIZATION; STOCHASTIC SYSTEMS;

COMPUTATIONAL COSTS; CONVEX FUNCTIONS; EMPIRICAL STUDIES; OBJECTIVE FUNCTIONS; POSITIVE SEMIDEFINITE; RATE OF CONVERGENCE; STOCHASTIC OPTIMIZATIONS; STRONG CONVEXITIES;

ITERATIVE METHODS;

EID: 84897492359     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (35)

1. Agarwal, A., Bartlett, P.L., P.Ravikumar, and Wainwright, M.J. Information-theoretic lower bounds on the oracle complexity of stochastic convex optimization. IEEE Trans. Inf. Theory, 58(5):3235-3249, 2012.
2. Bartlett, P.L., Bousquet, O., and Mendelson, S. Local rademacher complexities. Ann. Stat., 33(4):1497-1537, 2005. (Pubitemid 41423979)
3. Cesa-Bianchi, N. and Lugosi, G. Prediction, Learning, and Games. Cambridge University Press, 2006.
4. Chang, C. and Lin, C. LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol., 2 (3):27:1-27:27, 2011.
5. Chen, X., Lin, Q., and Pena, J. Optimal regularized dual averaging methods for stochastic optimization. In NIPS 25, pp. 404-412, 2012.
6. Clarkson, K.L. Coresets, sparse greedy approximation, and the frank-wolfe algorithm. ACM Trans. Algorithms, 6 (4):63:1-63:30, 2010.
7. Cotter, A., Shamir, O., Srebro, N., and Sridharan, K. Better mini-batch algorithms via accelerated gradient methods. In NIPS 24, pp. 1647-1655, 2011.
8. Dekel, O., Gilad-Bachrach, R., Shamir, O., and Xiao, L. Optimal distributed online prediction. In Getoor, Lise and Scheffer, Tobias (eds.), ICML, pp. 713-720, 2011.
9. Dekel, O., Gilad-Bachrach, R., Shamir, O., and Xiao, L. Optimal distributed online prediction using minibatches. J. Mach. Learn. Res., 13:165-202, 2012.
10. Frank, M. and Wolfe, P. An algorithm for quadratic programming. Naval Research Logistics Quarterly, 3(1-2):95-110, 1956.
11. Ghadimi, S. and Lan, G. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: A generic algorithmic framework. SIAM J. Optim., 22(4):1469-1492, 2012.
12. Hazan, E. Sparse approximate solutions to semidefinite programs. In LATIN, pp. 306-316, 2008.
13. Hazan, E. and Kale, S. Beyond the regret minimization barrier: an optimal algorithm for stochastic strongly-convex optimization. In COLT, pp. 421-436, 2011.
14. Hazan, E. and Kale, S. Projection-free online learning. In ICML, pp. 521-528, 2012.
15. Hu, C., Kwok, J., and Pan, W. Accelerated gradient methods for stochastic optimization and online learning. In NIPS 22, pp. 781-789, 2009.
16. Jaggi, M. Revisiting frank-wolfe: Projection-free sparse convex optimization. In ICML, 2013.
17. Jin, R., Wang, S., and Zhou, Y. Regularized distance metric learning: Theory and algorithm. In NIPS 22, pp. 862-870, 2009.
18. Juditsky, A. and Nesterov, Y. Primal-dual subgradient methods for minimizing uniformly convex functions. Technical report, 2010.
19. Juditsky, A., Nemirovski, A., and Tauvel, C. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems, 1(1):17-58, 2011.
20. Lacoste-Julien, S., Jaggi, M., Schmidt, M., and Pletscher, P. Block-coordinate frank-wolfe optimization for structural svm. In ICML, 2013.
21. Lan, G. An optimal method for stochastic composite optimization. Math. Program., 133:365-397, 2012.
22. Levitin, E.S. and Polyak, B.T. Constrained minimization methods. USSR Computational Mathematics and Mathematical Physics, 6(5):1-50, 1966.
23. Mahdavi, M., Yang, T., Jin, R., Zhu, S., and Yi, J. Stochastic gradient descent with only one projection. In NIPS 25, pp. 503-511, 2012.
24. Nemirovski, A. Prox-method with rate of convergence o(1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM J. Optim., 15(1):229-251, 2005. (Pubitemid 40360669)
25. Nemirovski, A. and Yudin, D.B. Problem complexity and method efficiency in optimization. John Wiley & Sons Ltd, 1983.
26. Nemirovski, A., Juditsky, A., Lan, G., and Shapiro, A. Robust stochastic approximation approach to stochastic programming. SIAM J. Optim., 19(4):1574-1609, 2009.
27. Nesterov, Y. Introductory lectures on convex optimization: a basic course, volume 87 of Applied optimization. Kluwer Academic Publishers, 2004.
28. Nesterov, Y. Smooth minimization of non-smooth functions. Math. Program., 103(1):127-152, 2005. (Pubitemid 40537674)
29. Nesterov, Y. Gradient methods for minimizing composite objective function. Core discussion papers, 2007.
30. Rakhlin, A., Shamir, O., and Sridharan, K. Making gradient descent optimal for strongly convex stochastic optimization. In ICML, pp. 449-456, 2012.
31. Roux, N.L., Manzagol, P., and Bengio, Y. Topmoumoute online natural gradient algorithm. In NIPS 20, pp. 849-856, 2008.
32. Shalev-Shwartz, S., Shamir, O., Srebro, N., and Sridharan, K. Stochastic convex optimization. In COLT, 2009.
33. Shalev-Shwartz, S., Singer, Y., Srebro, N., and Cotter, A. Pegasos: primal estimated sub-gradient solver for svm. Math. Program., 127(1):3-30, 2011.
34. Smale, S. and Zhou, D. Geometry on probability spaces. Constr. Approx., 30:311-323, 2009.
35. Zhang, T. Solving large scale linear prediction problems using stochastic gradient descent algorithms. In ICML, pp. 919-926, 2004.