Linear convergence with condition number independent access of full gradients

Advances in Neural Information Processing Systems

Volumn , Issue , 2013, Pages

  Zhang, Lijun ^a   Mahdavi, Mehrdad ^a   Jin, Rong ^a  

^a Michigan State University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; CONVEX OPTIMIZATION; NUMBER THEORY; OPTIMIZATION;

CONDITION NUMBERS; GRADIENT BASED ALGORITHM; LINEAR CONVERGENCE; OBJECTIVE FUNCTIONS; OPTIMAL ITERATIONS; OPTIMAL SOLUTIONS; OPTIMIZATION PROBLEMS; STOCHASTIC GRADIENT;

ITERATIVE METHODS;

EID: 84898971059     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (26)

1. A. Agarwal, P. L. Bartlett, P. Ravikumar, and M. J. Wainwright. Information-theoretic lower bounds on the oracle complexity of stochastic convex optimization. IEEE Transactions on Information Theory, 58(5):3235-3249, 2012.
2. D. P. Bertsekas. A new class of incremental gradient methods for least squares problems. SIAM Journal on Optimization, 7(4):913-926, 1997.
3. S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
4. N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.
5. M. Friedlander and M. Schmidt. Hybrid deterministic-stochastic methods for data fitting. SIAM Journal on Scientific Computing, 34(3):A1380-A1405, 2012.
6. S. Ghadimi and G. Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: a generic algorithmic framework. SIAM Journal on Optimization, 22(4):1469-1492, 2012.
7. A. Gittens and J. A. Tropp. Tail bounds for all eigenvalues of a sum of random matrices. ArXiv e-prints, arXiv:1104.4513, 2011.
8. T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York, 2009.
9. E. Hazan and S. Kale. Beyond the regret minimization barrier: an optimal algorithm for stochastic strongly-convex optimization. In Proceedings of the 24th Annual Conference on Learning Theory, pages 421-436, 2011.
10. A. Juditsky and Y. Nesterov. Primal-dual subgradient methods for minimizing uniformly convex functions. Technical report, 2010.
11. G. Lan. An optimal method for stochastic composite optimization. Mathematical Programming, 133:365-397, 2012.
12. K. Marti. On solutions of stochastic programming problems by descent procedures with stochastic and deterministic directions. Methods of Operations Research, 33:281-293, 1979.
13. K. Marti and E. Fuchs. Rates of convergence of semi-stochastic approximation procedures for solving stochastic optimization problems. Optimization, 17(2):243-265, 1986.
14. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574-1609, 2009.
15. A. Nemirovski and D. B. Yudin. Problem complexity and method efficiency in optimization. John Wiley & Sons Ltd, 1983.
16. Y. Nesterov. A method for unconstrained convex minimization problem with the rate of convergence O(1/k2). Doklady AN SSSR (translated as Soviet. Math. Docl.), 269:543-547, 1983.
17. Y. Nesterov. Introductory lectures on convex optimization: a basic course, volume 87 of Applied opti-mization. Kluwer Academic Publishers, 2004.
18. Y. Nesterov. Smooth minimization of non-smooth functions. Mathematical Programming, 103(1):127-152, 2005.
19. Y. Nesterov. Gradient methods for minimizing composite objective function. Core discussion papers, 2007.
20. D. P. Palomar and Y. C. Eldar, editors. Convex Optimization in Signal Processing and Communications. 2010, Cambridge University Press.
21. A. Rakhlin, O. Shamir, and K. Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In Proceedings of the 29th International Conference on Machine Learning, pages 449-456, 2012.
22. N. L. Roux, M. Schmidt, and F. Bach. A stochastic gradient method with an exponential convergence rate for finite training sets. In Advances in Neural Information Processing Systems 25, pages 2672-2680, 2012.
23. S. Shalev-Shwartz and T. Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14:567-599, 2013.
24. S. Sra, S. Nowozin, and S. J. Wright, editors. Optimization for Machine Learning. The MIT Press, 2011.
25. Q.Wu and D.-X. Zhou. Svm soft margin classifiers: Linear programming versus quadratic programming. Neural Computation, 17(5):1160-1187, 2005.
26. L. Zhang, T. Yang, R. Jin, and X. He. O(log T) projections for stochastic optimization of smooth and strongly convex functions. In Proceedings of the 30th International Conference on Machine Learning (ICML), pages 621-629, 2013.