Competing with the empirical risk minimizer in a single pass

Journal of Machine Learning Research

Volumn 40, Issue 2015, 2015, Pages

  Frostig, Roy ^a   Ge, Rong ^b   Kakade, Sham M ^b   Sidford, Aaron ^c  

^a STANFORD UNIVERSITY   (United States)

^b MICROSOFT RESEARCH   (United States)

^c MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; COMPUTATION THEORY; SAMPLING;

ALGORITHM'S PERFORMANCE; COMPUTATIONAL CONSTRAINTS; COMPUTATIONAL RESOURCES; ESTIMATION STRATEGIES; LOGISTIC REGRESSIONS; REGULARITY ASSUMPTION; STATISTICAL CONVERGENCE; STREAMING ALGORITHM;

RISK PERCEPTION;

EID: 84984710701     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (36)

1. A. Agarwal and L. Bottou. A lower bound for the optimization of finite sums. Technical report, arXiv, 2014. URL http://arxiv.org/abs/1410.0723.
2. A. Agarwal, P. L. Bartlett, P. Ravikumar, and M. J.Wainwright. Information-theoretic lower bounds on the oracle complexity of stochastic convex optimization. IEEE Transactions on Information Theory, 58 (5): 3235-3249, May 2012.
3. D. Anbar. On Optimal Estimation Methods Using Stochastic Approximation Procedures. University of California, 1971. URL http://books.google.com/books?id=MmpHJwAACAAJ.
4. F. Bach. Self-concordant analysis for logistic regression. Electronic Journal of Statistics, 4:384-414, 2010.
5. F. Bach and E. Moulines. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In Neural Information Processing Systems (NIPS), 2011.
6. F. Bach and E. Moulines. Non-strongly-convex smooth stochastic approximation with convergence rate O (1=n). In Neural Information Processing Systems (NIPS), 2013.
7. L. Bottou and O. Bousquet. The tradeoffs of large scale learning. In Neural Information Processing Systems (NIPS), volume 20, pages 161-168, 2008.
8. A. Caponnetto and E. De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7 (3): 331-368, 2007.
9. A. Defossez and F. Bach. Constant step size least-mean-square: Bias-variance trade-offs and optimal sampling distributions. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2015.
10. A. Dieuleveut and F. Bach. Non-parametric stochastic approximation with large step sizes. Technical report, arXiv, 2014. URL http://arxiv.org/abs/1408.0361.
11. V. Fabian. Asymptotically efficient stochastic approximation; the RM case. Annals of Statistics, 1 (3), 1973.
12. E. Hazan and S. Kale. Beyond the regret minimization barrier: Optimal algorithms for stochastic strongly-convex optimization. Journal of Machine Learning Research, 15:2489-2512, 2014. URL http://jmlr.org/papers/v15/hazan14a.html.
13. D. Hsu, S. M. Kakade, and T. Zhang. A tail inequality for quadratic forms of subgaussian random vectors. Technical report, arXiv, 2011. URL http://arxiv.org/abs/1110.2842.
14. D. Hsu, S. M. Kakade, and T. Zhang. Tail inequalities for sums of random matrices that depend on the intrinsic dimension. Electronic Communications in Probability, 17 (14): 1-13, 2012.
15. D. Hsu, S. M. Kakade, and T. Zhang. Random design analysis of ridge regression. Foundations of Computational Mathematics, 14 (3): 569-600, June 2014.
16. R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Neural Information Processing Systems (NIPS), 2013.
17. A. Juditsky and Y. Nesterov. Primal-dual subgradient methods for minimizing uniformly convex functions, 2010. URL https://hal.archives-ouvertes.fr/hal-00508933.
18. H. J. Kushner and D. S. Clark. Stochastic Approximation Methods for Constrained and Unconstrained Systems. Springer-Verlag, 1978.
19. H. J. Kushner and G. Yin. Stochastic Approximation and Recursive Algorithms and Applications. Applications of mathematics. Springer, 2003. ISBN 9780387008943. URL http://books.google.com/books?id=-0bIieuUJGkC.
20. N. Le Roux, M. Schmidt, and F. Bach. A stochastic gradient method with an exponential convergence rate for finite training sets. In Neural Information Processing Systems (NIPS), 2012.
21. E. L. Lehmann and G. Casella. Theory of Point Estimation. Springer Texts in Statistics. Springer, 1998. ISBN 9780387985022.
22. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19 (4): 1574-1609, 2009.
23. A. S. Nemirovski and D. B. Yudin. Problem complexity and method efficiency in optimization. Wiley (Chichester and New York), 1983.
24. Y. Nesterov. Introductory Lectures on Convex Optimization: A Basic Course. Applied Optimization. Springer, 2004. ISBN 9781402075537.
25. B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30 (4): 838-855, July 1992. ISSN 0363-0129.
26. A. Rakhlin, O. Shamir, and K. Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In International Conference on Machine Learning, (ICML), 2012.
27. H. Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 22 (3): 400-407, Sept 1951.
28. D. Ruppert. Efficient Estimations from a Slowly Convergent Robbins-Monro Process. Technical report, ORIE, Cornell University, 1988. URL http://hdl.handle.net/1813/8664.
29. S. Shalev-Shwartz and T. Zhang. Stochastic dual coordinate ascent methods for regularized loss. Journal of Machine Learning Research (JMLR), 14 (1): 567-599, February 2013.
30. S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan. Learnability, stability and uniform convergence. Journal of Machine Learning Research, 11:2635-2670, December 2010.
31. O. Shamir. The sample complexity of learning linear predictors with the squared loss. Technical report, arXiv, 2014. URL http://arxiv.org/abs/1406.5143.
32. T. Strohmer and R. Vershynin. A randomized Kaczmarz algorithm with exponential convergence. Journal of Fourier Analysis and Applications, 15:262-278, 2009.
33. Pierre Tarres and Yuan Yao. Online learning as stochastic approximation of regularization paths: Optimality and almost-sure convergence. IEEE Transactions on Information Theory, 60 (9): 5716-5735, 2014.
34. A. W. van der Vaart. Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2000. ISBN 9780521784504.
35. Yuan Yao. On complexity issues of online learning algorithms. IEEE Transactions on Information Theory, 56 (12), 2010.
36. Yiming Ying and Massimiliano Pontil. Online gradient descent learning algorithms. Foundations of Computational Mathematics, 8 (5): 561-596, 2008.