Non-asymptotic analysis of stochastic approximation algorithms for machine learning

Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011, NIPS 2011

Volumn , Issue , 2011, Pages

  Bach, Francis ^a   Moulines, Eric ^b  

^a ECOLE NORMALE SUPÉRIEURE   (France)

^b INSTITUT TELECOM   (France)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION ALGORITHMS; ARTIFICIAL INTELLIGENCE; LEARNING ALGORITHMS; LEARNING SYSTEMS; LEAST SQUARES APPROXIMATIONS; OPERATIONS RESEARCH; OPTIMIZATION; STOCHASTIC SYSTEMS;

CONVEX OBJECTIVES; KERNEL LOGISTIC REGRESSION; MACHINE LEARNING ALGORITHMS; MACHINE-LEARNING; MINIMISATION; NON-ASYMPTOTIC; OBJECTIVE FUNCTIONS; STANDARD MACHINES; STOCHASTIC APPROXIMATION ALGORITHMS; UNBIASED ESTIMATES;

ASYMPTOTIC ANALYSIS;

EID: 85162480829     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. M. N. Broadie, D. M. Cicek, and A. Zeevi. General bounds and finite-time improvement for stochastic approximation algorithms. Technical report, Columbia University, 2009.
2. H. J. Kushner and G. G. Yin. Stochastic approximation and recursive algorithms and applications. Springer-Verlag, second edition, 2003.
3. O. Yu. Kul'chitskiǐ and A. È. Mozgovoǐ. An estimate for the rate of convergence of recurrent robust identification algorithms. Kibernet. i Vychisl. Tekhn., 89:36-39, 1991.
4. B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838-855, 1992.
5. D. Ruppert. Efficient estimations from a slowly convergent Robbins-Monro process. Technical Report 781, Cornell University Operations Research and Industrial Engineering, 1988.
6. V. Fabian. On asymptotic normality in stochastic approximation. The Annals of Mathematical Statistics, 39(4):1327-1332, 1968.
7. Y. Nesterov and J. P. Vial. Confidence level solutions for stochastic programming. Automatica, 44(6):1559-1568, 2008.
8. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574-1609, 2009.
9. L. Bottou and Y. Le Cun. On-line learning for very large data sets. Applied Stochastic Models in Business and Industry, 21(2):137-151, 2005.
10. L. Bottou and O. Bousquet. The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems (NIPS), 20, 2008.
11. S. Shalev-Shwartz and N. Srebro. SVM optimization: inverse dependence on training set size. In Proc. ICML, 2008.
12. S. Shalev-Shwartz, Y. Singer, and N. Srebro. Pegasos: Primal estimated sub-gradient solver for svm. In Proc. ICML, 2007.
13. S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan. Stochastic convex optimization. In Conference on Learning Theory (COLT), 2009.
14. L. Xiao. Dual averaging methods for regularized stochastic learning and online optimization. Journal of Machine Learning Research, 9:2543-2596, 2010.
15. J. Duchi and Y. Singer. Efficient online and batch learning using forward backward splitting. Journal of Machine Learning Research, 10:2899-2934, 2009.
16. J. M. Borwein and A. S. Lewis. Convex Analysis and Nonlinear Optimization: Theory and Examples. Springer, 2006.
17. R. Durrett. Probability: theory and examples. Duxbury Press, third edition, 2004.
18. B. Schölkopf and A. J. Smola. Learning with Kernels. MIT Press, 2001.
19. J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, 2004.
20. Y. Nesterov. Introductory lectures on convex optimization: a basic course. Kluwer Academic Publishers, 2004.
21. K. Sridharan, N. Srebro, and S. Shalev-Shwartz. Fast rates for regularized objectives. Advances in Neural Information Processing Systems, 22, 2008.
22. N. N. Vakhania, V. I. Tarieladze, and S. A. Chobanyan. Probability distributions on Banach spaces. Reidel, 1987.
23. F. Bach and E. Moulines. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Technical Report 00608041, HAL, 2011.
24. A.S. Nemirovsky and D.B. Yudin. Problem complexity and method efficiency in optimization. Wiley & Sons, 1983.
25. A. Agarwal, P. L. Bartlett, P. Ravikumar, and M. J. Wainwright. Information-theoretic lower bounds on the oracle complexity of convex optimization, 2010. Tech. report, Arxiv 1009.0571.
26. E. Hazan and S. Kale. Beyond the regret minimization barrier: an optimal algorithm for stochastic strongly-convex optimization. In Proc. COLT, 2001.
27. G. Casella and R. L. Berger. Statistical Inference. Duxbury Press, 2001.
28. I. Steinwart. On the influence of the kernel on the consistency of support vector machines. Journal of Machine Learning Research, 2:67-93, 2002.