Online learning as stochastic approximation of regularization paths: Optimality and almost-sure convergence

IEEE Transactions on Information Theory

Volumn 60, Issue 9, 2014, Pages 5716-5735

  Tarres, Pierre ^a,b   Yao, Yuan ^c  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITÉ PARIS DAUPHINE   (France)

^c PEKING UNIVERSITY   (China)

Author keywords

Online learning; regularization path; reproducing kernel Hilbert space; stochastic approximations

Indexed keywords

APPROXIMATION THEORY; COMPUTATION THEORY; LEARNING ALGORITHMS; MULTIMEDIA SERVICES; OPTIMIZATION;

BERNSTEIN-TYPE INEQUALITIES; BIAS VARIANCE DECOMPOSITION; MARTINGALE DIFFERENCE SEQUENCES; ONLINE LEARNING; ONLINE LEARNING ALGORITHMS; REGULARIZATION PATHS; REPRODUCING KERNEL HILBERT SPACES; STOCHASTIC APPROXIMATIONS;

E-LEARNING;

EID: 84906663901     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2014.2332531     Document Type: Article

References (38)

1. A. Agarwal, P. L. Bartlett, P. Ravikumar, and M. J. Wainwright, "Information-Theoretic lower bounds on the oracle complexity of convex optimization," IEEE Trans. Inform. Theory, vol. 58, no. 5, pp. 3235-3249, May 2012
2. N. Aronszajn, "Theory of reproducing kernels," Trans. Amer. Math. Soc., vol. 68, no. 3, pp. 337-404, 1950
3. F. R. Bach and E. Moulines, "Non-Asymptotic analysis of stochastic approximation algorithms for machine learning," in Proc. Adv. Neural Inform. Process. Syst. (NIPS), 2011, pp. 451-459
4. M. Benaim, "Dynamics of stochastic approximation algorithms," in Séminaire de Probabilités, XXXIII (Lecture notes in Mathematics). Berlin, Germany: Springer-Verlag, pp. 1-68
5. A. Berlinet and C. Thomas-Agnan, Reproducing Kernel Hilbert Spaces in Probability and Statistics. Boston, MA, USA: Kluwer, 2004
6. D. P. Bertsekas, "Incremental proximal methods for large scale convex optimization," Math. Program., vol. 129, no. 2, pp. 163-195, Oct. 2011
7. A. Caponnetto and E. De Vito, "Optimal rates for regularized least squares algorithm," Found. Comput. Math., vol. 7, no. 3, pp. 331-368, Jul. 2007
8. C. Carmeli, E. De Vito, and A. Toigo, "Vector valued reproducing kernel hilbert spaces integrable, functions and mercer theorem," Anal. Appl., vol. 4, pp. 377-408, 2006
9. F. Cucker and S. Smale, "On the mathematical foundations of learning," Bull. Amer. Math. Soc., vol. 29, no. 1, pp. 1-49, 2002
10. I. Daubechies, Ten Lectures on Wavelets. Philadelphia, PA, USA: SIAM, 1992
11. M. Duflo, Algorithmes Stochastiques. Berlin, Germany: Springer-Verlag, 1996
12. B. Efron, I. Johnstone, T. Hastie, and R. Tibshirani, "Least angle regression," Ann. Statist., vol. 32, no. 2, pp. 407-451, 2004
13. H. W. Engl, M. Hanke, and A. Neubauer, "Regularization of inverse problems," in Mathematics and its Applications. Boston, MA, USA: Kluwer, 1996
14. T. Evgeniou, M. Pontil, and T. Poggio, "Regularized networks and support vector machines," Adv. Comput. Math., vol. 13, no. 1, pp. 1-50, Apr. 2000
15. L. Györfi, M. Kohler, A. Krzyzak, and H. Walk, A Distribution-Free Theory of Nonparametric Regression. New York, NY, USA: Springer-Verlag, 2002
16. R. P. Halmos and V. S. Sunder, Bounded Integral Operators in L2 Spaces (Results in Mathematics and Related Areas), vol. 96. Berlin, Germany: Springer-Verlag, 1978
17. J. Kiefer and J. Wolfowitz, "Stochastic estimation of the maximum of a regression function," Ann. Math. Statist., vol. 23, no. 3, pp. 462-466. 1952
18. J. Kivinen, A. J. Smola, and R. C. Williamson, "Online learning with kernels," IEEE Trans. Signal Process., vol. 52, no. 8, pp. 2165-2176, Aug. 2004
19. H. J. Kushner and G. G. Yin, Stochastic Approximations and Recursive Algorithms and Applications. Berlin, Germany: Springer-Verlag, 2003
20. M. Loéve, "Fonctions alèatoires du second ordre," in Processus Stochastiques et Mouvement Brownien, P. Lèvy, Ed. Paris, France: Gauthier-Villars, 1948
21. J. Neveu, Discrete-Parameter Martingales. Amsterdam, The Netherlands: North Holland, 1975
22. E. Parzen, "An approach to time series analysis," Ann. Math. Statist., vol. 32, no. 4, pp. 951-989, 1961
23. I. Pinelis, "Optimum bounds for the distributions of martingales in Banach spaces," Ann. Probab., vol. 22, no. 4, pp. 1679-1706, 1994
24. H. Robbins and S. Monro, "A stochastic approximation method," Ann. Math. Statist., vol. 22, no. 3, pp. 400-407, 1951
25. S. Rosset and J. Zhu, "Piecewise linear regularized solution paths," Ann. Statist., vol. 35, no. 3, pp. 1012-1030, 2007
26. J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis. Cambridge, U.K.: Cambridge Univ. Press, 2004
27. S. Smale and Y. Yao, "Online learning algorithms," Found. Comput. Math., vol. 6, no. 2, pp. 145-170, 2006
28. S. Smale and D.-X. Zhou, "Shannon sampling and function reconstruction from point values," Bull. Amer. Math. Soc., vol. 41, no. 3, pp. 279-305, 2004
29. S. Smale and D.-X. Zhou, "Learning theory estimates via integral operators and their approximations," Constructive Approx., vol. 26, no. 2, pp. 153-172, 2007
30. S. Smale and D.-X. Zhou, "Online learning with Markov sampling," Anal. Appl., vol. 7, no. 1, pp. 87-113, Jan. 2009
31. P. Tarrès and Y. Yao, "Learning as stochastic approximation of regularisation paths," in Proc. Learn. Theory Approx. Workshop, Math. Forschungsinstitut Oberwolfach, 2008
32. R. Tibshirani, "Regression shrinkage and selection via the lasso," J. Roy. Statist. Soc, Ser. B., vol. 58, no. 1, pp. 267-288, 1996
33. G. Wahba, Spline Models for Observational Data. Philadelphia, PA, USA: SIAM, 1990
34. D.Williams, Probability with Martingales. Cambridge, U.K.: Cambridge Univ. Press, 1991
35. Y. Yao, "A dynamic theory of learning," Ph.D dissertation, Dept. Math., Univ. Calfornia, Berkeley, CA, USA, 2006
36. Y. Yao, "On complexity issue of online learning algorithms," IEEE Trans. Inform. Theory, vol. 56, no. 12, pp. 6470-6481, Dec. 2010
37. Y. Ying and M. Pontil, "Online gradient descent learning algorithms," Found. Comput. Math., vol. 5, no. 5, pp. 561-596, 2008
38. D.-X. Zhou, "Density problem and approximation error in learning theory," in Proc. Abstract Appl. Anal., vol. 2013, 2013, Article ID 715683, doi:10.1155/2013/715683.