Improved bounds for the nyström method with application to kernel classification

IEEE Transactions on Information Theory

Volumn 59, Issue 10, 2013, Pages 6939-6949

  Jin, Rong ^a   Yang, Tianbao ^b   Mahdavi, Mehrdad ^a   Li, Yu Feng ^c   Zhou, Zhi Hua ^c  

^a Michigan State University   (United States)

^b GE GLOBAL RESEARCH   (United States)

^c NANJING UNIVERSITY   (China)

Author keywords

Approximation error; concentration inequality; kernel methods; Nystr m method; random matrix theory

Indexed keywords

APPROXIMATION ERRORS; CONCENTRATION INEQUALITY; KERNEL METHODS; M METHOD; RANDOM MATRIX THEORY;

EIGENVALUES AND EIGENFUNCTIONS; RANDOM VARIABLES;

MATRIX ALGEBRA;

EID: 84884361899     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2013.2271378     Document Type: Article

References (33)

1. C. Williams and M. Seeger, "Using the Nystrom method to speed up kernel machines," in Advances in Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 2001, vol. 13, pp. 682-688.
2. P. Drineas andM.W.Mahoney, "On the Nystrom method for approximating a Grammatrix for improved kernel-based learning," J. Machine Learning Res., vol. 6, pp. 2153-2175, 2005. (Pubitemid 41832630)
3. C. Fowlkes, S. Belongie, F. Chung, and J. Malik, "Spectral grouping using the Nystrom method," IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 2, pp. 214-225, Feb. 2004.
4. S. Kumar, M. Mohri, and A. Talwalkar, "Sampling techniques for the Nystrom method," in Proc. Conf. Artif. Intell. Statist., 2009, pp. 304-311.
5. V. D. Silva and J.B. Tenenbaum, "Global versus local methods in nonlinear dimensionality reduction," in Advances in Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 2003, vol. 15, pp. 705-712.
6. J. C. Platt, "Fast embedding of sparse music similarity graphs," in Advances in Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 2004, vol. 16.
7. A. Talwalkar, S. Kumar, and H. A. Rowley, "Large-scale manifold learning," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog., 2008, pp. 1-8.
8. K. Zhang, I. W. Tsang, and J. T. Kwok, "Improved Nystrom lowrank approximation and error analysis," in Proc. 25th Int. Conf. Mach. Learning, 2008, pp. 1232-1239.
9. M.-A.Belabbas and P. J.Wolfe, "Spectralmethods inmachine learning and new strategies for very large data sets," Proc. Nat. Acad. Sci. USA, vol. 106, pp. 369-374, 2009.
10. A. Talwalkar and A. Rostamizadeh, "Matrix coherence and the Nystrom method," presented at the Proc. Conf. Uncertainty Artif. Intell., 2010.
11. C. Cortes, M. Mohri, and A. Talwalkar, "On the impact of kernel approximation on learning accuracy," J. Mach. Learning Res.-Proc. Track, vol. 9, pp. 113-120, 2010.
12. M. Li, J. T. Kwok, and B.-L. Lu, "Making large-scale Nyström approximation possible," in Proc. 27th Int. Conf. Mach. Learning, 2010, pp. 631-638.
13. L. W. Mackey, A. S. Talwalkar, and M. I. Jordan, "Divide-and-conquer matrix factorization," in Advances in Neural Information Processing Systems, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, Eds. : , 2011, vol. 24, pp. 1134-1142.
14. A. Gittens, "The spectral norm error of the naive Nystrom extension," presented at the CoRR, 2011.
15. M. W. Mahoney, "Randomized algorithms for matrices and data," Found. Trends Mach. Learning, vol. 3, no. 2, pp. 123-224, 2011.
16. S. Kumar, M. Mohri, and A. Talwalkar, "Sampling methods for the Nystrom method," J. Mach. Learning Res., vol. 13, pp. 981-1006, 2012.
17. J. Chen and Y. Saad, "Lanczos vectors versus singular vectors for effective dimension reduction," IEEE Trans. Knowl. Data Eng., vol. 21, no. 8, pp. 1091-1103, Aug. 2009.
18. L. Wu, X. Ying, X. Wu, and Z.-H. Zhou, "Line orthogonality in adjacency eigenspace with application to community partition," in Proc. 22nd Int. Joint Conf. Artif. Intell., 2011, pp. 2349-2354.
19. M. Ji, T. Yang, B. Lin, R. Jin, and J. Han, "A simple algorithm for semi-supervised learning with improved generalization error bound," in Proc. 29th Int. Conf. Mach. Learning, 2012, pp. 1223-1230.
20. T. Yang, Y.-F. Li, M. Mahdavi, R. Jin, and Z.-H. Zhou, "Nystrom method vs random Fourier features: A theoretical and empirical comparison," in Advances in Neural Information Processing Systems. Cambridge, MA, USA: MIT Press, 2012, vol. 25, pp. 485-493.
21. A. Gittens and M. W. Mahoney, "Revisiting the Nystrom method for improved large-scalemachine learning," in Proc. 30th Int. Conf.Mach. Learning, 2013, pp. 567-575.
22. F. Bach, "Sharp analysis of low-rank kernel matrix approximations," in Proc. 26th Conf. Learning Theory, 2013.
23. J. B. Hough, M. Krishnapur, Y. Peres, and B. Virag, "Determinantal processes and independence," Probability Surveys, vol. 3, pp. 206-229, 2006.
24. P. Drineas, M. W. Mahoney, and S. Muthukrishnan, "Relative-error cur matrix decompositions," SIAM J. Matrix Anal. Appl., vol. 30, pp. 844-881, 2008.
25. S. Smale and D.-X. Zhou, "Geometry on probability spaces," Constructive Approximations, vol. 30, pp. 311-323, 2009.
26. E. Candés and J. Romberg, "Sparsity and incoherence in compressive sampling," Inverse Problems, vol. 23, no. 3, pp. 969-985, 2007. (Pubitemid 46838836)
27. D. L. Donoho, "Compressed sensing," IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289-1306, Apr. 2006.
28. V. Koltchinskii and M. Yuan, "Sparsity in multiple kernel learning," Annu. Statist., vol. 38, pp. 3660-3694, 2010.
29. M. Kloft and G. Blanchard, "The local Rademacher complexity of Lp-norm multiple kernel learning," in Advances in Neural Information Processing Systems. Cambridge, MA, USA: , 2011, vol. 23, pp. 2438-2446.
30. O. Dekel and Y. Singer, "Support vector machines on a budget," in Proc. Advances Neural Information Processing Systems, 2006, pp. 345-352.
31. T. Joachims and C.-N. J. Yu, "Sparse kernel SVMS via cutting-plane training," Mach. Learning, vol. 76, pp. 179-193, 2009.
32. P. L. Bartlett, O. Bousquet, and S. Mendelson, "Local Rademacher complexities," Ann. Statist., pp. 44-58, 2002.
33. V. Koltchinskii, Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems. New York, NY, USA: Springer-Verlag, 2011.