On the eigenspectrum of the gram matrix and the generalization error of Kernel-PCA

IEEE Transactions on Information Theory

Volumn 51, Issue 7, 2005, Pages 2510-2522

  Shawe Taylor, John ^a   Williams, Christopher K I ^b   Cristianini, Nello ^c   Kandola, Jaz ^d  

^a UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

^c UNIVERSITY OF CALIFORNIA   (United States)

^d Merrill Lynch International   (United Kingdom)

Author keywords

Concentration bounds; Gram matrices; Kernel methods; Principal components analysis (PCA); Rademacher complexity; Spectra of random matrices; Statistical learning theory

Indexed keywords

APPROXIMATION THEORY; COMPUTATIONAL COMPLEXITY; EIGENVALUES AND EIGENFUNCTIONS; MATRIX ALGEBRA; POLYNOMIALS; PRINCIPAL COMPONENT ANALYSIS; STATISTICAL METHODS; THEOREM PROVING;

CONCENTRATION BOUNDS; GRAM MATRICES; KERNEL METHODS; RADEMACHER COMPLEXITY; STATISTICAL LEARNING THEORY;

LEARNING SYSTEMS;

EID: 22844440983     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2005.850052     Document Type: Article

References (29)

1. N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines. Cambridge, U.K.: Cambridge Univ. Press, 2000.
2. S. Mika, B. Schölkopf, A. Smola, K.-R. Müller, M. Scholz, and G. Rátsch, "Kernel PCA and de-noising in feature spaces," Adv. Neural Inf. Process. Syst., vol. 11, 1998.
3. N. Cristianini, H. Lodhi, and J. Shawe-Taylor. (2000) Latent Semantic Kernels for Feature Selection. NeuroCOLT Working Group. [Online]. Available: http://www.neurocolt.org
4. J. Kleinberg, "Authoritative sources in a hyperlinked environment," in Proc. 9th ACM-SIAM Symp. Discrete Algorithms, San Francisco, CA, Jan. 1998.
5. S. Brin and L. Page, "The anatomy of a large-scale hypertextual (web) search engine," in Proc. 7th Int. World Wide Web Conf., Brisbane, Australia, Apr. 1998.
6. A. Y. Ng, A. X. Zheng, and M. I. Jordan, "Link analysis, eigenvectors, and stability," in Proc. 17th Int. Joint Conf Artificial Intelligence (IJCAI-01), Seattle, WA, Aug. 2001.
7. A. Guionnet and O. Zeitouni, "Concentration of the spectral measure for large matrices," Electron. Commun. Prob., vol. 5, pp. 119-136, 2000.
8. N. Alon, M. Krivelevich, and V. H. Vu, "On the concentration of eigenvalues of random symmetric matrices," Israel J. Math., vol. 131, pp. 259-267, 2002.
9. C. K. I. Williams and M. Seeger, "The effect of the input density distribution on kernel-based classifiers," in Proc. 17th Int. Conf. Machine Learning (ICML 2000), P. Langley, Ed. San Francisco, CA: Morgan Kaufmann, 2000.
10. C. T. H. Baker, The Numerical Treatment of Integral Equations. Oxford, U.K.: Clarendon, 1977.
11. H. Zhu, C. K. I. Williams, R. J. Rohwer, and M. Morciniec, "Gaussian regression and optimal finite dimensional linear models," in Neural Networks and Machine Learning, C. M. Bishop, Ed. Berlin, Germany: Springer-Verlag, 1998.
12. V. Koltchinskii and E. Gine, "Random matrix approximation of spectra of integral operators," Bernoulli, vol. 6, no. 1, pp. 113-167, 2000.
13. I. Johnstone. (2000) On the Distribution of the Largest Principal Component. Stanford Univ., Stanford, CA. [Online]. Available: http://www-stat.stanford.edu/imj/Reports/index.html
14. J. Shawe-Taylor, N. Cristianini, and J. Kandola, "On the concentration of spectral properties," in Advances in Neural Information Processing Systems, T. G. Diettrich, S. Becker, and Z. Ghahramani, Eds. Cambridge, MA: MIT Press, 2002, vol. 14.
15. J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis. Cambridge, U.K.: Cambridge Univ. Press, 2004.
16. W. Hoeffding, "Probability inequalities for sums of bounded random variables," J. Amer. Stat. Assoc., vol. 58, pp. 13-30, 1963.
17. K. Azuma, "Weighted sums of certain dependent random variables," Tohoku Math J., vol. 19, pp. 357-367, 1967.
18. C. McDiarmid, "On the method of bounded differences," in Surveys in Combinatorics 1989. Cambridge, U.K.: Cambridge Univ. Press, 1989, pp. 148-188.
19. S. Boucheron, G. Lugosi, and P. Massart, "A sharp concentration inequality with applications," Random Structures and Algorithms, vol. 16, pp. 277-292, 2000.
20. M. Talagrand, "New concentration inequalities in product spaces," Invent. Math., vol. 126, pp. 505-563, 1996.
21. H. Voss, "Variational characterization of eigenvalues of nonlinear eigen-problems," in Proc. Int. Conf. Mathematical and Computer Modeling in Science and Engineering, Prague, Czech Republic, Jan. 2003, pp. 379-383.
22. M. L. Eaton and D. E. Tyler, "On Wielandt's inequality and its application to the asymptotic distribution of the eigenvalues of a random symmetric matrix," Ann. Statist., vol. 19, no. 1, pp. 260-271, 1991.
23. A. T. James, "The distribution of the latent roots of the covariance matrix," Ann. Math. Statist., vol. 31, pp. 151-158, 1960.
24. D. N. Lawley, "Tests of significance for the latent roots of covariance and correlation matrices," Biometrika, vol. 43, no. 1/2, pp. 128-136, 1956.
25. T. W. Anderson, "Asymptotic theory for principal component analysis," Ann. Math. Statist., vol. 34, no. 1, pp. 122-148, 1963.
26. C. M. Waternaux, "Asymptotic distribution of the sample roots for a nonnormal population," Biometrika, vol. 63, no. 3, pp. 639-645, 1976.
27. M. Ledoux and M. Talagrand, Probability in Banach Spaces: Isoperimetry and Processes. Berlin, Germany: Springer-Verlag, 1991.
28. P. L. Bartlett and S. Mendelson, "Rademacher and Gaussian complexities: Risk bounds and structural results," J. Mach. Learning Res., vol. 3, pp. 463-482, 2002.
29. C. Blake and C. Merz. (1998) UCI Repository of Machine Learning Databases. [Online]. Available: http://www.ics.uci.edu/~mlearn/ ML-Repository.html