Clustered Nyström method for large scale manifold learning and dimension reduction

IEEE Transactions on Neural Networks

Volumn 21, Issue 10, 2010, Pages 1576-1587

  Zhang, Kai ^a   Kwok, James T ^b  

^a LAWRENCE BERKELEY NATIONAL LABORATORY   (United States)

^b HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

Dimension reduction; eigenvalue decomposition; kernel matrix; low rank approximation; manifold learning; Nystrm method; sampling

Indexed keywords

APPROXIMATION THEORY; CLUSTERING ALGORITHMS; EIGENVALUES AND EIGENFUNCTIONS; LEARNING ALGORITHMS; LEARNING SYSTEMS; MATRIX ALGEBRA; PRINCIPAL COMPONENT ANALYSIS; REDUCTION; SAMPLING;

DIMENSION REDUCTION; EIGENVALUE DECOMPOSITION; KERNEL MATRICES; LOW RANK APPROXIMATIONS; MANIFOLD LEARNING; NYSTRM METHOD;

INVERSE PROBLEMS;

EID: 77957779140     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2010.2064786     Document Type: Article

References (51)

1. V. N. Vapnik, The Nature of Statistical Learning Theory. New York: Springer-Verlag, 1995.
2. S. Mika, G. Rätsch, J. Weston, B. Schölkopf, and K. R. Müller, "Fisher discriminant analysis with kernels", in Proc. IEEE Neural Netw. Signal Process. Workshop, Aug. 1999, pp. 41-48.
3. B. Schölkopf, A. Smola, and K.-R. Müller, "Nonlinear component analysis as a kernel eigenvalue problem", Neural Comput., vol. 10, no. 5, pp. 1299-1319, Jul. 1998.
4. S. T. Roweis and L. K. Saul, "Nonlinear dimensionality reduction by locally linear embedding", Science, vol. 290, no. 5500, pp. 2323-2326, Dec. 2000.
5. J. B. Tenenbaum, V. de Silva, and J. C. Langford, "A global geometric framework for nonlinear dimensionality reduction", Science, vol. 290, no. 5500, pp. 2319-2323, Dec. 2000.
6. M. Belkin and P. Niyogi, "Laplacian eigenmaps and spectral techniques for embedding and clustering", in Advances in Neural Information Processing Systems 14. Cambridge, MA: MIT Press, 2002, pp. 585-591.
7. C. Ding and X. He, "Linearized cluster assignment via spectral ordering", in Proc. 21st Int. Conf. Mach. Learn., Banff, AB, Canada, Jul. 2004, pp. 30-37.
8. A. Y. Ng, M. I. Jordan, and Y. Weiss, "On spectral clustering: Analysis and an algorithm", in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2002.
9. H. Zha, X. He, C. Ding, H. Simon, and M. Gu, "Spectral relaxation for k-means clustering", in Advances in Neural Information Processing Systems 13. Cambridge, MA: MIT Press, 2001, pp. 1057-1064.
10. J. Zhang, K. Zhang, X. Xu, C. K. Tse, and M. Small, "Seeding the kernels in graphs: Toward multi-resolution community analysis", New J. Phys., vol. 11, p. 113003, Nov. 2009.
11. C. K. I. Williams and M. Seeger, "The effect of the input density distribution on kernel-based classifiers", in Proc. 17th Int. Conf. Mach. Learn., Stanford, CA, Jun. 2000, pp. 1159-1166.
12. S. Fine and K. Scheinberg, "Efficient SVM training using low-rank kernel representations", J. Mach. Learn. Res., vol. 2, pp. 243-264, Dec. 2001.
13. C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning. Cambridge, MA: MIT Press, 2006.
14. J. A. K. Suykens and J. Vandewalle, "Least squares support vector machine classifiers", Neural Process. Lett., vol. 9, no. 3, pp. 293-300, Jun. 1999.
15. C. K. I. Williams and M. Seeger, "Using the Nyström method to speed up kernel machines", in Advances in Neural Information Processing Systems 13. Cambridge, MA: MIT Press, 2001, pp. 682-688.
16. C. Fowlkes, S. Belongie, F. Chung, and J. Malik, "Spectral grouping using the Nyström method", IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 2, pp. 214-225, Feb. 2004.
17. T. Cox and M. Cox, Multidimensional Scaling. London, U. K.: Chapman & Hall, 1994.
18. J. Shi and J. Malik, "Normalized cuts and image segmentation", IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 8, pp. 888-905, Aug. 2000.
19. A. J. Smola and B. Schölkopf, "Sparse greedy matrix approximation for machine learning", in Proc. 17th Int. Conf. Mach. Learn., Stanford, CA, Jun. 2000, pp. 911-918.
20. S. Kumar, M. Mohri, and A. Talwalkar, "On sampling-based approximate spectral decomposition", in Proc. Int. Conf. Mach. Learn., Montreal, QC, Canada, Jun. 2009, pp. 585-592.
21. M. Ouimet and Y. Bengio, "Greedy spectral embedding", in Proc. 10th Int. Workshop Artif. Intell. Statist., Barbados, Jan. 2005, pp. 253-260.
22. F. R. Bach and M. I. Jordan, "Kernel independent component analysis", J. Mach. Learn. Res., vol. 3, pp. 1-48, Jul. 2002.
23. F. R. Bach and M. I. Jordan, "Predictive low-rank decomposition for kernel methods", in Proc. 22nd Int. Conf. Mach. Learn., Bonn, Germany, Aug. 2005, pp. 33-40.
24. D. Achlioptas and F. McSherry, "Fast computation of low rank matrix approximations", in Proc. 33rd Annu. ACM Symp. Theory Comput., Hersonissos, Greece, Jul. 2001, pp. 611-618.
25. P. Drineas, E. Drinea, and P. S. Huggins, "An experimental evaluation of a Monte-Carlo algorithm for singular value decomposition", in Proc. 8th Panhellenic Conf. Informat., Nicosia, Cyprus, 2003, pp. 279-296.
26. P. Drineas and M. W. Mahoney, "On the Nyström method for approximating a Gram matrix for improved kernel-based learning", J. Mach. Learn. Res., vol. 6, pp. 2153-2175, Dec. 2005.
27. A. Frieze, R. Kannan, and S. Vempala, "Fast Monte-Carlo algorithms for finding low-rank approximations", in Proc. 39th Annu. Symp. Found. Comput. Sci., Palo Alto, CA, Nov. 1998, pp. 370-378.
28. W.-Y. Chen, Y. Song, H. Bai, C.-J. Lin, and E. Y. Chang, "Parallel spectral clustering in distributed systems", IEEE Trans. Pattern Anal. Mach. Intell., to be published.
29. J. C. Platt, "Fastmap, metricmap, and landmark MDS are all Nyström algorithms", in Proc. 10th Int. Workshop Artif. Intell. Statist., Barbados, Jan. 2005, pp. 261-268.
30. V. de Silva and J. B. Tenenbaum, "Global vs. local methods in nonlinear dimensionality reduction", in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2003, pp. 705-712.
31. A. Talwalkar, S. Kumar, and H. Rowley, "Large scale manifold learning", in Proc. Int. Conf. Comput. Vis. Pattern Recognit., Anchorage, AK, Jun. 2008, pp. 1-8.
32. S. Kumar, M. Mohri, and A. Talwalkar, "Sampling techniques for the Nyström method", in Proc. Int. Conf. Artif. Intell. Statist., Clearwater Beach, FL, Apr. 2009, pp. 304-311.
33. K. Zhang, J. T. Kwok, and B. Parvin, "Prototype vector machine for large scale semi-supervised learning", in Proc. 26th Annu. Int. Conf. Mach. Learn., Montreal, QC, Canada, Jun. 2009, pp. 1233-1240.
34. R. B. Lehoucq, D. C. Sorensen, and C. Yang, ARPACK User Guide: Solution of Large Scale Eigenvalue Problems by Implicitly Restarted Arnoldi Methods. Philadelphia, PA: SIAM, 1998.
35. G. H. Golub and C. F. Van Loan, Matrix Computations, 3rd ed. Baltimore, MD: The Johns Hopkins Univ. Press, 1996.
36. C. T. H. Baker, The Numerical Treatment of Integral Equations. Oxford, U. K.: Clarendon, 1977.
37. S. Kumar, M. Mohri, and A. Talwalkar, "Ensemble Nyström method", in Advances in Neural Information Processing Systems 22. Cambridge, MA: MIT Press, 2009, pp. 1060-1068.
38. M. Li, J. T. Kwok, and B.-L. Lu, "Making large-scale Nyström approximation possible", in Proc. Int. Conf. Mach. Learn., Haifa, Israel, Jun. 2010, pp. 1-8.
39. K. Zhang and J. T. Kwok, "Block-quantized kernel matrix for fast spectral embedding", in Proc. 23rd Int. Conf. Mach. Learn., Pittsburgh, PA, Jun. 2006, pp. 1097-1104.
40. K. Zhang and J. T. Kwok, "Density-weighted Nyström method for computing large kernel eigensystems", Neural Comput., vol. 21, no. 1, pp. 121-146, Jan. 2009.
41. P. Drineas, R. Kannan, and M. W. Mahoney, "Fast Monte-Carlo algorithms for matrices II: Computing a low-rank approximation to a matrix", SIAM J. Comput., vol. 36, no. 1, pp. 158-183, 2006. (Pubitemid 46374022)
42. A. Gersho and R. M. Gray, Vector Quantization and Signal Compression. Boston, MA: Kluwer, 1992.
43. T. Kanungo, D. M. Mount, N. S. Netanyahu, C. D. Piatko, R. Silverman, and A. Y. Wu, "An efficient k-means clustering algorithm: Analysis and implementation", IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 881-892, Jul. 2002. (Pubitemid 34835464)
44. D. Pelleg and A. Moore, "Accelerating exact k-means algorithms with geometric reasoning", in Proc. 5th Int. Conf. Knowl. Discovery Data Mining, San Diego, CA, Aug. 1999, pp. 277-281.
45. E. Elkan, "Using the triangle inequality to accelerate k-means", in Proc. 21th Int. Conf. Mach. Learn., Washington D. C., Aug. 2003, pp. 147-153.
46. D. Lee, S. Baek, and K. Sung, "Modified k-means algorithm for vector quantizer design", IEEE Signal Process. Lett., vol. 4, no. 1, pp. 2-4, Jan. 1997.
47. I. S. Dhillon and D. S. Modha, "A data-clustering algorithm on distributed memory multiprocessors", in Large-Scale Parallel Data Mining. Berlin, Germany: Springer-Verlag, 2000, pp. 245-260.
48. C. Walder, K. I. Kim, and B. Schölkopf, "Sparse multiscale Gaussian process regression", in Proc. 25th Int. Conf. Mach. Learn., Helsinki, Finland, Jun. 2008, pp. 1112-1119.
49. W. Liu, J. He, and S.-F. Chang, "Large graph construction for scalable semi-supervised learning", in Proc. 27th Int. Conf. Mach. Learn., Haifa, Israel, Jun. 2010, pp. 1-8.
50. E. Parzen, "On estimation of a probability density function and mode", Ann. Math. Statist., vol. 33, no. 3, pp. 1065-1076, 1962.
51. K. Zhang, M. Tang, and J. T. Kwok, "Applying neighborhood consistency for fast clustering and kernel density estimation", in Proc. Int. Conf. Comput. Vis. Pattern Recognit., San Diego, CA, Jun. 2005. pp. 1001-1007.