A review of Nyström methods for large-scale machine learning

Information Fusion

Volumn 26, Issue , 2015, Pages 36-48

  Sun, Shiliang ^a   Zhao, Jing ^a   Zhu, Jiang ^a  

^a EAST CHINA NORMAL UNIVERSITY   (China)

Author keywords

Low rank approximation; Machine learning; Nystr m method; Sampling method

Indexed keywords

APPROXIMATION THEORY; ARTIFICIAL INTELLIGENCE;

GAUSSIAN PROCESS REGRESSION; LARGE-SCALE MACHINE LEARNING; LOW RANK APPROXIMATIONS; LOW-RANK MATRIX APPROXIMATIONS; M METHOD; MACHINE LEARNING APPLICATIONS; SAMPLING METHOD; STATE-OF-THE-ART TECHNIQUES;

LEARNING SYSTEMS;

EID: 84939997221     PISSN: 15662535     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.inffus.2015.03.001     Document Type: Review

References (72)

1. A. Frieze, R. Kannan, S. Vempala, Fast Monte-Carlo algorithms for finding low-rank approximations, in: Proceedings of the 39th Annual IEEE Symposium on Foundations of Computer Science, 1998, pp. 370-378.
2. S. Fine, and K. Scheinberg Efficient SVM training using low-rank kernel representations J. Machine Learn. Res. 2 2001 243 264
3. M.W. Mahoney, P. Drineas, CUR matrix decompositions for improved data analysis, in: Proceedings of the National Academy of Sciences, 2009, pp. 697-702.
4. C. Williams, and M. Seeger Using the Nyström method to speed up kernel machines Adv. Neural Inform. Process. Syst. 13 2001 682 688
5. A. Deshpande, L. Rademacher, S. Vempala, G. Wang, Matrix approximation and projective clustering via volume sampling, in: Proceedings of the 7th Annual ACM-SIAM Symposium on Discrete Algorithms, 2006, pp. 1117-1126.
6. P. Drineas, and M.W. Mahoney On the Nyström method for approximating a gram matrix for improved kernel-based learning J. Machine Learn. Res. 6 2005 2153 2175
7. S. Kumar, M. Mohri, A. Talwalkar, Ensemble Nyström method, in: Proceedings of Neural Information Processing Systems, 2009, pp. 1060-1068.
8. M. Li, J.T. Kwok, B. Lu, Making large-scale Nyström approximation possible, in: Proceedings of the 27th International Conference on Machine Learning, 2010, pp. 631-638.
9. N. Halko, P.G. Martinsson, and J.A. Tropp Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions SIAM Rev. 53 2011 217 288
10. C. Cortes, M. Mohri, A. Talwalkar, On the impact of kernel approximation on learning accuracy, in: Proceedings of 13th International Conference on Artificial Intelligence and Statistics, 2010, pp. 113-120.
11. P. Drineas, R. Kannan, and M.W. Mahoney Fast Monte Carlo algorithms for matrices ii: Computing a low-rank approximation to a matrix SIAM J. Comput. 36 2006 158 183
12. A.K. Farahat, A. Ghodsi, M.S. Kamel, A novel greedy algorithm for Nyström approximation, in: Proceedings of the 14th International Workshop on Artificial Intelligence and Statistics, 2011, pp. 269-277.
13. K. Zhang, I.W. Tsang, J.T. Kwok, Improved Nyström low-rank approximation and error analysis, in: Proceedings of the 25th International Conference on Machine Learning, 2008, pp. 1232-1239.
14. K. Zhang, and J.T. Kwok Clustered Nyström method for large scale manifold learning and dimension reduction IEEE Trans. Neural Netw. 21 2009 121 146
15. K. Zhang, and J.T. Kwok Density-weighted Nyström method for computing large kernel eigensystems Neural Comput. 21 2009 121 146
16. S. Kumar, M. Mohri, and A. Talwalkar Sampling methods for the Nyström method J. Machine Learn. Res. 13 2012 981 1006
17. S. Kumar, M. Mohri, A. Talwalkar, Sampling techniques for the Nyström method, in: Proceedings of the 12th International Workshop on Artificial Intelligence and Statistics, 2009, pp. 304-311.
18. A. Talwalkar, S. Kumar, H. Rowley, Large-scale manifold learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2008, pp. 3129-3152.
19. A. Talwalkar, S. Kumar, and H. Rowley Large-scale SVD and manifold learning J. Machine Learn. Res. 14 2013 3129 3252
20. J.C. Platt Fast embedding of sparse music similarity graphs Adv. Neural Inform. Process. Syst. 16 2004 511 578
21. A.Y. Ng, M.I. Jordan, Y. Weiss, On spectral clustering: analysis and an algorithm, in: Proceedings of Neural Information Processing Systems, 2001, pp. 849-856.
22. C. Fowlkes, S. Belongie, F. Chung, and J. Malik Spectral grouping using the Nyström method IEEE Trans. Pattern Anal. Machine Intell. 26 2002 214 225
23. M.A. Belabbas, P.J. Wolfe, Spectral methods in machine learning and new strategies for very large datasets, in: Proceedings of the National Academy of Sciences, 2009, pp. 369-374.
24. M. Li, X. Lian, J.T. Kwok, B. Lu, Time and space efficient spectral clustering via column sampling, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2011, pp. 2297-2304.
25. S. Si, C. Hsieh, I.S. Dhillon, Memory efficient kernel approximation, in: Proceedings of the 31st International Conference on Machine Learning, 2014, pp. 701-709.
26. C. Williams, C. Rasmussen, V.T.A. Schwaighofer, Observations on the Nyström method for Gaussian process prediction, Technical Report, 2002.
27. T. Yang, Y. Li, M. Mahdavi, R. Jin, and Z. Zhou Nyström method vs random fourier features: a theoretical and empirical comparison Adv. Neural Inform. Process. Syst. 24 2012 476 484
28. P. Parker, P.J. Wolfe, V. Tarok, A signal processing application of randomized low-rank approximation, in: Proceedings of the 13th IEEE Workshop on Statistical Signal Processing, 2005, pp. 345-350.
29. D.N. Spendley, P.J. Wolfe, Adaptive beamforming using fast low-rank covariance matrix approximations, in: Proceedings of IEEE Radar Conference, 2008, pp. 1-5.
30. M.A. Belabbas, P.J. Wolfe, On sparse representations of linear operators and the approximation of matrix products, in: Proceedings of the 42nd Annual Conference on Information Sciences and Systems, 2008, pp. 258-263.
31. M.A. Belabbas, and P.J. Wolfe On landmark selection and sampling in high-dimensional data analysis Philos. Trans. Roy. Soc. 367 2009 4295 4312
32. J. Tenenbaum, V. de Silva, and J. Langford A global geometric framework for nonlinear dimensionality reduction Science 290 2000 2319 2323
33. M. Belkin, and P. Niyogi Laplacian eigenmaps and spectral techniques for embedding and clustering Adv. Neural Inform. Process. Syst. 13 2001 585 591
34. T. Sim, S. Baker, and M. Bsat The CMU pose, illumination, and expression database IEEE Trans. Pattern Anal. Machine Intell. 25 2003 1615 1618
35. Z. Bai, J. Demmel, J. Dongarra, A. Ruhe, and H. van der Vorst Templates for the solution of algebraic eigenvalue problems: a practical guide 2000 SIAM
36. J. Shi, and J. Malik Normalized cuts and image segmentation IEEE Trans. Pattern Anal. Machine Intell. 22 2000 888 905
37. A. Gittens, M.W. Mahoney, Revisiting the Nyström method for improved large-scale machine learning, in: Proceedings of the 26th International Conference on Machine Learning, 2013, pp. 567-575.
38. C.T.H. Baker The Numerical Treatment of Integral Equations 1977 Glarendon Press Oxford
39. S. Wang, Z. Zhang, Efficient algorithms and error analysis for the modified Nyström method, in: Proceedings of the 17th International Conference on Artificial Intelligence and Statistics, 2014, pp. 1-11.
40. S. Wang, L. Luo, Z. Zhang, The Modified Nyström Method: Theories, Algorithms, and Extension, Technical Report, 2014.
41. A. Gittens, The spectral norm error of the naive Nyström extension, Technical Report, 2011.
42. S. Wang, and Z. Zhang Improving CUR matrix decomposition and Nyström approximation via adaptive sampling J. Machine Learn. Res. 14 2013 2729 2769
43. F. Shang, L.C. Jiao, J. Shi, M. Gong, and R.H. Shang Fast density-weighted low-rank approximation spectral clustering Data Mining Knowl. Discov. 23 2011 345 378
44. Z. Zhang The matrix ridge approximation: algorithms and applications Machine Learn. 97 2014 227 258
45. S. Kumar, M. Mohri, A. Talwalkar, On sampling-based approximate spectral decomposition, in: Proceedings of the 26th International Conference on Machine Learning, 2009, pp. 553-560.
46. P. Drineas, M.W. Mahoney, and S. Muthukrishnan Relative-error CUR matrix decompositions SIAM J. Matrix Anal. Appl. 30 2008 844 881
47. A. Talwalkar, A. Rostamizadeh, Matrix coherence and the Nyström method, in: Proceedings of the 26th Conference in Uncertainty in Artificial Intelligence, 2010, pp. 572-579.
48. E.J. Candès, and J. Romberg Sparsity and incoherence in compressive sampling Inverse Problems 23 2007 969 986
49. E.J. Candès, and B. Recht Exact matrix completion via convex optimization Found. Comput. Math. 9 2009 717 772
50. P. Drineas, M. Magdon-Ismail, M.W. Mahoney, and D.P. Woodruff Fast approximation of matrix coherence and statistical leverage J. Machine Learn. Res. 13 2012 3475 3506
51. M. Mohria, A. Talwalkar, Can matrix coherence be efficiently and accurately estimated?, in: Proceedings of the 14th International Workshop on Artificial Intelligence and Statistics, 2011, pp. 534-542.
52. A.J. Smola, B. Schölkopf, Sparse greedy matrix approximation for machine learning, in: Proceedings of International Conference on Machine Learning, 2000, pp. 911-918.
53. F.R. Bach, and M.I. Jordan Kernel independent component analysis J. Machine Learn. Res. 3 2002 1 48
54. E. Elkan, Using the triangular inequality to accelerate k-means, in: Proceedings of the 21th International Conference on Machine Learning, 2003, pp. 147-153.
55. T. Kanungo, D.M. Mount, N.S. Netanyahu, C.D. Piatko, R. Silverman, and A.Y. Wu An efficient k-means clustering algorithm: analysis and implementation IEEE Trans. Pattern Anal. Machine Intell. 24 2001 881 892
56. C. Boutsidis, P. Drineas, M. Magdon-Ismail, Near optimal column-based matrix reconstruction, in: Proceedings of 52nd IEEE Annual Symposium on Foundations of Computer Science, 2011, pp. 305-314.
57. C. Yip, M.W. Mahoney, A.S. Szalay, I. Csabai, T. Budavari, R. Wyse, and L. Dobos Objective identification of informative wavelength regions in galaxy spectra Astronom. J. 147 2014 1 36
58. J.C. Platt Fast embedding of sparse similarity graphs Adv. Neural Inform. Process. Syst. 15 2003 571 578
59. P. Paschou, E. Ziv, E.G. Burchard, S. Choudhry, W. Rodriguez-Cintron, M.W. Mahoney, and P. Drineas PCA-correlated SNPs for structure identification in worldwide human populations PLOS Genet. 3 2007 1672 1686
60. S. Sun, Tangent space intrinsic manifold regularization for data representation, in: Proceedings of the 1st IEEE China Summit and International Conference on Signal and Information Processing, 2013, pp. 179-183.
61. Y. Zhou, S. Sun, Semi-supervised tangent space discriminant analysis, Mathematical Problems in Engineering Special Issue on Machine Learning with Applications to Autonomous Systems, 2015.
62. J. Zhao, S. Sun, Variational dependent multi-output Gaussian process dynamical systems, in: Proceedings of the 17th International Conference on Discovery Science, 2014, pp. 350-361.
63. B. Scholkopf, and A. Smola Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond 2002 MIT Press
64. J. Shawe-Taylor, and S. Sun Kernel Methods and Support Vector Machines 2013 Elsevier Book Chapter for E-Reference Signal Processing
65. X. Xie, and S. Sun Multi-view Laplacian twin support vector machines Appl. Intell. 41 2014 1059 1068
66. J. Li, S. Sun, Nonlinear combination of multiple kernels for support vector machines, in: Proceedings of the 20th International Conference on Pattern Recognition, 2010, pp. 2889-2892.
67. Z. Wang, W. Jie, and D. Gao A novel multiple Nyström-approximating kernel discriminant analysis Neurocomputing 119 2013 385 398
68. S. Sun, Multitask learning for EEG-based biometrics, in: Proceedings of the 19th International Conference on Pattern Recognition, 2008, pp. 1-4.
69. X. Xie Multitask centroid twin support vector machines Neurocomputing 149 2015 1085 1091
70. J. Zhu, S. Sun, Single-task and multitask sparse Gaussian processes, in: Proceedings of the International Conference on Machine Learning and Cybernetics, 2013, pp. 1033-1038.
71. E.V. Bonilla, K. Chai, and C. Williams Multi-task Gaussian process prediction Adv. Neural Inform. Process. Syst. 20 2008 153 160
72. S. Sun, Infinite mixtures of multivariate Gaussian processes, in: Proceedings of the International Conference on Machine Learning and Cybernetics, 2013, pp. 1011-1016.