Sharp analysis of low-rank kernel matrix approximations

Journal of Machine Learning Research

Volumn 30, Issue , 2013, Pages 185-209

  Bach, Francis ^a  

^a ECOLE NORMALE SUPÉRIEURE   (France)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; REGRESSION ANALYSIS;

KERNEL LOGISTIC REGRESSION; KERNEL RIDGE REGRESSIONS; LOW RANK APPROXIMATIONS; POSITIVE-DEFINITE KERNEL; PREDICTIVE PERFORMANCE; PROBLEM INSTANCES; SIMPLE ALGORITHM; SUPERVISED LEARNING PROBLEMS;

MATRIX ALGEBRA;

EID: 84898034803     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (53)

1. S. Arlot and F. Bach. Data-driven calibration of linear estimators with minimal penalties. In Adv. NIPS, 2009.
2. F. Bach. Self-concordant analysis for logistic regression. Electronic Journal of Statistics, 4: 384-414, 2010.
3. F. Bach and M. I. Jordan. Predictive low-rank decomposition for kernel methods. In Proc. ICML, 2005.
4. G. Blanchard, P. Massart, R. Vert, and L. Zwald. Kernel projection machine: a new tool for pattern recognition. In Adv. NIPS, 2004.
5. G. Blanchard, O. Bousquet, and P. Massart. Statistical performance of support vector machines. The Annals of Statistics, 36(2): 489-531, 2008.
6. A. Bordes, S. Ertekin, J. Weston, and L. Bottou. Fast kernel classifiers with online and active learning. Journal of Machine Learning Research, 6: 1579-1619, 2005.
7. C. Boutsidis, M. W. Mahoney, and P. Drineas. An improved approximation algorithm for the column subset selection problem. In Proc. SODA, 2009.
8. A. Caponnetto and E. De Vito. Optimal rates for the regularized least-squares algorithm. Found. Comput. Math., 7(3): 331-368, 2007.
9. O. Chapelle. Training a support vector machine in the primal. Neural Computation, 19(5): 1155-1178, 2007.
10. C. Cortes, M. Mohri, and A. Talwalkar. On the impact of kernel approximation on learning accuracy. In Proc. AISTATS, 2010.
11. O. Dekel, S. Shalev-Shwartz, and Y. Singer. The Forgetron: A kernel-based perceptron on a fixed budget. In Adv. NIPS, 2005.
12. P. S. Dhillon, D. P. Foster, S. M. Kakade, and L. H. Ungar. A risk comparison of ordinary least squares vs. ridge regression. Technical Report 1105.0875, ArXiv, 2011.
13. P. Drineas, M. Magdon-Ismail, M. W. Mahoney, and D. P. Woodruff. Fast approximation of matrix coherence and statistical leverage. Journal of Machine Learning Research, 13: 3475-3506, 2012.
14. S. Fine and K. Scheinberg. Efficient SVM training using low-rank kernel representations. Journal of Machine Learning Research, 2: 243-264, 2001.
15. A. Gittens. The spectral norm error of the naive Nyström extension. Arxiv preprint arXiv: 1110.5305, 2011.
16. R. M. Gray. Toeplitz and circulant matrices: A review. Foundations and Trends in Communications and Information Theory, 2(3): 155-239, 2006.
17. C. Gu. Smoothing spline ANOVA models. Springer, 2002.
18. Z. Harchaoui, F. Bach, and E. Moulines. Testing for homogeneity with kernel Fisher discriminant analysis. Technical Report 00270806-v1, HAL, 2008.
19. T. J. Hastie and R. J. Tibshirani. Generalized Additive Models. Chapman & Hall, 1990.
20. W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301): 13-30, 1963.
21. D. Hsu, S. M. Kakade, and T. Zhang. An analysis of random design linear regression. In Proc. COLT, 2011.
22. D. Hsu, S. M. Kakade, and T. Zhang. Tail inequalities for sums of random matrices that depend on the intrinsic dimension. Electronic Communications in Probability, 17(14): 1-13, 2012.
23. R. Jin, T. Yang, M. Mahdavi, Y.-F. Li, and Z.-H. Zhou. Improved bound for the Nyström's method and its application to kernel classification. Technical Report 1111.2262v2, arXiv, 2001.
24. T. Joachims, T. Finley, and C.-N. Yu. Cutting-plane training of structural SVMs. Machine Learning, 77(1): 27-59, 2009.
25. I. M. Johnstone. Minimax Bayes, asymptotic minimax and sparse wavelet priors. Statistical Decision Theory and Related Topics, pages 303-326, 1994.
26. S.S. Keerthi, O. Chapelle, and D. DeCoste. Building support vector machines with reduced classifier complexity. Journal of Machine Learning Research, 7: 1493-1515, 2006.
27. S. Kumar, M. Mohri, and A. Talwalkar. Sampling methods for the Nyström method. Journal of Machine Learning Research, 13: 981-1006, 2012.
28. N. D. Lawrence, M. Seeger, and R. Herbrich. Fast sparse Gaussian process methods: The informative vector machine. In Adv. NIPS, 2002.
29. M. W. Mahoney. Randomized algorithms for matrices and data. Foundations and Trends in Machine Learning, 3(2): 123-224, 2011.
30. M. W. Mahoney and P. Drineas. CUR matrix decompositions for improved data analysis. Proceedings of the National Academy of Sciences, 106(3): 697-702, 2009.
31. O. A. Maillard and R. Munos. Compressed least-squares regression. In Adv. NIPS, 2009.
32. F. Orabona, J. Keshet, and B. Caputo. The Projectron: a bounded kernel-based perceptron. In Proc. ICML, 2008.
33. J. C. Platt. Fast training of support vector machines using sequential minimal optimization. In Advances in kernel methods, pages 185-208. MIT Press, 1999.
34. A. Rahimi and B. Recht. Random features for large-scale kernel machines. Adv. NIPS, 2007.
35. C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
36. R. M. Rifkin and R. A. Lippert. Value regularization and Fenchel duality. Journal of Machine Learning Research, 8: 441-479, 2007.
37. S. Sabato, N. Srebro, and N. Tishby. Tight sample complexity of large-margin learning. In Adv. NIPS, 2010.
38. B. Schölkopf and A. J. Smola. Learning with Kernels. MIT Press, 2002.
39. B. Schölkopf, K. Tsuda, and J.P. Vert. Kernel methods in computational biology. MIT press, 2004.
40. S. Shalev-Shwartz, Y. Singer, and N. Srebro. Pegasos: Primal estimated sub-gradient solver for SVM. In Proc. ICML, 2007.
41. J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Camb. U. P., 2004.
42. A. J. Smola and B. Schölkopf. Sparse greedy matrix approximation for machine learning. In Proc. ICML, 2000.
43. P. Speckman. Spline smoothing and optimal rates of convergence in nonparametric regression models. The Annals of Statistics, 13(3): 970-983, 1985.
44. I. Steinwart, D. Hush, C. Scovel, et al. Optimal rates for regularized least squares regression. In Proc. COLT, 2009.
45. A. Talwalkar and A. Rostamizadeh. Matrix coherence and the nyström method. In Proc. UAI, 2010.
46. P. Tarrès and Y. Yao. Online learning as stochastic approximation of regularization paths. Technical Report 1103.5538, arXiv, 2011.
47. J. A. Tropp. Improved analysis of the subsampled randomized Hadamard transform. Adv. Adapt. Data Anal., 13(1-2): 115-126, 2011.
48. J. A. Tropp. User-friendly tail bounds for sums of random matrices. Foundations of Computational Mathematics, 12(4): 389-434, 2012.
49. G. Wahba. Spline Models for Observational Data. SIAM, 1990.
50. Z. Wang, K. Crammer, and S. Vucetic. Breaking the curse of kernelization: Budgeted stochastic gradient descent for large-scale SVM training. Journal of Machine Learning Research, 13: 3103-3131, 2012.
51. C. Williams and M. Seeger. Using the Nyström method to speed up kernel machines. In Adv. NIPS, 2001.
52. T. Yang, Y.-F. Li, M. Mahdavi, R. Jin, and Z.-H. Zhou. Nyström method vs. random fourier features: A theoretical and empirical comparison. In Adv. NIPS, 2012.
53. J. Zhang, M. Marszałek, S. Lazebnik, and C. Schmid. Local features and kernels for classification of texture and object categories: A comprehensive study. International Journal of Computer Vision, 73(2): 213-238, 2007.