Randomized sketches for kernels: Fast and optimal nonparametric regression

Annals of Statistics

Volumn 45, Issue 3, 2017, Pages 991-1023

  Yang, Yun ^a   Pilanci, Mert ^b   Wainwright, Martin J ^b  

^a FLORIDA STATE UNIVERSITY   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Convex optimization; Dimensionality reduction; Kernel method; Nonparametric regression; Random projection

Indexed keywords

EID: 85020684874     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/16-AOS1472     Document Type: Article

References (35)

1. AILON, N. and LIBERTY, E. (2009). Fast dimension reduction using Rademacher series on dual BCH codes. Discrete Comput. Geom. 42 615-630. MR2556458
2. ALAOUI, A. E. andMAHONEY, M. W. (2014). Fast randomized kernel methods with statistical guarantees. Technical Report, UC Berkeley. Available at arXiv:1411.0306.
3. AMELUNXEN, D., LOTZ, M., MCCOY, M. B. and TROPP, J. A. (2014). Living on the edge: Phase transitions in convex programs with random data. Inf. Inference 3 224-294. MR3311453
4. ARONSZAJN, N. (1950). Theory of reproducing kernels. Trans. Amer. Math. Soc. 68 337-404. MR0051437
5. BACH, F. (2012). Sharp analysis of low-rank kernel matrix approximations. In International Conference on Learning Theory (COLT). Edinburgh.
6. BARTLETT, P. L., BOUSQUET, O. andMENDELSON, S. (2005). Local Rademacher complexities. Ann. Statist. 33 1497-1537. MR2166554
7. BERLINET, A. and THOMAS-AGNAN, C. (2004). Reproducing Kernel Hilbert Spaces in Probability and Statistics. Kluwer Academic, Boston, MA. MR2239907
8. BOUTSIDIS, C. andGITTENS, A. (2013). Improved matrix algorithms via the subsampled randomized Hadamard transform. SIAM J. Matrix Anal. Appl. 34 1301-1340. MR3101094
9. COHEN, M. B., NELSON, J. and WOODRUFF, D. P. (2015). Optimal approximate matrix product in terms of stable rank. Technical report.
10. DRINEAS, P. and MAHONEY, M. W. (2005). On the Nyström method for approximating a Gram matrix for improved kernel-based learning. J. Mach. Learn. Res. 6 2153-2175. MR2249884
11. GITTENS, A. and MAHONEY, M. W. (2013). Revisiting the nystrom method for improved large-scale machine learning. Preprint. Available at arXiv:1303.1849.
12. GU, C. (2002). Smoothing Spline ANOVA Models. Springer, New York. MR1876599
13. HALKO, N., MARTINSSON, P. G. and TROPP, J. A. (2011). Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53 217-288. MR2806637
14. HASTIE, T., TIBSHIRANI, R. and FRIEDMAN, J. (2001). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York. MR1851606
15. JOHNSTONE, I. M. (2016). Gaussian Estimation: Sequence and Wavelet Models. Springer, New York.
16. KIMELDORF, G. and WAHBA, G. (1971). Some results on Tchebycheffian spline functions. J. Math. Anal. Appl. 33 82-95. MR0290013
17. KOLTCHINSKII, V. (2006). Local Rademacher complexities and oracle inequalities in risk minimization. Ann. Statist. 34 2593-2656. MR2329442
18. KRAHMER, F. andWARD, R. (2011). New and improved Johnson-Lindenstrauss embeddings via the restricted isometry property. SIAM J. Math. Anal. 43 1269-1281. MR2821584
19. LEDOUX, M. (2001). The Concentration of Measure Phenomenon. Mathematical Surveys and Monographs 89. Amer. Math. Soc., Providence, RI. MR1849347
20. MAHONEY, M. W. (2011). Randomized algorithms for matrices and data. Faund. Trends Mach. Learn. 3 123-224.
21. MATOUŠEK, J. (2002). Lectures on Discrete Geometry. Graduate Texts in Mathematics 212. Springer, New York. MR1899299
22. MENDELSON, S. (2002). Geometric parameters of kernel machines. In Computational Learning Theory (Sydney, 2002). Lecture Notes in Computer Science 2375 29-43. Springer, Berlin. MR2040403
23. PILANCI, M. andWAINWRIGHT, M. J. (2015). Randomized sketches of convex programs with sharp guarantees. IEEE Trans. Inform. Theory 61 5096-5115. MR3386504
24. PILANCI, M. and WAINWRIGHT, M. J. (2016). Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squares. J. Mach. Learn. Res. 17. Paper No. 53, 38. MR3504613
25. RASKUTTI, G., WAINWRIGHT, M. J. and YU, B. (2012). Minimax-optimal rates for sparse additive models over kernel classes via convex programming. J. Mach. Learn. Res. 13 389-427. MR2913704
26. SARLÓS, T. (2006). Improved approximation algorithms for large matrices via random projections. In Foundations of Computer Science, 2006. FOCS '06. 47th Annual IEEE Symposium. Berkeley, CA.
27. SAUNDERS, C.,GAMMERMAN, A. andVOVK, V. (1998). Ridge regression learning algorithm in dual variables. In Proceedings of the Fifteenth International Conference on Machine Learning, ICML '98 515-521. Morgan Kaufmann, San Francisco, CA.
28. SHAWE-TAYLOR, J. and CRISTIANINI, N. (2004). Kernel Methods for Pattern Analysis. Cambridge Univ. Press, Cambridge.
29. STONE, C. J. (1982). Optimal global rates of convergence for nonparametric regression. Ann. Statist. 10 1040-1053. MR0673642
30. TROPP, J. A. (2011). Improved analysis of the subsampled randomized Hadamard transform. Adv. Adapt. Data Anal. 3 115-126. MR2835584
31. TROPP, J. A. (2012). User-friendly tail bounds for sums of random matrices. Found. Comput. Math. 12 389-434. MR2946459
32. VAN DE GEER, S. A. (2000). Empirical Processes in M-Estimation. Cambridge Univ. Press, Cambridge.
33. WAHBA, G. (1990). Spline Models for Observational Data. CBMS-NSF Regional Conference Series in Applied Mathematics 59. SIAM, Philadelphia, PA. MR1045442
34. WILLIAMS, C. and SEEGER, M. (2001). Using the Nyström method to speed up kernel machines. In Proceedings of the 14th Annual Conference on Neural Information Processing Systems 682-688. Vancouver, BC, Canada.
35. ZHANG, Y., DUCHI, J. C. andWAINWRIGHT, M. J. (2013). Divide and conquer kernel ridge regression. In Computational Learning Theory (COLT) Conference. Princeton, NJ.