Efficient cross-validation for kernelized least-squares regression with sparse basis expansions

Machine Learning

Volumn 87, Issue 3, 2012, Pages 381-407

  Pahikkala, Tapio ^a   Suominen, Hanna ^b,c   Boberg, Jorma ^a  

^a UNIVERSITY OF TURKU   (Finland)

^b CSIRO   (Australia)

^c AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

Author keywords

Cross validation; Hold out; Kernel methods; Least squares support vector machine; Regularized least squares; Sparse basis expansions

Indexed keywords

CROSS VALIDATION; HOLD-OUT; KERNEL METHODS; LEAST SQUARE; LEAST SQUARES SUPPORT VECTOR MACHINES; SPARSE BASIS;

ALGORITHMS; VECTORS;

LEAST SQUARES APPROXIMATIONS;

EID: 84862027224     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-012-5287-6     Document Type: Article

References (41)

1. A. Airola T. Pahikkala T. Salakoski 2011 On learning and cross-validation with decomposed Nyström approximation of kernel matrix Neural Processing Letters 33 1 17 30 10.1007/s11063-010-9159-4
2. S. An W. Liu S. Venkatesh 2007 Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression Pattern Recognition 40 8 2154 2162 1115.68125 10.1016/j.patcog.2006.12.015
3. Cauwenberghs, G., & Poggio, T. (2001). Incremental and decremental support vector machine learning. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems (Vol. 13, pp. 409-415). Cambridge: MIT Press.
4. G. C. Cawley N. L. C. Talbot 2004 Fast exact leave-one-out cross-validation of sparse least-squares support vector machines Neural Networks 17 10 1467 1475 1073.68072 10.1016/j.neunet.2004.07.002
5. K. De Brabanter J. De Brabanter J. Suykens B. De Moor 2010 Optimized fixed-size kernel models for large data sets Computational Statistics & Data Analysis 54 6 1484 1504 2600850 10.1016/j.csda.2010.01.024
6. T. G. Dietterich 1998 Approximate statistical tests for comparing supervised classification learning algorithms Neural Computation 10 7 1895 1923 10.1162/089976698300017197
7. A. Elisseeff T. Evgeniou M. Pontil 2005 Stability of randomized learning algorithms Journal of Machine Learning Research 6 55 79 2249815 1222.68192
8. I. E. Frank J. H. Friedman 1993 A statistical view of some chemometrics regression tools Technometrics 35 2 109 135 0775.62288 10.2307/1269656
9. Golub, G. H., & Van Loan, C. (1989). Matrix computations (2nd ed.). Baltimore: Johns Hopkins University Press.
10. Green, P., & Silverman, B. (1994). Nonparametric regression and generalized linear models, a roughness penalty approach. London: Chapman & Hall.
11. Horn, R., & Johnson, C. (1985). Matrix analysis. Cambridge: Cambridge University Press.
12. M. Karasuyama I. Takeuchi R. Nakano 2009 Efficient leave-m-out cross-validation of support vector regression by generalizing decremental algorithm New Generation Computing 27 307 318 1185.68535 10.1007/s00354-008- 0067-3
13. Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. In C. Mellish (Ed.), Proceedings of the fourteenth international joint conference on artificial intelligence (Vol. 2, pp. 1137-1143). San Mateo: Morgan Kaufmann.
14. Kumar, S., Mohri, M., & Talwalkar, A. (2009). Sampling techniques for the Nyström method. In D. van Dyk & M. Welling (Eds.), JMLR workshop and conference proceedings: Vol. 5. Proceedings of the 12th international conference on artificial intelligence and statistics (pp. 304-311).
15. C. Nadeau Y. Bengio 2003 Inference for the generalization error Machine Learning 52 3 239 281 1039.68104 10.1023/A:1024068626366
16. Pahikkala, T., Boberg, J., & Salakoski, T. (2006a). Fast n-fold cross-validation for regularized least-squares. In T. Honkela, T. Raiko, J. Kortela, & H. Valpola (Eds.), Proceedings of the ninth Scandinavian conference on artificial intelligence (SCAI 2006) (pp. 83-90). Espoo: Helsinki University of Technology.
17. Pahikkala, T., Tsivtsivadze, E., Boberg, J., & Salakoski, T. (2006b). Graph kernels versus graph representations: a case study in parse ranking. In T. Gärtner, G. C. Garriga, & T. Meinl (Eds.), Proceedings of the ECML/PKDD'06 workshop on mining and learning with graphs (pp. 181-188).
18. T. Pahikkala S. Pyysalo J. Boberg J. Järvinen T. Salakoski 2009 Matrix representations, linear transformations, and kernels for disambiguation in natural language Machine Learning 74 2 133 158 10.1007/s10994-008-5082-6
19. Pahikkala, T., Suominen, H., Boberg, J., & Salakoski, T. (2009b). Efficient hold-out for subset of regressors. In M. Kolehmainen, P. Toivanen, & B. Beliczynski (Eds.), Proceedings of the 9th international conference on adaptive and natural computing algorithms (pp. 350-359). Berlin: Springer.
20. T. Pahikkala E. Tsivtsivadze A. Airola J. Järvinen J. Boberg 2009 An efficient algorithm for learning to rank from preference graphs Machine Learning 75 1 129 165 10.1007/s10994-008-5097-z
21. K. Pelckmans J. De Brabanter J. Suykens B. De Moor 2005 The differogram: non-parametric noise variance estimation and its use for model selection Neurocomputing 69 1-3 100 122 10.1016/j.neucom.2005.02.015
22. K. Pelckmans J. Suykens B. De Moor 2006 Additive regularization trade-off: fusion of training and validation levels in kernel methods Machine Learning 62 217 252 10.1007/s10994-005-5315-x
23. T. Poggio F. Girosi 1990 Networks for approximation and learning Proceedings of the IEEE 78 9 1481 1497 10.1109/5.58326
24. T. Poggio S. Smale 2003 The mathematics of learning: Dealing with data Notices of the American Mathematical Society 50 5 537 544 1968413 1083.68100
25. J. Quiñonero-Candela C. E. Rasmussen 2005 A unifying view of sparse approximate Gaussian process regression Journal of Machine Learning Research 6 1939 1959 1222.68282
26. Rasmussen, C. E., & Williams, C. K. I. (2005). Gaussian processes for machine learning (adaptive computation and machine learning). Cambridge: MIT Press.
27. R. Rifkin A. Klautau 2004 In defense of one-vs-all classification Journal of Machine Learning Research 5 101 141 2247975 1222.68287
28. Rifkin, R., & Lippert, R. (2007). Notes on regularized least squares (Technical Report MIT-CSAIL-TR-2007-025). Massachusetts Institute of Technology, Cambridge, Massachusetts, USA.
29. Rifkin, R., Yeo, G., & Poggio, T. (2003). Regularized least-squares classification. In J. Suykens, G. Horvath, S. Basu, C. Micchelli, & J. Vandewalle (Eds.), NATO science series III: Computer and system sciences, Chap. 7: Vol. 190. Advances in learning theory: methods, model and applications (pp. 131-154). Amsterdam: IOS Press.
30. Saunders, C., Gammerman, A., & Vovk, V. (1998). Ridge regression learning algorithm in dual variables. In J. W. Shavlik (Ed.), Proceedings of the fifteenth international conference on machine learning (pp. 515-521). San Mateo: Morgan Kaufmann.
31. R. A. Schiavo D. J. Hand 2000 Ten more years of error rate research International Statistical Review 68 3 295 310 1107.62339 10.1111/j.1751-5823. 2000.tb00332.x
32. Schölkopf, B., Herbrich, R., & Smola, A. (2001). A generalized representer theorem. In D. Helmbold & R. Williamson (Eds.), Proceedings of the 14th annual conference on computational learning theory (pp. 416-426). Berlin: Springer.
33. Shewchuk, J. R. (1994). An introduction to the conjugate gradient method without the agonizing pain (Technical report). Carnegie Mellon University, Pittsburgh, PA, USA.
34. Smola, A., & Bartlett, P. (2001). Sparse greedy gaussian process regression. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems (Vol. 3, pp. 619-625). Cambridge: MIT Press.
35. J. Suykens J. Vandewalle 1999 Least squares support vector machine classifiers Neural Processing Letters 9 3 293 300 1721843 10.1023/A: 1018628609742
36. Suykens, J., & Vandewalle, J. (1999b). Multiclass least squares support vector machines. In International joint conference on neural networks (IJCNN'99) (Vol. 2, pp. 900-903). New York: Inst. Elect. Electronics Eng.
37. Suykens, J., Van Gestel, T., De Brabanter, J., De Moor, B., & Vandewalle, J. (2002). Least squares support vector machines. Singapore: World Scientific.
38. Vapnik, V. (1995). The nature of statistical learning theory. New York: Springer.
39. P. Vincent Y. Bengio 2002 Kernel matching pursuit Machine Learning 48 165 187 0998.68120 10.1023/A:1013955821559
40. Wahba, G. (1990). Spline models for observational data. series in applied mathematics (Vol. 59). Philadelphia: SIAM.
41. Williams, C. K. I., & Seeger, M. (2001). Using the Nyström method to speed up kernel machines. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems (Vol. 13, pp. 682-688). Cambridge: MIT Press.