Efficient approximate leave-one-out cross-validation for kernel logistic regression

Machine Learning

Volumn 71, Issue 2-3, 2008, Pages 243-264

  Cawley, Gavin C ^a   Talbot, Nicola L C ^a  

^a UNIVERSITY OF EAST ANGLIA   (United Kingdom)

Author keywords

Kernel logistic regression; Model selection

Indexed keywords

ALGORITHMS; LEAST SQUARES APPROXIMATIONS; PATTERN RECOGNITION; PROBABILITY DISTRIBUTIONS;

EXPECTATION PROPAGATION (EP); KERNEL LOGISTIC REGRESSION;

LEARNING SYSTEMS;

EID: 43049121679     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-008-5055-9     Document Type: Article

References (51)

1. Allen, D. M. (1974). The relationship between variable selection and prediction. Technometrics, 16, 125-127.
2. Anderson, E., Bai, Z., Bischof, C., Blackford, S., Demmel, J., Dongarra, J., Du Croz, J., Greenbaum, A., Hammarling, S., McKenney, A., & Sorensen, D. (1999). LAPACK users' guide (3rd ed.). Philadelphia: Society for Industrial and Applied Mathematics.
3. Bo, L., Wang, L., & Jiao, L. (2006). Feature scaling for kernel Fisher discriminant analysis using leave-one-out cross validation. Neural Computation, 18(4), 961-978.
4. Boser, B. E., Guyon, I. M., & Vapnik, V. (1992). A training algorithm for optimal margin classifiers. In D. Haussler (Ed.), Proceedings of the fifth annual ACM workshop on computational learning theory (pp. 144-152), Pittsburgh, PA, July 1992.
5. Boyd, S., & Vandenberghe, L. (2004). Convex optimization. Cambridge: Cambridge University Press.
6. Brown, M. P. S., Grundy, W. N., Lin, D., Cristianini, N., Sugnet, C. W., Furey, T. S., Ares, M. Jr., & Haussler, D. (2000). Knowledge-based analysis of microarray gene expression data by using support vector machines. Proceedings of the National Academy of Sciences, 97(1), 262-267.
7. Cawley, G. C., Janacek, G. J., & Talbot, N. L. C. (2007). Generalised kernel machines. In Proceedings of the IEEE/INNS international joint conference on neural networks (IJCNN-07), Orlando, FL, USA, 12-17 August 2007.
8. Cawley, G. C., & Talbot, N. L. C. (2003). Efficient leave-one-out cross-validation of kernel Fisher discriminant classifiers. Pattern Recognition, 36(11), 2585-2592.
9. Cawley, G. C., & Talbot, N. L. C. (2004a). Efficient model selection for kernel logistic regression. In Proceedings of the 17th international conference on pattern recognition (ICPR-2004) (Vol. 2, pp. 439-442), Cambridge, United Kingdom, 23-26 August 2004.
10. Cawley, G. C., & Talbot, N. L. C. (2004b). Fast leave-one-out cross-validation of sparse least-squares support vector machines. Neural Networks, 17(10), 1467-1475.
11. Cawley, G. C., & Talbot, N. L. C. (2007). Preventing over-fitting in model selection via Bayesian regularization of the hyper-parameters. Journal of Machine Learning Research, 8, 841-861.
12. Cawley, G. C., Talbot, N. L. C., Janacek, G. J., & Peck, M. W. (2006). Parametric accelerated life survival analysis using sparse Bayesian kernel learning methods. IEEE Transactions on Neural Networks, 17(2), 471-481.
13. Chapelle, O. (2006). Leave k out for kernel machines. unpublished research note. 2 October 2006.
14. Chapelle, O. (2007). Training a support vector machine in the primal. Neural Computation, 19(5), 1155-1178.
15. Chapelle, O., Vapnik, V., Bousquet, O., & Mukherjee, S. (2002). Choosing multiple parameters for support vector machines. Machine Learning, 46(1), 131-159.
16. Cook, R. D., & Weisberg, S. (1982). Monographs on statistics and applied probability. Residuals and influence in regression. New York: Chapman and Hall.
17. Cortes, C., & Vapnik, V. (1995). Support vector networks. Machine Learning, 20, 273-297.
18. Geman, S., Bienenstock, E., & Doursat, R. (1992). Neural networks and the bias/variance dilemma. Neural Computation, 4(1), 1-58.
19. Golub, G. H., & Van Loan, C. F. (1996). Matrix computations (3rd. ed.). Baltimore: The Johns Hopkins University Press.
20. Green, P. J., & Silverman, B. W. (1994). Monographs on statistics and applied probability : Vol. 58. Nonparametric regression and generalized linear models-a roughness penalty approach. London: Chapman & Hall/CRC.
21. Kimeldorf, G. S., & Wahba, G. (1971). Some results on Tchebycheffian spline functions. Journal of Mathematical Analysis and Applications, 33, 82-95.
22. Lachenbruch, P. A., & Mickey, M. R. (1968). Estimation of error rates in discriminant analysis. Technometrics, 10(1), 1-11.
23. Luntz, A., & Brailovsky, V. (1969). On estimation of characters obtained in statistical procedure of recognition. Techicheskaya Kibernetica, 3 (in Russian).
24. Mercer, J. (1909). Functions of positive and negative type and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society of London, A, 209, 415-446.
25. Mika, S., Rätsch, G., Weston, J., Schölkopf, B., & Müller, K.-R. (1999). Fisher discriminant analysis with kernels. In Neural networks for signal processing (Vol. IX, pp. 41-48). New York: IEEE Press.
26. Mika, S., Rätsch, G., Weston, J., Schölkopf, B., Smola, A. J., & Müller, K.-R. (2000). Invariant feature extraction and classification in feature spaces. In S. A. Solla, T. K. Leen, & K.-R. Müller (Eds.), Advances in neural information processing systems (Vol. 12, pp. 526-532). Cambridge: MIT Press.
27. Mika, S., Rätsch, G., Weston, J., Schölkopf, B., Smola, A., & Müller, K.-R. (2003). Constructing descriptive and discriminative features: Rayleigh coefficients in kernel feature spaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(5), 623-628.
28. Minka, T. (2001). Expectation propagation for approximate Bayesian inference. In Proceedings of uncertainty in artificial intelligence (pp. 362-369).
29. Müller, K.-R., Mika, S., Rätsch, G., Tsuda, K., & Schölkopf, B. (2001). An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks, 12(2), 181-201.
30. Nabney, I. T. (1999). Efficient training of RBF networks for classification. In: Proceedings of the ninth international conference on artificial neural networks (Vol. 1, pp. 210-215), Edinburgh, United Kingdom, 7-10 September, 1999.
31. Nelder, J. A., & Mead, R. (1965). A simplex method for function minimization. Computer Journal, 7, 308-313.
32. Opper, M., & Winther, O. (2000). Gaussian processes for classification: mean-field algorithms. Neural Computation, 12(11), 2665-2684.
33. Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian processes for machine learning. Adaptive computation and machine learning. Cambridge: MIT Press.
34. Saunders, C., Gammermann, A., & Vovk, V. (1998). Ridge regression in dual variables. In J. Shavlik (Ed.), Proceedings of the fifteenth international conference on machine learning (ICML-1998). San Mateo: Morgan Kaufmann.
35. Schölkopf, B., & Smola, A. J. (2002). Learning with kernels-support vector machines, regularization, optimization and beyond. Cambridge: MIT Press.
36. Schölkopf, B., Smola, A. J., & Müller, K. (1997). Kernel principal component analysis. In W. Gerstner, A. Germond, M. Hasler, & J.-D. Nicoud (Eds.), Lecture notes in computer science : Vol. 1327. Proceedings of the international conference on artificial neural networks (ICANN-1997) (pp. 583-588). Berlin: Springer.
37. Schölkopf, B., Herbrich, R., & Smola, A. J. (2002). A generalized representer theorem. In Proceedings of the fourteenth international conference on computational learning theory (pp. 416-426), Amsterdam, The Netherlands, 16-19 July 2002.
38. Seaks, T. (1972). SYMINV: an algorithm for the inversion of a positive definite matrix by the Cholesky decomposition. Econometrica, 40(5), 961-962.
39. Shawe-Taylor, J., & Cristianini, N. (2004). Kernel methods for pattern analysis. Cambridge: Cambridge University Press.
40. Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society, B, 36(1), 111-147.
41. Sundararajan, S., & Keerthi, S. S. (2001). Predictive approaches for choosing hyperparameters in Gaussian processes. Neural Computation, 13(5), 1103-1118.
42. Suykens, J. A. K., De Brabanter, J., Lukas, L., & Vandewalle, J. (2002a). Weighted least squares support vector machines: robustness and sparse approximation. Neurocomputing, 48(1-4), 85-105.
43. Suykens, J. A. K., Van Gestel, T., De Brabanter, J., De Moor, B., & Vanderwalle, J. (2002b). Least squares support vector machines. Singapore: World Scientific.
44. Tikhonov, A. N., & Arsenin, V. Y. (1977). Solutions of ill-posed problems. New York: Wiley.
45. Vapnik, V. (1998). Statistical learning theory. New York: Wiley.
46. Vapnik, V., & Chapelle, O. (2000). Bounds on error expectation for SVM. In: A. J. Smola, P. L. Bartlett, B. Schölkopf, & D. Schuurmans (Eds.), Advances in large margin classifiers (pp. 261-280).
47. Wahba, G. (1990). Spline models for observational data. Philadelphia: SIAM.
48. Weisberg, S. (1985). Applied linear regression (2nd ed.). New York: Wiley.
49. Williams, P. M. (1991). A Marquardt algorithm for choosing the step-size in backpropagation learning with conjugate gradients. Cognitive Science Research Paper CSRP-229, University of Sussex, Brighton, UK, February 1991.
50. Williams, C. K. I., & Barber, D. (1998). Bayesian classification with Gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12), 1342-1351.
51. Zhu, J., & Hastie, T. (2005). Kernel logistic regression and the import vector machine. Journal of Computational & Graphical Statistics, 14(1), 185-205.