Preventing over-fitting during model selection via bayesian regularisation of the hyper-parameters

Journal of Machine Learning Research

Volumn 8, Issue , 2007, Pages 841-861

  Cawley, Gavin C ^a   Talbot, Nicola L C ^a  

^a UNIVERSITY OF EAST ANGLIA   (United Kingdom)

Author keywords

Bayesian regularisation; Kernel methods; Model selection

Indexed keywords

APPROXIMATION THEORY; BAYESIAN NETWORKS; COMPUTATIONAL METHODS; COMPUTER OPERATING SYSTEMS; PARAMETER ESTIMATION; STATISTICAL METHODS;

BAYESIAN REGULARISATION; DATA SETS; KERNEL PARAMETERS; MODEL SELECTION; VARIANCE;

LEARNING SYSTEMS;

EID: 34247558132     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (52)

1. M. Allen. The relationship between variable selection and prediction. Technometrics, 16:125-127, 1974.
2. E. Anderson, Z. Bai, C. Bischof, S. Blackford, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, and D. Sorenson. LAPACK Users' Guide. SIAM Press, third edition, 1999.
3. M. Bishop. Curvature-driven smoothing: a learning algorithm for feedforward networks. IEEE Transactions on Neural Networks, 4(5):882-884, September 1993.
4. M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, 1995.
5. Bo, L. Wang, and L. Jiao. Feature scaling for kernel Fisher discriminant analysis using leave-one-out cross-validation. Neural Computation, 18:961-978, April 2006.
6. W. L. Buntine and A. S. Weigend. Bayesian back-propagation. Complex Systems, 5:603-643, 1991.
7. G. C. Cawley. Leave-one-out cross-validation based model selection criteria for weighted LS-SVMs. In Proceedings of the International Joint Conference on Neural Networks (IJCNN-2006), pages 2970-2977, Vancouver, BC, Canada, July 16-21 2006.
8. G. C. Cawley and N. L. C. Talbot. Efficient leave-one-out cross-validation of kernel Fisher discriminant classifiers. Pattern Recognition, 36(11):2585-2592, November 2003.
9. G. C. Cawley and N. L. C. Talbot. Fast leave-one-out cross-validation of sparse least-squares support vector machines. Neural Networks, 17(10):1467-1475, December 2004.
10. G. C. Cawley and N. L. C. Talbot. Approximate leave-one-out cross-validation for kernel logistic regression. Machine Learning (submitted), 2007.
11. Chapelle, V. Vapnik, O. Bousquet, and S. Mukherjee. Choosing multiple parameters for support vector machines. Machine Learning, 46(1):131-159, 2002.
12. Cortes and V. Vapnik. Support vector networks. Machine Learning, 20:273-297, 1995.
13. J. Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1-30, 2006.
14. H. Drucker and Y. Le Cun. Improving generalization performance using double back-propagation. IEEE Transactions on Neural Networks, 3(6):991-997, 1992.
15. S. Geman, E. Bienenstock, and R. Doursat. Neural networks and the bias/variance dilema. Neural Computation, 4(1):1-58, 1992.
16. G. H. Golub and C. F. Van Loan. Matrix Computations. The Johns Hopkins University Press, Baltimore, third edition edition, 1996.
17. S. Gradshteyn and I. M. Ryzhic. Table of Integrals, Series and Products. Academic Press, fifth edition, 1994.
18. Guyon, A. R. Saffari Azar Alamdari, G. Dror, and J. Buhmann. Performance prediction challenge. In Proceedings of the International Joint Conference on Neural Networks (IJCNN-2006), pages 1649-1656, Vancouver, BC, Canada, July 16-21 2006.
19. T. Joachims. Learning to Classify Text using Support Vector Machines - Methods, Theory and Algorithms. Kluwer Academic Publishers, 2002.
20. S. S. Keerthi, K. B. Duan, S. K. Shevade, and A. N. Poo. A fast dual algorithm for kernel logistic regression. Machine Learning, 61(1-3):151-165, November 2005.
21. G. S. Kimeldorf and G. Wahba. Some results on Tchebycheffian spline functions. J. Math. Anal. Applic., 33:82-95, 1971.
22. R. Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the Fourteenth International Conference on Artificial Intelligence (IJCAI), pages 1137-1143, San Mateo, CA, 1995. Morgan Kaufmann.
23. P. A. Lachenbruch and M. R. Mickey. Estimation of error rates in discriminant analysis. Technometrics, 10(1): 1-11, February 1968.
24. A. Luntz and V. Brailovsky. On estimation of characters obtained in statistical procedure of recognition (in Russian). Techicheskaya Kibernetica, 3, 1969.
25. J. C. MacKay. Bayesian interpolation. Neural Computation, 4(3):415-447, 1992a.
26. J. C. MacKay. The evidence framework applied to classification networks. Neural Computation, 4(5):720-736, 1992b.
27. J. C. MacKay. A practical Bayesian framework for backprop networks. Neural Computation, 4 (3):448-472, 1992c.
28. J. C. MacKay. Hyperparameters: Optimise or integrate out? In G. Heidbreder, editor, Maximum Entropy and Bayesian Methods. Kluwer, 1994.
29. J. Mercer. Functions of positive and negative type and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society of London, A, 209:415-446, 1909.
30. A. Micchelli. Interpolation of scattered data: Distance matrices and conditionally positive definite functions. Constructive Approximation, 2:11-22, 1986.
31. S. Mika, G. Ratsch, J. Weston, B. Schölkopf, and K.-R. Müller. Fisher discriminant analysis with kernels. In Neural Networks for Signal Processing, volume IX, pages 41-48. IEEE Press, New York, 1999.
32. S. Mika, G. Rätsch, J. Weston, B. Schölkopf, A. J. Smola, and K.-R. Müller. Invariant feature extraction and classification in feature spaces. In S. A. Solla, T. K. Leen, and K.-R. Müller, editors, Advances in Neural Information Processing Systems, volume 12, pages 526-532. MIT Press, 2000.
33. th Annual Conference on Uncertainty in Artificial Intelligence, pages 362-369. Morgan Kauffmann, 2001.
34. J. A. Nelder and R. Mead. A simplex method for function minimization. Computer Journal, 7: 308-313, 1965.
35. st International Conference on Machine Learning, pages 671-678, Banff, Alberta, Canada, July 4-8 2004.
36. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press, 2006.
37. G. Rätsch, T. Onoda, and K.-R. Müller. Soft margins for AdaBoost. Machine Learning, 42(3): 287-320, 2001.
38. K. Saadi, N. L. C. Talbot, and G. C. Cawley. Optimally regularised kernel Fisher discriminant analysis. In Proceedings of the 17th International Conference on Pattern Recognition (ICPR-2004), volume 2, pages 427-430, Cambridge, United Kingdom, August 23-26 2004.
39. W. S. Sarle. Stopped training and other remidies for overfitting. In Proceedings of the 27th Symposium on the Interface of Computer Science and Statistics, pages 352-360, Pittsburgh, PA, USA, June 21-24 1995.
40. B. Schölkopf, K. Tsuda, and J.-P. Vert. Kernel Methods in Computational Biology. MIT Press, 2004.
41. T. Seaks. SYMINV : An algorithm for the inversion of a positive definite matrix by the Cholesky decomposition. Econometrica, 40(5):961-962, September 1972.
42. J. Shawe-Taylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, 2004.
43. Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society, B 36(1):111-147, 1974.
44. S. Sundararajan and S. S. Keerthi. Predictive approaches for choosing hyperparameters in Gaussian processes. Neural Computation, 13(5):1103-1118, May 2001.
45. J. A. K. Suykens, T. Van Gestel, J. De Brabanter, B. De Moor, and J. Vandewalle. Least Squares Support Vector Machines. World Scientific, 2002.
46. J. A. K. Suykens and J. Vandewalle. Least squares support vector machine classifiers. Neural Processing Letters, 9(3):293-300, June 1999.
47. A. N. Tikhonov and V. Y. Arsenin. Solutions of Ill-posed Problems. John Wiley, New York, 1977.
48. E. Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research, 1:211-244, June 2000.
49. G. Wahba. Spline Models for Observational Data. SIAM Press, Philadelphia, PA, 1990.
50. K. I. Williams and D. Barber. Bayesian classification with Gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12):1342-1351, December 1998.
51. P. M. Williams. A Marquardt algorithm for choosing the step size in backpropagation learning with conjugate gradients. Technical Report CSRP-229, University of Sussex, February 1991.
52. P. M. Williams. Bayesian regularization and pruning using a Laplace prior. Neural Computation, 7 (1):117-143, 1995.