PAC-Bayesian generalisation error bounds for Gaussian process classification

Journal of Machine Learning Research

Volumn 3, Issue 2, 2003, Pages 233-269

  Seeger, Matthias ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

Bayesian Learning; Convex Duality; Gaussian Processes; Generalisation Error Bounds; Gibbs Classifier; Kernel Machines; PAC Bayesian Framework; Sparse Approximations

Indexed keywords

FEATURE EXTRACTION; LEARNING ALGORITHMS; PROBABILITY; THEOREM PROVING;

GAUSSIAN PROCESSES;

LEARNING SYSTEMS;

EID: 0041464774     PISSN: 15324435     EISSN: None     Source Type: Journal    
DOI: 10.1162/153244303765208386     Document Type: Article

References (51)

1. Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2:499-526, 2002.
2. Christopher Burges. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2):121-167, 1998.
3. Thomas Cover and Joy Thomas. Elements of Information Theory. Series in Telecommunications. John Wiley & Sons, 1st edition, 1991.
4. N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector Machines and Other Kernel Based Methods. Cambridge University Press, 2000.
5. Lehel Csató and Manfred Opper. Sparse on-line Gaussian processes. Neural Computation, 14:641-668, 2002.
6. Mark N. Gibbs. Bayesian Gaussian Processes for Regression and Classification. PhD thesis, University of Cambridge, 1997.
7. Thore Graepel, Ralf Herbrich, and Robert Williamson. Prom margin to sparsity. In T. Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems 13, pages 210-216. MIT Press, 2001.
8. P.J. Green and Bernhard Silverman. Nonparametric Regression and Generalized Linear Models. Monographs on Statistics and Probability. Chapman & Hall, 1994.
9. David Haussler, Michael Kearns, and Robert Schapire. Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension. Machine Learning, 14:83-113, 1994.
10. David Haussler and Manfred Opper. Mutual information, metric entropy and cumulative relative entropy risk. Annals of Statistics, 25(6):2451-2492, 1997.
11. Ralf Herbrich. Learning Kernel Classifiers. MIT Press, 1st edition, 2001.
12. Ralf Herbrich and Thore Graepel. A PAC-Bayesian margin bound for linear classifiers: Why SVMs work. In T. Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems 13, pages 224-230. MIT Press, 2001.
13. Ralf Herbrich and Robert Williamson. Algorithmic luckiness. In T. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems 14, pages 391-397. MIT Press, 2002.
14. Roger Horn and Charles Johnson. Matrix Analysis. Cambridge University Press, 1st edition, 1985.
15. Shunsuke Ihara. Information Theory for Continuous Systems. World Scientific, 1st edition, 1993.
16. Tommi Jaakkola and David Haussler. Probabilistic kernel regression models. In D. Heckerman and J. Whittaker, editors, Workshop on Artificial Intelligence and Statistics 7. Morgan Kaufmann, 1999.
17. R. E. Kaas and A. E. Raftery. Bayes factors and model uncertainty. Journal of the American Statistical Association, 90:773-795, 1995.
18. S. Kullback. Information Theory and Statistics. John Wiley & Sons, 1959.
19. John Langford and Rich Caruana. (Not) bounding the true error. In T. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems 14, pages 809-816. MIT Press, 2002.
20. John Langford and Matthias Seeger. Bounds for averaging classifiers. Technical Report CMU-CS-01-102, Carnegie Mellon University, January 2001.
21. Neil D. Lawrence, Matthias Seeger, and Ralf Herbrich. Fast sparse Gaussian process methods: The informative vector machine. To appear in Neural Information Processing Systems 15, 2002.
22. Nick Littlestone and Manfred Warmuth. Relating data compression and learnability. Technical report, University of California, Santa Cruz, 1986.
23. Z. Luo and G. Wahba. Hybrid adaptive splines. Journal of the American Statistical Association, 92:107-116, 1997.
24. K. V. Mardia, J. T. Kent, and J. M. Bibby. Multivariate Statistics. Academic Press, 1st edition, 1979.
25. David McAllester. PAC-Bayesian model averaging. In Conference on Computational Learning Theory 12, pages 164-170, 1999a.
26. David McAllester. Some PAC-Bayesian theorems. Machine Learning, 37(3):355-363, 1999b.
27. David McAllester. PAC-Bayesian stochastic model selection. To appear in Machine Learning. See www.autoreason.com., 2002.
28. P. McCullach and J.A. Nelder. Generalized Linear Models. Number 37 in Monographs on Statistics and Applied Probability. Chapman & Hall, 1st edition, 1983.
29. Ron Meir and Tong Zhang. Data-dependent bounds for Bayesian mixture methods. To appear in Neural Information Processing Systems 15. See citeseer.nj.nee.com/536920.html, 2002.
30. Thomas Minka. A Family of Algorithms for Approximate Bayesian Inference. PhD thesis, Massachusetts Institute of Technology, January 2001.
31. Radford M. Neal. Monte Carlo implementation of Gaussian process models for Bayesian classification and regression. Technical Report 9702, Department of Statistics, University of Toronto, January 1997.
32. Manfred Opper and Ole Whither. Gaussian processes for classification: Mean field algorithms. Neural Computation, 12(11):2655-2684, 2000.
33. John C. Platt. Fast training of support vector machines using sequential minimal optimization. In B. Schölkopf, C. Burges, and A. Smola, editors, Advances in Kernel Methods: Support Vector Learning, pages 185-208. MIT Press, 1998.
34. William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. Numerical Recipes in C. Cambridge University Press, 2nd edition, 1992.
35. R. Rockafellar. Convex Analysis. Princeton University Press, 1970.
36. F. Rosenblatt. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6):386-408, 1958.
37. Bernhard Schölkopf and Alexander Smola. Learning with Kernels. MIT Press, 1st edition, 2002.
38. Matthias Seeger. Bayesian model selection for support vector machines, Gaussian processes and other kernel classifiers. In S. Solla, T. Leen, and K.-R. Müller, editors, Advances in Neural Information Processing Systems 12, pages 603-609. MIT Press, 2000.
39. Matthias Seeger. PAC-Bayesian generalization error bounds for Gaussian process classification. Technical Report EDI-INF-RR-0094, Division of Informatics, University of Edinburgh, 2002. See www.dai.ed.ac.uk/̃seeger/papers.html.
40. Matthias Seeger, Neil D. Lawrence, and Ralf Herbrich. Sparse Bayesian learning: The informative vector machine. Technical report, Department of Computer Science, Sheffield, UK, 2002. See www.dcs.shef.ac.uk/̃neil/papers/.
41. John Shawe-Taylor, Peter L. Bartlett, Robert C. Williamson, and Martin Anthony. Structural risk minimization over data-dependent hierarchies. IEEE Transactions on Information Theory, 44(5): 1926-1940, 1998.
42. Alex Smola and Peter Bartlett. Sparse greedy Gaussian process regression. In T. Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems 13, pages 619-625. MIT Press, 2001.
43. Peter Sollich. Learning curves for Gaussian processes. In M. Kearns, S. Solla, and D. Cohn, editors, Advances in Neural Information Processing Systems 11, pages 344-350. MIT Press, 1999.
44. Michael Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research, 1:211-244, 2001.
45. Volker Tresp. A Bayesian committee machine. Neural Computation, 12(11):2719-2741, 2000.
46. L. G. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134-1142, 1984.
47. Vladimir N. Vapnik. Statistical Learning Theory. Wiley, 1st edition, 1998.
48. Grace Wahba. Spline Models for Observational Data. CBMS-NSF Regional Conference Series. SIAM Society for Industrial and Applied Mathematics, 1990.
49. Christopher Williams and Matthias Seeger. Using the Nyström method to speed up kernel machines. In T. Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems 18, pages 682-688. MIT Press, 2001.
50. Christopher K. I. Williams. Prediction with Gaussian processes: From linear regression to linear prediction and beyond. In M. I. Jordan, editor, Learning in Graphical Models. Kluwer, 1997.
51. Christopher K.I. Williams and David Barber. Bayesian classification with Gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12):1342-1351, 1998.