The subspace information criterion for infinite dimensional hypothesis spaces

Journal of Machine Learning Research

Volumn 3, Issue 2, 2003, Pages 323-359

  Sugiyama, Masashi ^a   Müller, Klaus Robert ^b,c  

^a TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

^b FRAUNHOFER FIRST   (Germany)

^c UNIVERSITY OF POTSDAM   (Germany)

Author keywords

Cross validation; Finite sample statistics; Gaussian processes; Generalization error; Kernel regression; Model selection; Reproducing kernel Hilbert space; Subspace information criterion; Unbiased estimators

Indexed keywords

ERROR ANALYSIS; INFORMATION SCIENCE; LEARNING ALGORITHMS; THEOREM PROVING;

SUPPORT VECTOR MACHINES;

LEARNING SYSTEMS;

EID: 0041965965     PISSN: 15324435     EISSN: None     Source Type: Journal    
DOI: 10.1162/153244303765208412     Document Type: Article

References (74)

1. H. Akaike. A new look at the statistical model identification. IEEE Transactions on Automatic Control, AC-19(6):716-723, 1974.
2. H. Akaike. Likelihood and the Bayes procedure. In N. J. Bernardo, M. H. DeGroot, D. V. Lindley, and A. F. M. Smith, editors, Bayesian Statistics, pages 141-166, Valencia, 1980. University Press.
3. A. Albert. Regression and the Moore-Penrose Pseudoinverse. Academic Press, New York and London, 1972.
4. S. Amari, N. Fujita, and S. Shinomoto. Four types of learning curves. Neural Computation, 4(4):605-618, 1992.
5. N. Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society, 68:337-404, 1950.
6. C. M. Bishop. Neural Networks for Pattern Recognition. Clarendon Press, Oxford, 1995.
7. O. Bunke and B. Droge. Bootstrap and cross-validation estimates of the prediction error for linear regression models. Annals of Statistics, 12:1400-1424, 1984.
8. C. J. C. Burges. A tutorial on support vector machines for pattern recognition. Knowledge Discovery and Data Mining, 2(2):121-167, 1998.
9. S. S. Chen, D. L. Donoho, and M. A. Saunders. Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing, 20(1):33-61, 1998.
10. V. Cherkassky, X. Shao, F. M. Mulier, and V. N. Vapnik. Model complexity control for regression using VC generalization bounds. IEEE Transactions on Neural Networks, 10(5):1075-1089, 1999.
11. D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active learning with statistical models. Journal of Artificial Intelligence Research, 4:129-145, 1996.
12. P. Craven and G. Wahba. Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross-validation. Numerische Mathematik, 31:377-403, 1979.
13. N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press, Cambridge, UK, 2000.
14. I. Daubechies. Ten Lectures on Wavelets. Society for Industrial and Applied Mathematics, Philadelphia and Pennsylvania, 1992.
15. L. Devroye, L. Györfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Number 31 in Applications of mathematics. Springer, New York, 1996.
16. D. L. Donoho. De-noising by soft thresholding. IEEE Transactions on Information Theory, 41(3):613-627, 1995.
17. D. L. Donoho and I. M. Johnstone. Ideal spatial adaptation via wavelet shirinkage. Biometrika, 81:425-455, 1994.
18. V. V. Fedorov. Theory of Optimal Experiments. Academic Press, New York, 1972.
19. K. Fukumizu. Statistical active learning in multilayer perceptrons. IEEE Transactions on Neural Networks, 11(1):17-26, 2000.
20. S. Geman, E. Bienenstock, and R. Doursat. Neural networks and the bias/variance dilemma. Neural Computation, 4(1):1-58, 1992.
21. M. N. Gibbs. Bayesian Gaussian Processes for Regression and Classification. PhD thesis, Cambridge University, 1997.
22. M. N. Gibbs and D. J. C. MacKay. Efficient implementation of Gaussian processes, 1997. Available electronically at ftp://www.inference.phy.cam.ac.uk/pub/www/mng10/GP/gpros.ps.gz.
23. F. Girosi. An equivalence between sparse approximation and support vector machines. Neural Computation, 10(6): 1455-1480, 1998.
24. R. E. Henkel. Tests of Significance. SAGE Publication, Beverly Hills, 1979.
25. T. Heskes. Bias/variance decompositions for likelihood-based estimators. Neural Computation, 10(6):1425-1433, 1998.
26. P. J. Huber. Robust Statistics. John Wiley, New York, 1981.
27. J. J. Hunter. A survey of generalized inverses and their use in stochastic modeling. Research Letters in the Information and Mathematical Sciences, 1(1):25-36, 2000.
28. A. K. Jain and R. C. Dubes. Algorithms for Clustering Data. Prentice Hall, 1988.
29. I. T. Jolliffe. Principal Component Analysis. Springer-Verlag, New York, 1986.
30. G. S. Kimeldorf and G. Wahba. A correspondence between Bayesan estimation on stochastic processes and smoothing by splines. Annals of Mathematical Statistics, 41(2):495-502, 1970.
31. T. Kohonen. Self-Organizing Maps. Springer, Berlin, 1995.
32. S. Konishi and G. Kitagawa. Generalized information criteria in model selection. Biometrika, 83:875-890, 1996.
33. D. J. C. MacKay. Bayesian interpolation. Neural Computation, 4(3):415-447, 1992a.
34. D. J. C. MacKay. Information-based objective functions for active data selection. Neural Computation, 4(4):590-604, 1992b.
35. C. L. Mallows. Choosing a subset regression. Presented at the Central Regional Meeting of the Institute of Mathematical Statistics, 1964.
36. p. Technometrics, 15(4):661-675, 1973.
37. K.-R. Müller, S. Mika, G. Rätsch, K. Tsuda, and B. Schölkopf. An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks, 12(2):181-201, 2001.
38. N. Murata, S. Yoshizawa, and S. Amari. Network information criterion - Determining the number of hidden units for an artificial neural network model. IEEE Transactions on Neural Networks, 5(6):865-872, 1994.
39. R. Nishii. Asymptotic properties of criteria for selection of variables in multiple regression. Annals of Statistics, 12:758-765, 1984.
40. M. J. L. Orr. Introduction to radial basis function networks. Technical report, Center for Cognitive Science, University of Edinburgh, 1996. Available electronically at http://www.anc.ed.ac.uk/̃mjo/papers/intro.ps.gz.
41. C. E. Rasmussen, R. M. Neal, G. E. Hinton, D. van Camp, M. Revow, Z. Ghahramani, R. Kustra, and R. Tibshirani. The DELVE manual, 1996. Available electronically at http://www.cs.toronto.edu/̃delve/.
42. J. Rissanen. Modeling by shortest data description. Automatica, 14:465-471, 1978.
43. J. Rissanen. Stochastic complexity. Journal of the Royal Statistical Society, Series B, 49(3):223-239, 1987.
44. J. Rissanen. Fisher information and stochastic complexity. IEEE Transactions on Information Theory, IT-42(1):40-47, 1996.
45. C. Saunders, A. Gammerman, and V. Vovk. Ridge regression learning algorithm in dual variables. In Proceedings of the 15th International Conference on Machine Learning, pages 515-521, 1998.
46. R. Schatten. Norm Ideals of Completely Continuous Operators. Springer-Verlag, Berlin, 1970.
47. B. Schölkopf, C. J. C. Burges, and A. J. Smola, editors. Advances in Kernel Methods: Support Vector Machines. The MIT Press, Cambridge, MA, 1998.
48. B. Schölkopf, A. Smola, R. Williamson, and P. Bartlett. New support vector algorithms. Neural Computation, 12(5):1207-1245, 2000.
49. B. Schölkopf and A. J. Smola. Learning with Kernels. MIT Press, Cambridge, MA, 2002.
50. G. Schwarz. Estimating the dimension of a model. Annals of Statistics, 6:461-464, 1978.
51. R. Shibata. An optimal selection of regression variables. Biometrika, 68(1):45-54, 1981.
52. A. J. Smola and B. Schölkopf. Sparse greedy matrix approximation for machine learning. In Proceedings of ICML-2000, International Conference on Machine Learning, pages 911-918, 2000.
53. A. J. Smola, B. Schölkopf, and K.-R. Müller. The connection between regularization operators and support vector kernels. Neural Networks, 11(4):637-649, 1998.
54. N. Sugiura. Further analysis of the data by Akaike's information criterion and the finite corrections. Communications in Statistics: Theory and Methods, 7(1):13-26, 1978.
55. M. Sugiyama, D. Imaizumi, and H. Ogawa. Subspace information criterion for image restoration - Optimizing parameters in linear filters. IEICE Transactions on Information and Systems, E84-D (9): 1249-1256, 2001.
56. M. Sugiyama and H. Ogawa. Incremental active learning for optimal generalization. Neural Computation, 12(12):2909-2940, 2000.
57. M. Sugiyama and H. Ogawa. Subspace information criterion for model selection. Neural Computation, 13(8):1863-1889, 2001.
58. M. Sugiyama and H. Ogawa. Optimal design of regularization term and regularization parameter by subspace information criterion. Neural Networks, 15(3):349-361, 2002a.
59. M. Sugiyama and H. Ogawa. Theoretical and experimental evaluation of subspace information criterion. Machine Learning, Special Issue on New Methods for Model Selection and Model Combination, 48(1/2/3):25-50, 2002b.
60. M. Sugiyama and H. Ogawa. A unified method for optimizing linear image restoration filters. Signal Processing, 82(11):1773-1787, 2002c.
61. K. Takeuchi. Distribution of information statistics and validity criteria of models. Mathematical Science, 153:12-18, 1976. (In Japanese).
62. A. Tanaka, H. Imai, and M. Miyakoshi. Choosing the parameter of image restoration filters by modified subspace information criterion. IEICE Transactions on Fundamentals, E85-A (5):1104-1110, 2002.
63. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58(1):267-288, 1996.
64. K. Tsuda, M. Sugiyama, and K.-R. Müller. Subspace information criterion for non-quadratic regularizers - Model selection for sparse regressors. IEEE Transactions on Neural Networks, 13(1):70-80, 2002.
65. V. Vapnik and O. Chapelle. Bounds on error expectation for support vector machines. Neural Computation, 12(9):2013-2036, 2000.
66. V. N. Vapnik. Estimation of Dependencies Based on Empirical Data. Springer-Verlag, New York, 1982.
67. V. N. Vapnik. The Nature of Statistical Learning Theory. Springer-Verlag, Berlin, 1995.
68. V. N. Vapnik. Statistical Learning Theory. John Wiley & Sons, Inc., New York, 1998.
69. H. Wahba. Spline Model for Observational Data. Society for Industrial and Applied Mathematics, Philadelphia and Pennsylvania, 1990.
70. S. Watanabe. Algebraic analysis for non-identifiable learning machines. Neural Computation, 13(4):899-933, 2001.
71. C. K. I. Williams. Prediction with Gaussian processes: From linear regression to linear prediction and beyond. In M. I. Jordan, editor, Learning in Graphical Models, pages 599-621. The MIT Press, Cambridge, 1998.
72. C. K. I. Williams and C. E. Rasmussen. Gaussian processes for regression. In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, editors, Advances in Neural Information Processing Systems, volume 8, pages 514-520. The MIT Press, 1996.
73. P. M. Williams. Bayesian regularization and pruning using a Laplace prior. Neural Computation, 7(1):117-143, 1995.
74. K. Yamanishi. A decision-theoretic extension of stochastic complexity and its application to learning. IEEE Transactions on Information Theory, IT-44(4): 1424-1439, 1998.