Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory

Journal of Machine Learning Research

Volumn 11, Issue , 2010, Pages 3571-3594

  Watanabe, Sumio ^a  

^a TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

Author keywords

Birational invariant; Cross validation; Information criterion; Singular learning machine

Indexed keywords

AKAIKE INFORMATION CRITERION; ASYMPTOTIC BEHAVIORS; ASYMPTOTIC EQUIVALENCE; BIRATIONAL INVARIANT; CROSS VALIDATION; EXPECTATION VALUES; GENERALIZATION ERROR; GEOMETRICAL STRUCTURE; HYPER-PARAMETER; INFORMATION CRITERION; LEARNING MACHINES; LEARNING THEORY; LEAVE-ONE-OUT; MODEL SELECTION; SINGULAR LEARNING MACHINE; STATISTICAL MODELS; TRAINING SAMPLE;

LEARNING SYSTEMS; RANDOM VARIABLES; STATISTICAL TESTS;

ASYMPTOTIC ANALYSIS;

EID: 79551500649     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (46)

1. H. Akaike. A new look at the statistical model identification. IEEE Trans. on Automatic Control, 19:716-723, 1974.
2. S. Amari. A universal theorem on learning curves, Neural Networks, 6(2):161-166, 1993.
3. M. Aoyagi. Stochastic complexity and generalization error of a restricted Boltzmann machine in Bayesian estimation. Journal of Machine Learning Research, 11:1243-1272, 2010.
4. M. Aoyagi, S.Watanabe. Stochastic complexities of reduced rank regression in Bayesian estimation. Neural Networks, 18(7):924-933, 2005.
5. M.F. Atiyah. Resolution of singularities and division of distributions. Communications of Pure and Applied Mathematics, 13:145-150. 1970.
6. M.W. Browne. Cross-Validation Methods. Journal of Mathematical Psychology, 44:108-132, 2000.
7. H. Cramer. Mathematical Methods of Statistics. Princeton University Press, 1949.
8. M. Drton, B. Sturmfels, and S. Sullivant. Lecures on Algebraic Statistics. Birkhäuser, Berlin, 2009.
9. S. Geisser. The predictive sample reuse method with applications. Journal of the American Statistical Association, 70:320-328, 1975.
10. I.M. Gelfand and G.E. Shilov. Generalized Functions. Academic Press, San Diego, 1964.
11. A.E. Gelfand, D.K. Dey, H. Chang. Model determination using predictive distributions with implementation via sampling-based method. Bayesian Statistics, 4:147-167, Oxford University Press, Oxford, 1992.
12. A. Gelman, J.B. Carlin, H.S. Stern, D.B. Rubin. Bayesian Data Analysis. Chapman & Hall CRC, Boca Raton, 2004.
13. K. Hagiwara. On the problem in model selection of neural network regression in overrealizable scenario. Neural Computation, 14:1979-2002, 2002.
14. J. A. Hartigan. A failure of likelihood asymptotics for normal mixtures. In Proceedings of the Barkeley Conference in Honor of J. Neyman and J. Kiefer, Vol. 2, pages 807-810, 1985.
15. H. Hironaka. Resolution of singularities of an algebraic variety over a field of characteristic zero. Annals of Mathematics, 79:109-326, 1964.
16. M. Kashiwara. B-functions and holonomic systems. Inventiones Mathematicae, 38:33-53, 1976.
17. J. Kollór, S.Mori, C.H.Clemens, A.Corti. Birational Geometry of Algebraic Varieties. Cambridge Tract in Mathematics Cambridge University Press, Cambridge, 1998.
18. C.I. Mosier. Problems and designs of cross-validation. Educational and Psychological Measurement, 11:5-11, 1951.
19. M. Mustata. Singularities of pairs via jet schemes. Journal of the American Mathematical Society, 15:599-615. 2002.
20. E. Levin, N. Tishby, S.A. Solla. A statistical approaches to learning and generalization in layered neural networks. Proceedings of IEEE, 78(10):1568-1574. 1990.
21. S. Lin. Asymptotic approximation of marginal likelihood integrals. arXiv:1003.5338, 2010.
22. H. Linhart, W. Zucchini. Model Selection. John Wiley and Sons, New York, 1986.
23. K. Nagata and S. Watanabe, Asymptotic behavior of exchange ratio in exchange Monte Carlo method. International Journal of Neural Networks, 21(7):980-988, 2008.
24. T. Oaku. Algorithms for the b-function and D-modules associated with a polynomial. Journal of Pure Applied Algebra, 117:495-518, 1997.
25. M. Peruggia. On the variability of case-detection importance sampling weights in the Bayesian linear model. Journal of American Statistical Association, 92:199-207, 1997.
26. A.E. Raftery, M.A. Newton, J.M. Satagopan, P.N. Krivitsly. Estimating the integrated likelihood via posterior simulation using the harmonic mean identity. Bayesian Statistics, 8:1-45, Oxford University Press, Oxford, 2007.
27. D. Rusakov, D. Geiger. Asymptotic model selection for naive Bayesian network. Journal of Machine Learning Research. 6:1-35, 2005.
28. M. Saito. On real log canonical thresholds, arXiv:0707.2308v1, 2007.
29. G. Schwarz. Estimating the dimension of a model. Annals of Statistics, 6(2):461-464. 1978.
30. D.J. Spiegelhalter, N.G. Best, B.P. Carlin, A. Linde. Bayesian measures of model complexity and fit. Journal of Royal Statistical Society, Series B, 64(4):583-639, 2002.
31. M. Stone. An asymptotic equivalence of choice of model by cross-validation and Akaike's criterion. Journal of the Royal Statistical Society. 39(B):44-47, 1977.
32. A. Takemura, T.Kuriki. On the equivalence of the tube and Euler characteristic methods for the distribution of the maximum of the gaussian fields over piecewise smooth domains. Annals of Applied Probability, 12(2):768-796, 2002.
33. A. W. van der Vaart, J. A. Wellner. Weak Convergence and Empirical Processes. Springer, 1996.
34. A. Vehtari, J. Lampinen. BayesianModel Assessment and Comparison Using Cross-Validation Predictive Densities. Neural Computation, 14(10):2439-2468, 2002.
35. S. Watanabe. Generalized Bayesian framework for neural networks with singular Fisher information matrices. In the Proceedings of the International Symposium on Nonlinear Theory and Its applications, pages 207-210, 1995.
36. S. Watanabe. Algebraic analysis for nonidentifiable learning machines. Neural Computation, 13(4):899-933, 2001a. (Pubitemid 33594310)
37. S. Watanabe. Algebraic geometrical methods for hierarchical learning machines. Neural Networks. 14(8):1049-1060, 2001b. (Pubitemid 32846604)
38. S. Watanabe. Almost all learning machines are singular. In the Proceedings of the IEEE Int. Conf. FOCI, pages 383-388, 2007.
39. S. Watanabe. Algebraic Geometry and Statistical Learning Theory. Cambridge University Press, Cambridge, UK, 2009.
40. S. Watanabe. Equations of states in singular statistical estimation. Neural Networks. 23(1):20-34, 2010a.
41. S. Watanabe. Equations of states in statistical learning for an unrealizable and regular case. IEICE Transactions. E93A(3):617-626, 2010b.
42. S. Watanabe. A limit theorem in singular regression problem. Advanced Studies of Pure Mathematics. 57:473-492, 2010c.
43. S. Watanabe. Asymptotic learning curve and renormalizable condition in statistical learning theory. Journal of Physics Conference Series, 233, No.012014, 2010d.
44. K. Yamazaki, S. Watanabe. Singularities in mixture models and upper bounds of stochastic complexity. Neural Networks. 16(7):1029-1038, 2003.
45. K. Yamazaki, S. Watanabe. Algebraic geometry and stochastic complexity of hidden Markov models. Neurocomputing, 69:62-84, 2005. (Pubitemid 41644424)
46. P. Zwiernik. An asymptotic approximation of the marginal likelihood for general Markov models. arXiv:1012.0753v1, 2010.