Learning coefficients of layered models when the true distribution mismatches the singularities

Neural Computation

Volumn 15, Issue 5, 2003, Pages 1013-1033

  Watanabe, Sumio ^a   Amari, Shun Ichi ^b  

^a TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

^b RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0042879444     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/089976603765202640     Document Type: Article

References (19)

1. Akaike, H. (1980). Likelihood and Bayes procedure. In J. M. Bernald, Bayesian statistics (pp. 143-166). Valencia, Spain: University Press.
2. Amari, S., Park, H., & Ozeki, T. (2002). Geometrical singularities in the neuromanifold of multilayer perceptrons. In T. G. Dietterich, S. Becker, & Z. Ghahramani (Eds.), Advances in neural information processing systems, 14. Cambridge, MA: MIT Press.
3. Atiyah, M. F. (1970). Resolution of singularities and division of distributions. Comm. Pure and Appl. Math., 13, 145-150.
4. Efron, B., & Morris, C. (1973). Stein's estimation rule and its competitors - an empirical Bayes approach. Journal of the American Statistical Association, 68, 117-130.
5. Hagiwara, K. (2002). On the problem in model selection of neural network regression in overrealizable scenario. Neural Computation, 14, 1979-2002.
6. Hartigan, J. A. (1985). A failure of likelihood ratio asymptotics for normal mixtures. In Proceedings of the Berkeley Conference in Honor of J. Neyman and J. Kiefer, 2 (pp. 807-810). Monterey, CA: Wadsworth Advanced Books, and Hayward, CA: Institute of Mathematical Statistics.
7. Hironaka, H. (1964). Resolution of singularities of an algebraic variety over a field of characteristic zero. Annals of Mathematics, 79, 109-326.
8. Kass, R. E., & Steffey, D. (1989). Approximate Bayesian inference in conditionally independent hierarchical models (parametric empirical Bayes models). Journal of the American Statistical Association, 84, 717-726.
9. Rusakov, D., & Geiger, D. (in press). Asymptotic model selection for naive Bayesian networks. In Proceedings of UAI'02. San Mateo, CA: Morgan Kaufmann.
10. Stein, C. (1956). Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proc. of the 3rd Berkeley Symp. on Mathematical Statistics and Probability, 1 (pp. 197-206). Berkeley: University of California.
11. Watanabe, K., & Watanabe, S. (in press). Upper bounds of Bayes generalization errors in reduced rank approximation. IEICE Transactions, D-II-86J.
12. Watanabe, S. (1995). A generalized Bayesian framework for neural networks with singular Fisher information matrices. International Symposium on Nonlinear Theory and Its Applications, 2, 207-210.
13. Watanabe, S. (1999). Algebraic analysis for singular statistical estimation. In Lecture notes in computer science, 1720 (pp. 39-50). Berlin: Springer-Verlag.
14. Watanabe, S. (2001a). Algebraic analysis for nonidentifiable learning machines. Neural Computation, 13(4), 899-933.
15. Watanabe, S. (2001b). Algebraic information geometry for learning machines with singularities. In T. K. Leen, T. G. Dietterich, & K.-R. Müller (Eds.), Advances in neural information processing systems, 13 (pp. 329-336). Cambridge, MA: MIT Press.
16. Watanabe, S. (2001c). Algebraic geometrical methods for hierarchical learning machines. International Journal of Neural Networks, 14(8), 1049-1060.
17. Watanabe, S. (2001d). Learning efficiency of redundant neural networks in Bayesian estimation. IEEE Trans. on Neural Networks, 12(6), 1475-1486.
18. Watanabe, S., & Amari, S. (forthcoming). The effect of singularities in a learning machine when the true parameters do not lie on singularities. In Advances in neural information processing systems, 15.
19. Yamazaki, K., & Watanabe, S. (2002). Resolution of singularities in mixture models and the upper bounds of the stochastic complexity. In Proceedings of the International Conference on Neural Information Processing.