Learning and inference in hierarchical models with singularities

Systems and Computers in Japan

Volumn 34, Issue 7, 2003, Pages 34-42

  Amari, Shun Ichi ^a   Ozeki, Tomoko ^a   Park, Hyeyoung ^a  

^a RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

Author keywords

Bayesian predictive distribution; Gaussian random field; Information geometry; Maximum likelihood estimate; Singularity

Indexed keywords

DATA PROCESSING; GAUSSIAN NOISE (ELECTRONIC); LEARNING SYSTEMS; PARAMETER ESTIMATION;

BAYESIAN PREDICTIVE DISTRIBUTION;

HIERARCHICAL SYSTEMS;

EID: 0038030649     PISSN: 08821666     EISSN: None     Source Type: Journal    
DOI: 10.1002/scj.10353     Document Type: Review

References (25)

1. Amari S. Natural gradient works efficiently in learning. Neural Computation 1998;10:251-276.
2. Amari S, Nagaoka H. Information geometry. AMS and Oxford University Press; 2000.
3. Amari S, Ozeki T. Differential and algebraic geometry of multilayer perceptrons. IEICE Trans 2001;E84-A:31-38.
4. Amari S, Park H, Fukumizu K. Adaptive method of realizing natural gradient learning for multilayer perceptrons. Neural Computation 2000;12:1399-1409.
5. Brockett RW. Some geometric questions in the theory of linear systems. IEEE Trans Automatic Control 1976;21:449-455.
6. Chernoff H. On the distribution of the likelihood ratio. Ann Math Stat 1954;25:573-578.
7. Chernoff H, Lander E. Asymptotic distribution of the likelihood ratio test that a mixture of two binomials is a single binomial. J Stat Planning Inference 1995;43:19-40.
8. Dacunha-Castelle D, Gassiat É. Testing in locally conic models, and application to mixture models. Probability Stat 1997;1:285-317.
9. Fukumizu K. Likelihood ratio of unidentifiable models and multilayer neural networks. Research Memorandum, 780, Institute of Statistical Mathematics, 2001.
10. Fukumizu K. Generalization error of a linear neural network with a singular Fisher information matrix. Tech Rep IEICE 1996;NC96-3:17-24.
11. Fukumizu K, Amari S. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Networks 2000;13:317-327.
12. Hagiwara K, Kuno K, Usui S. On the problem in model selection of neural network regression in over-realizable scenario. Proc Int Joint Conference of Neural Networks, IV, p 461-466, 2000.
13. Hartigan JA. A failure of likelihood asymptotics for normal mixtures. Proc Barkeley Conf in Honor of J Neyman and J. Kiefer 1985;2:807-810.
14. Hotelling H. Tubes and spheres in n-spaces, and a class of statistical problems. Am J Math 1939;61:440-460.
15. Kitahara M, Hayasaka T, Toda N, Usui S. On the probability distribution of estimators of regression model using 3-layered neural networks. Workshop on Information-Based Induction Sciences (IBIS 2000), p 21-26. (in Japanese)
16. Komaki F. On asymptotic properties of predictive distributions. Biometrika 1996;83:299-313.
17. Kuriki S, Takemura A. Tail probabilities of the maxima of multilinear forms and their applications. Ann Stat 2001;29:328-371.
18. Lindsay BG. Mixture models: Theory, geometry, and applications. Institute of Mathematical Statistics; 1995.
19. MCLachlan G, Peel D. Finite mixture models. John Wiley & Sons; 2000.
20. Park H, Amari S, Fukumizu K. Adaptive natural gradient learning algorithms for various stochastic models. Neural Networks 2000;13:755-764.
21. Watanabe S. Algebraic analysis for non-identifiable learning machines. Neural Computation 2001;13:899-933.
22. Watanabe S. Algebraic information geometry for learning machines with singularities. Adv Neural Inf Process Syst 2001;13:329-336.
23. Watanabe S. Algebraic geometry of learning machines with singularities and their prior distributions. Trans Japan Soc Artif Intell 2001;16:308-315. (in Japanese)
24. Weyl H. On the volume of tubes. Am J Math 1939;61:461-472.
25. Yamazaki K, Watanabe S. A stochastic algorithm to calculate the prediction error for learning machines with singularities. Tech Rep IEICE 2000;NC2000-64:23-30. (in Japanese)