Almost all learning machines are singular

Proceedings of the 2007 IEEE Symposium on Foundations of Computational Intelligence, FOCI 2007

Volumn , Issue , 2007, Pages 383-388

  Watanabe, Sumio ^a  

^a TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION THEORY; FUNCTION EVALUATION; INFORMATION THEORY; MATRIX ALGEBRA; RANDOM PROCESSES;

BOLTZMANN MACHINES; LIKELIHOOD FUNCTIONS; MATHEMATICAL FOUNDATION; SINGULAR LEARNING MACHINES;

LEARNING SYSTEMS;

EID: 34548859795     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/FOCI.2007.371500     Document Type: Conference Paper

References (27)

1. S.-i. Amari, H. Park, and T. Ozeki, "Singularities Affect Dynamics of Learning in Neuromanifolds," Neural Comput., 18(5),pp.1007-1065, 2006.
2. M.Aoyagi,S.Watanabe,"Resolution of singularities and generalization error with Bayesian estimation for layered neural network," Vol.J88-D-II, No.10, pp.2112-2124, 2005.
3. M.Aoyagi, S.Watanabe/'Stochastic complexities of reduced rank regression in Bayesian estimation," Neural Networks, Vol.18,No.7, pp.924-933, 2005.
4. M.F.Atiyah,"Resolution of singularities and division of distributions," Comm. Pure Appl. Math., Vol.13,pp. 145-150, 1970.
5. K. Hagiwara, "On the Problem in Model Selection of Neural Network Regression in Overrealizable Scenario, " Neural Comput., Vol.14,Vol.8, pp.1979-2002, 2002.
6. J.A.Hartigan,"A failure of likelihood asymptotics for normal mixture," Proc. of Barkeley Conf. in honor of Jerzy Neyman and Jack Keifer, Vol.2, pp.807-810,1985.
7. T. Hayasaka, M. Kitahara, and S. Usui, "On the Asymptotic Distribution of the Least-Squares Estimators in Unidentifiable Models," Neural Comput., Vol.16 ,No.1, pp.99 - 114, 2004.
8. H. Hironaka, "Resolution of singularities of an algebraic variety over a field of characteristic zero," Ann. of Math., Vol.79, 109-326,1964.
9. T.Hosino, K.Watanabe,S.Watanabe,"Stochastic complexity of Hidden Markov Models on the Variational Bayesian Learning," to appear in IEICE Transactions (D-II).
10. T.Hosino, K.Watanabe, S.Watanabe,"Free Energy of Stochastic Context Free Grammar on Variational Bayes. " pp.407-416, Proc. of ICONIP,2006.
11. M. Kashiwara, "B-functions and holonomic systems," Inventions Math., 38, 33-53,1976.
12. K.Nagata, S.Watanabe, "The Exchange Monte Carlo Method for Bayesian Learning in Singular Learning Machines," Proc. of WCCI2006,(Canada, Cancouver), 2006.
13. S. Nakajima, S. Watanabe, "Variational Bayes Solution of Linear Neural Networks and its Generalization Performance," Neural Computation, to appear.
14. Y. Nishiyama, S.Watanabe, "Asymptotic Behavior of Stochastic Complexity of Complete Bipartite Graph-type Boltzmann Machines," Proc. of ICONIP2006, (Chaina,HongKong) to appear, 2006.
15. K.Watanabe, S.Watanabe, "Stochastic Complexities of Gaussian Mixtures in Variational Bayesian Approximation," Journal of Machine Learning Research, Vol.7, (Apr),pp. 625-644, 2006.
16. K.Watanabe, S.Watanabe, "Stochastic complexities of general mixture models in variational Bayesian leaning," Neural Networks, to appear.
17. S.Watanabe, "Generalized Bayesian framework for neural networks with singular Fisher information matrices," Proc. of International Symposium on Nonlinear Theory and Its applications, (Las Vegas), pp.207-210, 1995.
18. S.Watanabe, "Algebraic analysis for singular statistical estimation," Proc. of International Journal of Algorithmic Learning Theory, Lecture Notes on Computer Sciences, 1720, pp.39-50, 1999.
19. S. Watanabe, "Algebraic information geometry for learning machines with singularities," Advances in Neural Information Processing Systems, (Denver, USA), pp.329-336. 2001.
20. S.Watanabe, "Algebraic Analysis for Nonidentifiable Learning Machines," Neural Computation, Vol.13, No.4, pp.899-933, 2001.
21. S. Watanabe, "Algebraic geometrical methods for hierarchical learning machines," Neural Networks, Vol.14, No.8,pp.1049-1060, 2001.
22. S. Watanabe, "Learning efficiency of redundant neural networks in Bayesian estimation," IEEE Transactions on Neural Networks, Vol.12, No.6, 1475-1486,2001.
23. S.Watanabe, S.-I.Amari,"Leaming coefficients of layered models when the true distribution mismatches the singularities", Neural Computation, Vol.15,No.5,1013-1033, 2003.
24. S.Watanabe,"Algebraic geometry of singular learning machines and symmetry of generalization and training errors," Neurocomputing, Vol.67,pp. 198-213,2005.
25. K.Yamazaki, S.Watanabe,"Singularities in mixture models and upper bounds of stochastic complexity." International Journal of Neural Networks, Vol.16, No.7, pp.1029-1038,2003.
26. K.Yamazaki, S.Watanabe," Singularities in Complete bipartite graph-type Boltzmann machines and upper bounds of stochastic complexities", IEEE Trans. on Neural Networks, Vol. 16 (2), pp.312-324, 2005.
27. K. Yamazaki and S. Watanabe, "Algebraic geometry and stochastic complexity of hidden Markov models", Neurocomputing, Vol.69, pp.62-84, 2005.