Singularities affect dynamics of learning in neuromanifolds

Neural Computation

Volumn 18, Issue 5, 2006, Pages 1007-1065

  Amari, Shun Ichi ^a   Park, Hyeyoung ^b   Ozeki, Tomoko ^b  

^a RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

^b Kyungpook National University   (South Korea)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33646407804     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2006.18.5.1007     Document Type: Article

References (56)

1. Akaho, S., & Kappen, H. J. (2000). Nonmonotonic generalization bias of gaussian mixture models. Neural Computation, 12, 6, 1411-1428.
2. Akaike, H. (1974). A new look at the statistical model identification. IEEE Trans. Appl. Comp., AC-19, 716-723.
3. Amari, S. (1967). Theory of adaptive pattern classifiers. IEEE Trans., EC-16(3), 299-307.
4. Amari, S. (1977). Dynamics of pattern formation in lateral-inhibition type neural fields. Biological Cybernetics. 27, 77-87.
5. Amari, S. (1987). Differential geometry of a parametric family of invertible linear systems - Riemannian metric, dual affine connections and divergence. Mathematical Systems Theory, 20, 53-82.
6. Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10, 251-276.
7. Amari, S. (2003). New consideration on criteria of model selection. In L. Rutkowski & J. Kacprzyk (Eds.), Neural networks and soft computing (Proceedings of the Sixth International Conference on Neural Networks and Soft Computing) (pp. 25-30). Heidelberg: Physica Verlag.
8. Amari, S., & Burnashev, M. V. (2003). On some singularities in parameter estimation problems. Problems of Information Transmission. 39, 352-372.
9. Amari, S., & Murata, N. (1993). Statistical theory of learning curves under entropic loss criterion. Neural Computation, 5, 140-154.
10. Amari, S., & Nagaoka, H. (2000). Information geometry. New York: AMS and Oxford University Press.
11. Amari, S., & Nakahara, H. (2005). Difficulty of singularity in population coding. Neural Computation, 17, 839-858.
12. Amari, S., & Ozeki, T. (2001). Differential and algebraic geometry of multilayer perceptrons. IEICE Trans., E84-A, 31-38.
13. Amari, S., Ozeki, T., & Park, H. (2003). Learning and inference in hierarchical models with singularities. Systems and Computers in Japan, 34, 34-42.
14. Amari, S., Park, H., & Fukumizu, K. (2000). Adaptive method of realizing natural gradient learning for multilayer perceptrons. Neural Computation, 12, 1399-1409.
15. Amari, S., Park, H., & Ozeki, T. (2001). Statistical inference in nonidentifiable and singular statistical models. J. of the Korean Statistical Society, 30(2), 179-192.
16. Amari, S., Park, H., & Ozeki, T. (2002). Geometrical singularities in the neuromanifold of multilayer perceptrons. In T. G. Dietterich, S. Becker, & Z. Ghahramani (Eds.), Advances in neural information processing systems, 14 (pp. 343-350). Cambridge, MA: MIT Press.
17. Brockett, R. W. (1976). Some geometric questions in the theory of linear systems. IEEE Trans. on Automatic Control, 21, 449-455.
18. Chen, A. M., Lu, H., & Hecht-Nielsen, R. (1993). On the geometry of feedforward neural network error surfaces. Neural Computation, 5, 910-927.
19. Dacunha-Castelle, D., & Gassiat, É. (1997). Testing in locally conic models, and application to mixture models. Probability and Statistics, 1, 285-317.
20. Fukumizu, K. (1999). Generalization error of linear neural networks in unidentifiable cases. In O. Watanabe & T. Yokomori (Eds.), Algorithmic learning theory: Proceedings of the 10th International Conference on Algorithmic Learning Theory (ALT'99) (pp. 51-62). Berlin: Springer-Verlag.
21. Fukumizu, K. (2003). Likelihood ratio of unidentifiable models and multilayer neural networks. Annals of Statistics, 32(3), 833-851.
22. Fukumizu, K., & Amari, S. (2000), Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Networks, 13, 317-327.
23. Hagiwara, K. (2002a). On the problem in model selection of neural network regression in overrealizable scenario. Neural Computation, 14, 1979-2002.
24. Hagiwara, K. (2002b). Regularization learning, early stopping and biased estimator. Neurocomputing, 48, 937-955.
25. Hagiwara, K., Hayasaka, T., Toda, N., Usui, S., & Kuno, K. (2001). Upper bound of the expected training error of neural network regression for a gaussian noise sequence. Neural Networks, 14, 1419-1429.
26. Hagiwara, K., Toda, N., & Usui, S. (1993). On the problem of applying AIC to determine the structure of a layered feed-forward neural network. Proceedings of IJCNN (Vol. 3, pp. 2263-2266). Nagoya, Japan.
27. Hartigan, J. A. (1985). A failure of likelihood asymptotics for normal mixtures. Proc. Barkeley Conf. in Honor of J. Neyman and J. Kiefer, 2, 807-810.
28. Hotelling, H. (1939). Tubes and spheres in n-spaces, and a class of statistical problems. Amer. J. Math., 61, 440-460.
29. Inoue, M., Park, H., & Okada, M. (2003). On-line learning theory of soft committee machines with correlated hidden units - Steepest gradient descent and natural gradient descent. J. Phys. Soc. Jpn., 72(4), 805-810.
30. Kang, K., Oh, J.-H., Kwon, S., & Park, Y. (1993). Generalization in a two-layer neural networks. Phys. Rev. E, 48(6), 4805-4809.
31. Kitahara, M., Hayasaka, T., Toda, N., & Usui, S. (2000). On the statistical properties of least squares estimators of layered neural networks (in Japanese). IEICE Transactions, J86-D-II, 563-570.
32. Kůrková, V., & Kainen, P. C. (1994). Functionally equivalent feedforward neural networks. Neural Computation, 6, 543-558.
33. Liu, X., & Shao, Y. (2003). Asymptotics for likelihood ratio tests under loss of identifiability. Annals of Statistics, 32(3), 807-832.
34. Minsky, M., & Papert, S. (1969). Perceptrons. Cambridge, MA: MIT Press.
35. Murata, N., Yoshizawa, S., & Amari, S. (1994). Network information criterion -determining the number of hidden units for an artificial neural network model. IEEE Transactions on Neural Networks, 5(6), 865-872.
36. Park, H., Amari, S., & Fukumizu, K. (2000). Adaptive natural gradient learning algorithms for various stochastic models. Neural Networks, 13, 755-764.
37. Park, H., Inoue, M., & Okada, M. (2003). On-line learning dynamics of multilayer perceptrons with unidentifiable parameters. Submitted to J. Phys. A: Math. Gen., 36, 11753-11764.
38. Rattray, M., & Saad, D. (1999). Analysis of natural gradient descent for multilayer neural networks. Physical Review E, 59, 4523-4532.
39. Rattray, M., Saad, D., & Amari, S. (1998). Natural gradient descent for on-line learning. Physical Review Letters, 81, 5461-5464.
40. Riegler, P., & Biehl, M. (1995). On-line backpropagation in two-layered neural networks. J. Phys. A; Math. Gen., 28, L507-L513.
41. Risssanen, J. (1986). Stochastic complexity and modeling. Ann. Statist. 14, 1080-1100.
42. Rosenblatt, F. (1961). Principles of neurodynamics. New York: Spartan.
43. Rüger, S. M., & Ossen, A. (1997). The metric of weight space. Neural Processing Letters, 5, 63-72.
44. Rumelhart, D., Hinton, G. E., & Williams, R. J. (1986). Learning internal representations by error backpropagation. In D. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing: Explorations in the microstructure of cognition (Vol. 1). Cambridge, MA: MIT Press.
45. Saad, D., & Solla, A. (1995). On-line learning in soft committee machines. Phys. Rev. E, 52, 4225-4243.
46. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-164.
47. Sussmann, H. J. (1992). Uniqueness of the weights for minimal feedforward nets with a given input-output map. Neural Networks. 5, 589-593.
48. Watanabe, S. (2001a). Algebraic analysis for non-identifiable learning machines. Neural Computation, 23, 899-933.
49. Watanabe, S. (2001b). Algebraic geometrical methods for hierarchical learning machines. Neural Networks, 24(8), 1409-1060.
50. Watanabe, S. (2001c). Algebraic information geometry for learning machines with singularities. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems, 13 (pp. 329-336). Cambridge, MA: MIT Press.
51. Watanabe, S., & Amari, S. (2003). Learning coefficients of layered models when the true distribution mismatches the singularities. Neural Computation, 15(5), 1013-1033.
52. Weyl, H. (1939). On the volume of tubes. Amer. J. Math., 61, 461-472.
53. Wu, S., Amari, S., & Nakahara, H. (2002). Population coding and decoding in a neural field: A computational study. Neural Computation, 14, 999-1026.
54. Wu, S., Nakahara, H., & Amari, S. (2001). Population coding with correlation and an unfaithful model. Neural Computation, 23, 775-797.
55. Yamazaki, K., & Watanabe, S. (2002). A probabilistic algorithm to calculate the learning curves of hierarchical learning machines with singularities. Trans. on IEICE, J85-D-II(3), 363-372.
56. Yamazaki, K., & Watanabe, S. (2003). Singularities in mixture models and upper bounds of stochastic complexity. International Journal of Neural Networks, 16(7), 1029-1038.