Dynamics of learning near singularities in layered networks

Neural Computation

Volumn 20, Issue 3, 2008, Pages 813-843

  Wei, Haikun ^a,b,c   Zhang, Jun ^a,d   Cousseau, Florent ^a,e   Ozeki, Tomoko ^a,f   Amari, Shun Ichi ^a  

^a RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

^b SOUTHEAST UNIVERSITY   (China)

^c KYUSHU INSTITUTE OF TECHNOLOGY   (Japan)

^d UNIVERSITY OF MICHIGAN   (United States)

^e UNIVERSITY OF TOKYO   (Japan)

^f TOKAI UNIVERSITY   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; NONLINEAR SYSTEM; NORMAL DISTRIBUTION;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS; NORMAL DISTRIBUTION;

EID: 41549141920     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2007.12-06-414     Document Type: Article

References (41)

1. Amari, S. (1967). Theory of adaptive pattern classifiers. IEEE Trans. Electronic Computers, EC-16(3), 299-307.
2. Amari, S. (1977). Neural theory of association and concept-formation. Biological Cybernetics, 26, 175-185.
3. Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10, 251-276.
4. Amari, S., & Nagaoka, H. (2000). Methods of information geometry. New York: Oxford University Press.
5. Amari, S., & Nakahara, H. (2005). Difficulty of singularity in population coding. Neural Computation, 17, 839-858.
6. Amari, S., & Ozeki, T. (2001). Differential and algebraic geometry of multilayer perceptrons. IEICE Trans., E84-A, 31-38.
7. Amari, S., Park, H., & Ozeki, T. (2002). Geometrical singularities in the neuromanifold of multilayer perceptrons. In T. G. Dietterich, S. Becker, & Z. Ghahramani (Eds.), Advances in neural information processing systems, 14 (pp. 343-350). Cambridge, MA: MIT Press.
8. Amari, S., Park, H., & Ozeki, T. (2006). Singularities affect dynamics of learning in neuromanifolds. Neural Computation, 18, 1007-1065.
9. Biehl, M., & Caticha, N. (2002). Statistical mechanics of on line learning and generalization. In M. A. Arbib (Ed.), Handbook of brain theory and neural networks. Cambridge, MA: MIT Press.
10. Biehl, M., Riegler, P., & Wöhler, C. (1996). Transient dynamics of online learning in two-layered neural networks. J. Phys. A: Math. Gen., 29, 4769-4780.
11. Biehl, M., & Schwarze, H. (1995). Learning by online gradient descent. J. Phys. A: Math. Gen., 28, 643-656.
12. Chen, A., Lu, H., & Hecht-Nielsen, R. (1993). On the geometry of feedforward neural network error surfaces. Neural Computation, 5, 910-927.
13. Cousseau, F., Ozeki, T., & Amari, S. (in press). Dynamics of learning in multilayer perceptrons near the singularity. IEEE Trans. Neural Networks.
14. Dacunha-Castelle, D., & Gassiat, E. (1997). Testing in locally conic models, and application to mixture models. Probability and Statistics, 1, 285-317.
15. Feller, W. (1971). An introduction to probability theory and its applications (Vol. 2), New York: Wiley.
16. Freeman, J., & Saad, D. (1997a). Online learning in radial basis function networks. Neural Computation, 9, 1601-1622.
17. Freeman, J., & Saad, D. (1997b). Dynamics of online learning in radial basis function networks. Physical Review E, 56, 907-916.
18. Fukumizu, K. (1996). A regularity condition of information matrix of a multilayer perceptron network. Neural Networks, 9, 871-879.
19. Fukumizu, K. (2003). Likelihood ratio of unidentifiable models and multilayer neural networks. Annals of Statistics, 31, 833-851.
20. Fukumizu, K., & Amari, S. (2000). Local minima and plateaus in hierarchical structure of multilayer perceptrons. Neural Networks, 13, 317-327.
21. Hagiwara, K. (2002). On the problem in model selection of neural network regression in overrealizable scenario. Neural Computation, 14, 1979-2002.
22. Hartigan, J. (1985). A failure of likelihood asymptotics for normal mixtures. In L. M. Lecam & R. A. Olshen (Eds.), Proc. Berkeley Conf. in Honor of J. Neyrnan and J. Kiefer (Vol. 2, pp. 807-810). Belmont, CA: Wadsworth.
23. Heskes, T., & Kappen, B. (1991). Learning process in neural networks. Physical Review, A44, 2718-2762.
24. Huh, N., Oh, J., & Kang, K. (2000). Online learning of a mixture-of-experts neural network. J. Phys. A: Math. Gen., 33, 8663-8672.
25. Inoue, M., Park, H., & Okada, M. (2003). online learning theory of soft committee machines with correlated hidden units: Steepest gradient descent and natural gradient descent. J. Phys. Soc. Japan, 72, 805-810.
26. Inoue, M., Park, H., & Okada, M. (2004). Dynamics of the adaptive natural gradient descent method for soft committee machines. Physical Review E, 69(5), 056120.
27. Karayiannis, N., & Randolph-Gips, M. (2003). On the construction and training of reformulated radial basis function neural networks. IEEE Trans. Neural Networks, 14, 835-846.
28. Kurková, V., & Kainen, P. (1994). Functionally equivalent feedforward neural networks. Neural Computation, 6, 543-558.
29. Milnor, J. (1985). On the concept of attractor. Comm. in Math. Phys. 99, 177-195.
30. Park, H., Amari, S., & Fukumizu, K. (2000). Adaptive natural gradient learning algorithms for various stochastic models. Neural Networks, 13, 755-764.
31. Park, H., Inoue, M., & Okada, M. (2003). Online learning dynamics of multilayer perceptrons with unidentifiable parameters. J. Phys. A: Math. Gen., 36, 11753-11764.
32. Park, H., Inoue, M., & Okada, M. (2005). Slow dynamics due to singularities of hierarchical learning machines. Progress of Theoretical Physics Supplement, 157, 275-279.
33. Riegler, P., & Biehl, M. (1995). Online backpropagation in two-layered neural networks. J. Phys. A: Math.Gen., 28, L507-L513.
34. Saad, D., & Solla, A. (1995a). On-line learning in soft committee machines. Physical Review E, 52, 4225-4243.
35. Saad, D., & Solla, A. (1995b). Exact solution for online learning in multilayer neural networks. Physical Review Letters, 74, 4337-4340.
36. Sussmann, H. (1992). Uniqueness of the weights for minimal feedforward nets with a given input-output map. Neural Networks, 5, 589-593.
37. Watanabe, S. (2001a). Algebraic analysis for non-identifiable learning machines. Neural Computation, 13, 899-933.
38. Watanabe, S. (2001b). Algebraic geometrical methods for hierarchical learning machines. Neural Networks, 14, 1409-1060.
39. Watanabe, S., & Amari, S. (2003). Learning coefficients of layered models when the true distribution mismatches the singularities. Neural Computation, 15, 1013-1033.
40. Wei, H., & Amari, S. (2006). Online learning dynamics of radial basis function neural networks near the singularity. In Proc. IJCNN2006 (pp. 4770-4776). Piscataway, NJ: IEEE Press.
41. Wei, H., & Amari, S. (in press). Dynamics of learning near singularities in radial basis function networks. Neural Networks.