Dynamics of learning in multilayer perceptrons near singularities

IEEE Transactions on Neural Networks

Volumn 19, Issue 8, 2008, Pages 1313-1328

  Cousseau, Florent ^a,b   Ozeki, Tomoko ^a,c   Amari, Shun Ichi ^a  

^a RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

^b UNIVERSITY OF TOKYO   (Japan)

^c TOKAI UNIVERSITY   (Japan)

Author keywords

Dynamics of learning; Multilayer perceptrons; Natural gradient (NGD) learning; Singularity; Standard gradient (SGD) learning

Indexed keywords

ARTIFICIAL INTELLIGENCE; CLARIFICATION; COMPUTATIONAL METHODS; COMPUTER SIMULATION; CYBERNETICS; DYNAMICS; EDUCATION; ELECTRIC LOADS; GRADIENT METHODS; MATRIX ALGEBRA; MULTILAYER NEURAL NETWORKS; MULTILAYERS; NEURAL NETWORKS; PATTERN RECOGNITION SYSTEMS; STANDARDS; TRAJECTORIES;

ALGEBRAIC GEOMETRY; ANALYTICAL RESULTS; ANALYTICAL SOLUTIONS; COMPUTER SIMULATIONS; CRITICAL REGIONS; DYNAMICAL BEHAVIORS; DYNAMICS OF LEARNING; FISHER INFORMATION; INFORMATION GEOMETRY; MULTI-LAYER PERCEPTRON; MULTILAYER PERCEPTRONS; NATURAL GRADIENT; NATURAL GRADIENT (NGD) LEARNING; PARAMETER SPACES; RIEMANNIAN METRIC; SINGULARITY; STANDARD GRADIENT (SGD) LEARNING;

FISHER INFORMATION MATRIX;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; METHODOLOGY; THEORETICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; MODELS, THEORETICAL; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED;

EID: 49649091991     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2008.2000391     Document Type: Article

References (24)

1. S. Amari, "Natural gradient works efficiently in learning," Neural Comput., vol. 10, pp. 251-276, 1998.
2. M. Rattray and D. Saad, "Analysis of natural gradient descent for multilayer neural networks," Phys. Rev. E, Stat. Phys. Plasmas Fluids Relat. Interdiscip. Top., vol. 59, pp. 4523-4532, 1999.
3. S. Amari, H. Park, and T. Ozeki, "Singularities affect dynamics of learning in neuromanifolds," Neural Comput., vol. 18, pp. 1007-1065, 2006.
4. D. Saad and S. A. Solla, "On-line learning in soft committee machines," Phys. Rev. E, Stat. Phys. Plasmas Fluids Relat. Interdiscip. Top., vol. 52, pp. 4225-4243, 1995.
5. S. Amari and H. Nagaoka, Methods of Information Geometry. Oxford, U.K.: American Mathematical Society/Oxford Univ. Press, 2000.
6. S. Akaho and H. J. Kappen, "Nonmonotonic generalization bias of Gaussian mixture models," Neural Comput., vol. 12, no. 6, pp. 1411-1428, 2000.
7. S. Amari and T. Ozeki, "Differential and algebraic geometry of multilayer perceptrons," IEICE Trans., vol. E84-A, pp. 31-38, 2001.
8. S. Amari, H. Park, and T. Ozeki, Advances in Neural Information Processing Systems, T. G. Dietterich, S. Becker, and Z. Ghahramani, Eds. Cambridge, MA: MIT Press, 2002, vol. 14, pp. 343-350.
9. K. Fukumizu, "Likelihood ratio of unidentifiable models and multilayer neural networks," Ann. Statist., vol. 31, no. 3, pp. 833-851, 2003.
10. K. Fukumizu and S. Amari, "Local minima and plateaus in hierarchical structures of multilayer perceptrons," Neural Netw., vol. 13, pp. 317-327, 2000.
11. K. Hagiwara, "On the problem in model selection of neural network regression in overrealizable scenario," Neural Comput., vol. 14, pp. 1979-2002, 2002.
12. S. Watanabe, "Algebraic analysis for nonidentifiable learning machines," Neural Comput., vol. 13, pp. 899-933, 2001.
13. K. Yamazaki and S. Watanabe, "Singularities in mixture models and upper bounds of stochastic complexity," Int. J. Neural Netw., vol. 16, no. 7, pp. 1029-1038, 2003.
14. S. Amari, H. Park, and K. Fukumizu, "Adaptive method of realizing natural gradient learning for multilayer perceptrons," Neural Comput., vol. 12, pp. 1399-1409, 2000.
15. S. Amari, "Theory of adaptive pattern classifiers," IEEE Trans. Electron. Comput., vol. EC-16, no. 3, pp. 299-307, Jun. 1967.
16. T. M. Heskes and B. Kappen, "Learning processes in neural networks," Phys. Rev. A, Gen. Phys., vol. 44, pp. 2718-2726, 1991.
17. M. Rattray, D. Saad, and S. Amari, "Natural gradient descent for on-line learning," Phys. Rev. Lett., vol. 81, pp. 5461-5464, 1998.
18. S. Geman, "Some averaging and stability results for random differential equations," SIAM J. Appl. Math., vol. 36, pp. 86-105, 1980.
19. S. Amari, Differential-Geometrical Method in Statistics. Berlin, Germany: Springer-Verlag, 1985.
20. S. Amari, K. Kurata, and H. Nagaoka, "Information geometry of Boltzmann machines," IEEE Trans. Neural Netw., vol. 3, no. 2, pp. 260-271, Mar. 1992.
21. H. Park, S. Amari, and K. Fukumizu, "Adaptive natural gradient learning algorithms for various stochastic models," Neural Netw., vol. 13, pp. 755-764, 2000.
22. H. Park, M. Inoue, and M. Okada, "Online learning dynamics of multilayer perceptrons with unidentifiable parameters," J. Phys. A, Math. Gen., vol. 36, pp. 11753-11764, 2003.
23. J. Harris, Algebraic Geometry. New York: Springer-Verlag, 1992.
24. H. Hironaka, "Resolution of singularities of an algebraic variety over a field of characteristic zero," Ann. Math., vol. 79, pp. 109-326, 1964.