Singularity and slow convergence of the em algorithm for gaussian mixtures

Neural Processing Letters

Volumn 29, Issue 1, 2009, Pages 45-59

  Park, Hyeyoung ^a   Ozeki, Tomoko ^b  

^a Kyungpook National University   (South Korea)

^b TOKAI UNIVERSITY   (Japan)

Author keywords

EM algorithm; Gradient descent learning; Learning dynamics; Singularity; Slow convergence

Indexed keywords

ALGORITHMS; ARRAY PROCESSING; CONVERGENCE OF NUMERICAL METHODS; DYNAMICS; EDUCATION; ELECTRIC NETWORK ANALYSIS; LEARNING SYSTEMS; MAXIMUM LIKELIHOOD; MAXIMUM LIKELIHOOD ESTIMATION; TRELLIS CODES;

EM ALGORITHM; GRADIENT DESCENT LEARNING; LEARNING DYNAMICS; SINGULARITY; SLOW CONVERGENCE;

LEARNING ALGORITHMS;

EID: 60449094542     PISSN: 13704621     EISSN: 1573773X     Source Type: Journal    
DOI: 10.1007/s11063-009-9094-4     Document Type: Article

References (23)

1. Amari S, Nagaoka H (2000) Information geometry. AMS and Oxford University Press, New York
2. S Amari H Park K Fukumizu 2000 Adaptive method of realizing natural gradient learning for multilayer perceptrons Neural Comput 12 1399 1409
3. Amari S, Park H, Ozeki T (2002) Geometrical singularities in the neuromanifold of multilayer perceptrons. In: Dietterich TG, Becker S, Ghahramani Z (eds) Advances in neural information processing systems, vol 14. MIT Press, Cambridge, MA, pp 343-350
4. S Amari T Ozeki H Park 2003 Learning and inference in hierarchical models with singularities Syst Comput Jpn 34 34 42
5. S Amari H Park T Ozeki 2006 Singularities affect dynamics of learning in neuromanifolds Neural Comput 18 5 1007 1065
6. K Fukumizu 2003 Likelihood ratio of unidentifiable models and multilayer neural networks Ann Stat 31 3 833 851
7. K Fukumizu S Amari 2000 Local minima and plateaus in hierarchical structures of multilayer perceptrons Neural Netw 13 317 327
8. K Hagiwara 2002 On the problem in model selection of neural network regression in overrealizable scenario Neural Comput 14 1979 2002
9. K Hagiwara T Hayasaka N Toda S Usui K Kuno 2001 Upper bound of the expected training error of neural network regression for a Gaussian noise sequence Neural Netw 14 1419 1429
10. Hartigan JA (1985) A failure of likelihood asymptotics for normal mixtures. In: LeCam L, Olshen RA (eds) Proceedings of the Berkeley conference in honor of Jerzy Neyman and Jack Kiefer, vol II. Wadsworth, Monterey, CA, pp 807-810
11. M Inoue H Park M Okada 2003 On-line learning theory of soft committee machines with correlated hidden units-steepest gradient descent and natural gradient descent J Phys Soc Jpn 72 4 805 810
12. McLachlan G, Peel D (2000) Finite mixture models. Wiley, New York
13. J Ma L Xu 2005 Asymptotic convergence properties of the EM algorithm with respect to the overlap in the mixtures Neurocomputing 68 105 129
14. J Ma L Xu MI Jordan 2000 Asymptotic convergence rate of the EM algorithm for Gaussian mixtures Neural Comput 12 2881 2907
15. H Park S Amari K Fukumizu 2000 Adaptive natural gradient learning algorithms for various stochastic models Neural Netw 13 755 764
16. H Park M Inoue M Okada 2003 Online learning dynamics of multilayer perceptrons with unidentifiable parameters J Phys A Math Gen 36 11753 11764
17. H Park M Inoue M Okada 2004 Slow dynamics due to singularities of hierarchical learning machines Prog Theor Phys Suppl 157 275 279
18. RA Redner HF Walker 1984 Mixture densities, maximum likelihood and the EM algorithm SIAM Rev 26 159 239
19. D Saad A Solla 1995 On-line learning in soft committee machines Phys Rev E 52 4225 4243
20. S Watanabe 2001 Algebraic analysis for non-identifiable learning machines Neural Comput 13 899 933
21. S Watanabe 2001 Algebraic geometrical methods for hierarchical learning machines Neural Netw 14 8 1049 1060
22. H Wei J Zhang F Cousseau T Ozeki S Amari 2006 Dynamics of learning near singularities in layered networks Neural Comput 20 813 843
23. L Xu MI Jordan 1996 On convergence properties of the EM algorithm for gaussian mixtures Neural Comput 8 129 151