Using the EM algorithm to train neural networks: Misconceptions and a new algorithm for multiclass classification

IEEE Transactions on Neural Networks

Volumn 15, Issue 3, 2004, Pages 738-749

  Ng, Shu Kay ^a   McLachlan, Geoffrey John ^a  

^a UNIVERSITY OF QUEENSLAND   (Australia)

Author keywords

Expectation conditional maximization (ECM) algorithm; Expectation maximization (EM) algorithm; Mixture of experts; Multiclass classification; Multilayer perceptron (MLP); Variational relaxation

Indexed keywords

ASYMPTOTIC STABILITY; COMPUTATIONAL COMPLEXITY; COMPUTER SIMULATION; CONVERGENCE OF NUMERICAL METHODS; EXPERT SYSTEMS; LEARNING ALGORITHMS; LEARNING SYSTEMS; PATTERN RECOGNITION; VARIATIONAL TECHNIQUES;

EXPECTATION-CONDITIONAL MAXIMIZATION ALGORITHM; EXPECTATION-MAXIMIZATION ALGORITHM; MIXTURE OF EXPERTS; MULTICLASS CLASSIFICATION; VARIATIONAL RELAXATION;

MULTILAYER NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK;

ALGORITHMS; NEURAL NETWORKS (COMPUTER);

EID: 2542607597     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2004.826217     Document Type: Article

References (35)

1. S. Amari, "Information geometry of the EM algorithm and em algorithms for neural networks," Neural Networks, vol. 8, pp. 1379-1408, 1995.
2. B. Cheng and D. M. Titterington, "Neural networks: A review from a statistical perspective (with discussion)," Statist. Sci., vol. 9, pp. 2-54, 1994.
3. B. D. Ripley, "Statistical aspects of neural networks," in Networks and Chaos - Statistical and Probabilistic Aspects, O.E. Barndorff-Nielsen, J. L. Jensen, and W. S. Kendall, Eds. London, U.K.: Chapman Hall, 1993, pp. 40-123.
4. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm (with discussion)," J. Roy. Statist. Soc. B, vol. 39, pp. 1-38, 1977.
5. M. I. B. Chai, A. Chai, and P. Sullivan, "Boundary detection of retinoblastoma tumors with neural networks," Comput. Med. Imaging Graph., vol. 25, pp. 257-264, 2001.
6. S. Ma and C. Ji, "Fast training of recurrent networks based on the EM algorithm," IEEE Trans. Neural Networks, vol. 9, pp. 11-26, 1998.
7. V. Tresp, T. Briegel, and J. Moody, "Neural-network models for the blood glucose metabolism of a diabetic," IEEE Trans. Neural Networks, vol. 10, pp. 1204-1213, Sept. 1999.
8. Y. Yamamoto and P. N. Nikiforuk, "A new supervised learning algorithm for multilayered and interconnected neural networks," IEEE Trans. Neural Networks, vol. 11, pp. 36-46, Jan. 2000.
9. S. Amari, "Information geometry of neural networks - An overview," in Mathematics of Neural Networks Models, Algorithms, and Applications, S. W. Ellacott, J. C. Mason, and I. J. Anderson, Eds. Boston, MA: Kluwer, 1997, pp. 15-23.
10. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, "Adaptive mixtures of local experts," Neural Comput., vol. 3, pp. 79-87, 1991.
11. M. I. Jordan and L. Xu, "Convergence results for the EM approach to mixtures of experts architectures," Neural Networks, vol. 8, pp. 1409-1431, 1995.
12. M. I. Jordan and R. A. Jacobs, "Hierarchical mixtures of experts and the EM algorithm," Neural Comput., vol. 6, pp. 181-214, 1994.
13. R. A. Jacobs, F. Peng, and M. A. Tanner, "A bayesian approach to model selection in hierarchical mixtures-of-experts architectures," Neural Networks, vol. 10, pp. 231-241, 1997.
14. G. J. McLachlan and T. Krishnan, The EM Algorithm and Extensions. New York: Wiley, 1997.
15. K. Chen, L. Xu, and H. Chi, "Improved learning algorithms for mixture of experts in multiclass classification," Neural Networks, vol. 12, pp. 1229-1252, 1999.
16. R. Langari, L. Wang, and J. Yen, "Radial basis function networks, regression weights, and the expectation-maximization algorithm," IEEE Trans. Syst., Man, Cybern. A, vol. 27, pp. 613-623, 1997.
17. L. Wang and R. Langari, "Sugeno model, fuzzy discretization, and the EM algorithm," Fuzzy Sets Syst., vol. 82, pp. 279-288, 1996.
18. J. S. Bridle, "Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition," in Neuro-Computing: Algorithms, Architectures and Applications, F. F. Soulié and J. Hérault, Eds. Berlin, Germany: Springer, 1990, pp. 227-236.
19. B. D. Ripley, Pattern Recognition and Neural Networks. Cambridge, MA: Cambridge Univ. Press, 1996, ch. 5.
20. S. Ma, C. Ji, and J. Farmer, "An efficient EM-based training algorithm for feedforward neural networks," Neural Networks, vol. 10, pp. 243-256, 1997.
21. M. Aitkin and R. Foxall, "Statistical modeling of artificial neural networks using the multi-layer perceptron," Statist. Comput., vol. 13, pp. 227-239, 2003.
22. T. L. Lai and S. P. S. Wong, "Stochastic neural networks with applications to nonlinear time series," J. Amer. Statist. Assoc., vol. 96, pp. 968-981, 2001.
23. G. C. G. Wei and M. A. Tanner, "A monte carlo implementation of the EM algorithm and the poor man's data augmentation algorithm," J. Amer. Statist. Assoc., vol. 85, pp. 699-704, 1990.
24. R. A. Levine and G. Casella, "Implementations of the monte carlo EM algorithm," J. Comput. Graph. Statist., vol. 10, pp. 422-439, 2001.
25. M. A. Tanner, Tools for Statistical Inference, 3 ed. New York: Springer-Verlag, 1996, sec. 4.5.
26. L. K. Saul and M. I. Jordan, "Attractor dynamics in feedforward neural networks," Neural Comput., vol. 12, pp. 1313-1335, 2000.
27. R. L. Streit and T. E. Luginbuhl, "Maximum likelihood training of probabilistic neural networks," IEEE Trans. Neural Networks, vol. 5, pp. 764-783, Sept. 1994.
28. G. J. McLachlan and D. Peel, Finite Mixture Models. New York: Wiley, 2000.
29. L. H. Ungar, T. Johnson, and R. D. De Veaux, "Radial basis function neural networks for process control," in CIMPRO Proceedings, 1994, http://www.cis.upenn.edu/~ungar/papers/cimpro.ps, pp. 357-364.
30. P. A. McCullagh and J. Nelder, Generalized Linear Models, 2 ed. London, U.K.: Chapman Hall, 1989.
31. X. L. Meng and D. B. Rubin, "Maximum likelihood estimation via the ECM algorithm: A general framework," Biometrika, vol. 80, pp. 267-278, 1993.
32. X. L. Meng, "On the rate of convergence of the ECM algorithm," Ann. Statist., vol. 22, pp. 326-339, 1994.
33. N. A. Campbell and R. J. Mahon, "A multivariate study of variation in two species of rock crab of genus leptograpsus," Austral. J. Zool., vol. 22, pp. 417-425, 1974.
34. R. A. Fisher, "The use of multiple measurements in taxonomic problems," Ann. Eugen., vol. 7, pp. 179-188, 1936.
35. C. L. Blake and C. J. Merz, (1998) UCI Repository of Machine Learning Databases. Univ. Calif., Dept. Inform. Comput. Sci., Irvine, CA. [Online]. Available: http://www.ics.uci.edu/~mlearn/MLRepository.html