The α-EM algorithm: Surrogate likelihood maximization using α-logarithmic information measures

IEEE Transactions on Information Theory

Volumn 49, Issue 3, 2003, Pages 692-706

  Matsuyama, Yasuo ^a  

^a WASEDA UNIVERSITY   (Japan)

Author keywords

EM algorithm; logarithm; Convergence speed; Convex divergence; Exponential family; Independent component analysis; Minorization maximization; Supervised and unsupervised learning; Surrogate function; Vector quantization

Indexed keywords

ALGORITHMS; COMPUTER SIMULATION; CONVERGENCE OF NUMERICAL METHODS; INDEPENDENT COMPONENT ANALYSIS; LEARNING SYSTEMS; MAGNETIC RESONANCE IMAGING; MATRIX ALGEBRA; NONLINEAR PROGRAMMING; PROBABILITY DENSITY FUNCTION; VECTOR QUANTIZATION;

CONVERGENCE SPEED; CONVEX DIVERGENCE; EXPECTATION MAXIMIZATION ALGORITHM; EXPONENTIAL FAMILY; LIKELIHOOD MAXIMIZATION; SURROGATE FUNCTION;

MAXIMUM LIKELIHOOD ESTIMATION;

EID: 0037355593     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2002.808105     Document Type: Article

References (41)

1. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm (with discussiob)," J. Roy. Statist. Soc. B, vol. 39, pp. 1-38, 1977.
2. T. K. Moon, "The expectation-maximization algorithm," IEEE Signal Processing Society Mag., pp. 47-60, Nov. 1996.
3. M. A. Tanner, Tools for Statistical Inference, 3rd ed. New York: Springer-Verlag, 1996.
4. G. J. McLachlan and T. Krishnan, The EM Algorithm and Extensions. New York: Wiley-Interscience, 1997.
5. X.-L. Meng and D. van Dyk, "The EM algorithm-An old folk-song sung to a fast new tune (with discussion)," J. Roy. Statist. Soc. B, vol. 59, pp. 511-567, 1997.
6. J. M. Ortega and W. C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables. New York: Academic, 1970, sec. 8.3 (d).
7. K. Lange, D. R. Hunter, and I. Yang, "Optimization transfer using surrogate objective functions (with discussion)," J. Comput. Graph. Statist., vol. 9, pp. 1-59, 2000.
8. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley-Intersciences, 1991.
9. A. Rényi, "On measures of entropy and information," in Proc. 4th Berkeley Symp. Mathematical Statistics and Probability, vol. 1, 1960, pp. 547-561.
10. J. Havrda and F. Charvát, "Qualification method of classification processes: Concept of structural α-entropy," Kybernetika, vol. 3, pp. 30-35, 1967.
11. I. Csiszár, "Information-type measures of difference of probability distributions and indirect observations," Studia Sci. Math. Hung., vol. 2, pp. 299-318, 1967.
12. S. Amari, "Differential geometry theory of statistics," in Differential Geometry in Statistical Inference, S.-I. Amari, O. E. Barndorff-Nielsen, R. E. Kass, S. L. Luritzen, and C. R. Rao, Eds. Hayward, CA: Inst. Math. Statist., 1985, vol. 10, Institute of Mathematical Statistics Lecture Notes, pp. 21-94.
13. J. N. Kapur and H. K. Kesavan, Entropy Optimization Principles with Applications. San Diego, CA: Academic, 1992.
14. S. Amari and H. Nagaoka, Methods of Information Geometry. Tokyo, Japan: Iwanami, 1993.
15. A. N. Iusem, B. F. Svaiter, and M. Teboulle, "Entropy-like proximal methods in convex programming," Math. Oper. Res., vol. 19, pp. 790-814, 1994.
16. Y. Matsuyama, "The weighted EM learning and monitoring structure," in Information Processing Society of Japan, 54th Conv. Rec., vol. 2, Mar. 1997, pp. 161-162.
17. ____, The α-EM Algorithm: A Block Connectable Generalized Learning Tool for Neural Networks (Lecture Notes in Computer Science). Berlin, Germany: Springer-Verlag, 1997, vol. 1240, pp. 483-492.
18. ____, "The weighted EM algorithm and block monitoring," in Proc. Int. Conf. Neural Networks, vol. 3, June 1997, pp. 1936-1941.
19. ____, "Non-logarithmic information measures α-EM algorithms, and speedup of learning," in Proc. Int. Symp. Information Theory, Cambridge, MA, Aug. 1998, p. 385.
20. ____, "The α-EM algorithm and its basic properties," Trans. Inst. Electron. Inform. Commun. Eng., vol. J82-D-I, pp. 1347-1358, 1999.
21. I. A. Ibragimov and R. Z. Has'minskii, Statistical Estimation: Asymptotic Theory. New York: Springer-Verlag, 1981.
22. M. J. Schervish, Theory of Statistics. New York: Springer-Verlag, 1995.
23. G. B. Folland, Real Analysis: Modern Techniques and Their Applications, 2nd ed. New York: Wiley, 1999.
24. C. F. J. Wu, "On the convergence properties of the EM algorithm," Ann. Statist., vol. 11, pp. 95-103, 1883.
25. W. I. Zangwill, Nonlinear Programming: A Unified Approach. Englewood Cliffs, NJ: Prentice-Hall, 1969.
26. D. G. Luenberger, Introduction to Linear and Nonlinear Programming. Reading, MA: Addison-Wesley, 1973.
27. X. L. Meng and D. B. Rubin, "Maximum likelihood estimation via the ECM algorithm: A general framework," Biometrika, vol. 80, pp. 267-278, 1993.
28. C. Liu and D. B. Rubin, "The ECME algorithm: A simple extension of EM and ECM wuth faster monotone convergence," Biometrika, vol. 81, pp. 633-648, 1994.
29. J. A. Fessler and A. O. Hero, "Space-alternating generalized expectation-maximization algorithm," IEEE Trans. Signal Processing, vol. 42, pp. 2664-2677, Oct. 1994.
30. R. A. Redner and H. F. Walker, "Mixture densities, maximum likelihood and the EM algorithm," SIAM Rev., vol. 26, pp. 195-239, 1984.
31. L. Xu and M. I. Jordan, "On convergence properties of the EM algorithm for Gaussian mixtures," Neural Comput., vol. 8, pp. 129-151, 1995.
32. Y. Matsuyama, "Harmonic competition: A self-organizing multiple criteria optimization," IEEE Trans. Neural Networks, vol. 7, pp. 652-668, May 1996.
33. ____, "Multiple descent cost competition: Restorable self-organization and multimedia information processing," IEEE Trans. Neural Networks, vol. 9, pp. 106-122, Jan. 1998.
34. Y. Matsuyama, N. Takeda, S. Furukawa, and T. Niimoto, "A hierarchy from α-EM algorithm to vector quantization and self-organization," in Proc. Int. Conf. Neural Information Processing, vol. 1, Kita-Kyushu, Japan, Oct. 1998, pp. 233-238.
35. L. Xu, M. I. Jordan, and G. E. Hinton, "An alternative model for mixtures of experts," in Advances in Neural Information Processing. Cambridge, MA: MIT Press, 1995, vol. 7, pp. 633-640.
36. Y. Matsuyama, S. Furukawa, N. Takeda, and T. Ikeda, "Fast α-weighted EM learning for neural networks of module mixtures," in Proc. Int. Joint Conf. Neural Networks, vol. 3, Anchorage, AK, May 1998, pp. 2306-2311.
37. A. Ben-Tal and A. Nemirovski, Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications. Philadelphia, PA: SIAM/MPS, 2001, pp. 96-96.
38. J. A. Fessler and A. O. Hero III, "Penalized maximum-likelihood image reconstruction using space-alternating generalized EM algorithm," IEEE Trans. Image Processing, vol. 4, pp. 1417-1438, Oct. 1995.
39. Y. Matsuyama, N. Katsumata, Y. Suzuki, and S. Imahara, "The α-ICA algorithm," in Proc. 2nd Int. Workshop Independent Component Analysis and Blind Signal Separation, Helsinki, Finland, June 2000, pp. 297-302.
40. Y. Matsuyama, S. Imahara, and N. Katsumata, "Optimization transfer for computational learning: A hierarchy for f-ICA and alpha-EM to their offsprings," in Proc. Int. Joint Conf. Neural Networks, vol. 3, Honolulu, HI, May 2002, pp. 1883-1888.
41. Y. Matsuyama and R. Kawamura, "Supervised map ICA: Applications to brain functional MRI," in Proc. Int. Conf. Neural Information Processing, vol. 5, Singapore, Nov. 2002, pp. 2259-2263.