Theory and use of the em algorithm

Foundations and Trends in Signal Processing

Volumn 4, Issue 3, 2010, Pages 223-296

  Gupta, Maya R ^a   Chen, Yihua ^a  

^a University of Washington   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DIRICHLET DISTRIBUTIONS; EM ALGORITHMS; EXPECTATION-MAXIMIZATION ALGORITHMS; GAUSSIAN MIXTURE MODELS; OPTIMAL MIXTURES; PRACTICAL ISSUES; SUPERIMPOSED SIGNAL;

ESTIMATION; HIDDEN MARKOV MODELS; OBJECT RECOGNITION; OPTIMIZATION;

ALGORITHMS;

EID: 79955056761     PISSN: 19328346     EISSN: 19328354     Source Type: Journal    
DOI: 10.1561/2000000034     Document Type: Article

References (65)

1. M. M. Ali, C. Khompatraporn, and Z. B. Zabinsky, "A numerical evaluation of several stochastic algorithms on selected continuous global optimization test problems," Journal of Global Optimization, vol. 31, no. 4, pp. 635-672, April 2005. (Pubitemid 40963404)
2. T. L. Bailey and C. Elkan, "Unsupervised learning of multiple motifs in biopolymers using expectation maximization," Machine Learning, vol. 21, pp. 51-80, 1995.
3. A. Banerjee, X. Guo, and H. Wang, "On the optimality of conditional expectation as a Bregman predictor," IEEE Transactions on Information Theory, vol. 51, no. 7, pp. 2664-2669, July 2005. (Pubitemid 41136414)
4. L. E. Baum, T. Petrie, G. Soules, and N. Weiss, "A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains," The Annals of Mathematical Statistics, vol. 41, no. 1, pp. 164-171, February 1970.
5. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, UK: Cambridge University Press, 2004.
6. R. A. Boyles, "On the convergence of the EM algorithm," Journal of the Royal Statistical Society, Series B (Methodological), vol. 45, no. 1, pp. 47-50, 1983.
7. P. Bryant and J. A. Williamson, "Asymptotic behavior of classification maximum likelihood estimates," Biometrika, vol. 65, no. 2, pp. 273-281, 1978. (Pubitemid 8379488)
8. G. Celeux and J. Diebolt, "The SEM algorithm: a probabilistic teacher algorithm derived from the EM algorithm for the mixture problem," Computational Statistics Quaterly, vol. 2, pp. 73-82, 1985.
9. G. Celeux and G. Govaert, "A classification EM algorithm for clustering and two stochastic versions," Computational Statistics Data Analysis, vol. 14, pp. 315-332, 1992.
10. Y. Chen and J. Krumm, "Probabilistic modeling of traffic lanes from GPS traces," in Proceedings of 18th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, 2010.
11. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," Journal ofthe Royal Statistical Society, Series B (Methodological), vol. 39, no. 1, pp. 1-38, 1977.
12. M. Feder and E. Weinstein, "Parameter estimation of superimposed signals using the EM algorithm," IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 36, no. 4, pp. 477-489, April 1988. (Pubitemid 18638688)
13. G. B. Folland, Real Analysis: Modern Techniques and Their Applications. New York, NY: John Wiley & Sons, 2nd Edition, 1999.
14. B. A. Frigyik, A. Kapila, and M. R. Gupta, "Introduction to the Dirichlet distribution and related processes," Department of Electrical Engineering, University of Washignton, UWEETR-2010-0006, 2010.
15. B. A. Frigyik, S. Srivastava, and M. R. Gupta, "Functional Bregman divergence and Bayesian estimation of distributions," IEEE Transactions on Information Theory, vol. 54, no. 11, pp. 5130-5139, November 2008.
16. M. Gales and S. Young, "The application of hidden Markov models in speech recognition," Foundations and Trends in Signal Processing, vol. 1, no. 3, pp. 195-304, 2008.
17. A. Gersho and R. M. Gray, Vector Quantization and Signal Compression. Norwell, MA: Kluwer, 1991.
18. A. Goldsmith, Wireless Communications. Cambridge, UK: Cambridge University Press, 2005.
19. M. I. Gurelli and L. Onural, "On a parameter estimation method for Gibbs-Markov random fields," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 16, no. 4, pp. 424-430, April 1994.
20. H. O. Hartley, "Maximum likelihood estimation from incomplete data," Biometrics, vol. 14, no. 2, pp. 174-194, June 1958.
21. T. Hastie, R. Tibshirani, and J. Friedman, The Elements ofStatistical Learning: Data Mining, Inference, and Prediction. New York, NY: Springer, 2nd Edition, 2009.
22. S. Haykin, "Cognitive radio: Brain-empowered wireless communications," IEEE Journal on Selected Areas in Communications, vol. 23, no. 2, pp. 201-220, February 2005. (Pubitemid 40245487)
23. M. Hazen and M. R. Gupta, "A multiresolutional estimated gradient architecture for global optimization," in Proceedings of the IEEE Congress on Evolutionary Computation, pp. 3013-3020, 2006. (Pubitemid 47130878)
24. M. Hazen and M. R. Gupta, "Gradient estimation in global optimization algorithms," in Proceedings of the IEEE Congress on Evolutionary Computation, pp. 1841-1848, 2009.
25. T. Hebert and R. Leahy, "A generalized EM algorithm for 3-D Bayesian reconstruction from Poisson data using Gibbs priors," IEEE Transactions on Medical Imaging, vol. 8, no. 2, pp. 194-202, June 1989.
26. T. J. Hebert and K. Lu, "Expectation-maximization algorithms, null spaces, and MAP image restoration," IEEE Transactions on Image Processing, vol. 4, no. 8, pp. 1084-1095, August 1995.
27. M. Jamshidian and R. I. Jennrich, "Acceleration of the EM algorithm by using quasi-Newton methods," Journal of the Royal Statistical Society, Series B (Methodological), vol. 59, no. 3, pp. 569-587, 1997. (Pubitemid 127686968)
28. D. R. Jones, "A taxonomy of global optimization methods based on response surfaces," Journal ofGlobal Optimization, vol. 21, no. 4, pp. 345-383, December 2001. (Pubitemid 33672374)
29. K. Lange, "Convergence of EM image reconstruction algorithms with Gibbs smoothing," IEEE Transactions on Medical Imaging, vol. 9, no. 4, pp. 439-446, December 1990.
30. K. Lange, "A gradient algorithm locally equivalent to the EM algorithm," Journal of the Royal Statistical Society, Series B (Methodological), vol. 57, no. 2, pp. 425-437, 1995.
31. J. Li and R. M. Gray, Image Segmentation and Compression Using Hidden Markov Models. New York, NY: Springer, 2000.
32. S. P. Lloyd, "Least squares quantization in PCM," IEEE Transactions on Information Theory, vol. 28, no. 2, pp. 129-137, First published in 1957 as a Bell Labs technical note, 1982.
33. L. B. Lucy, "An iterative technique for the rectification of observed distributions," Astronomical Journal, vol. 79, no. 6, pp. 745-754, June 1974.
34. D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms. Cambridge, UK: Cambridge University Press, 2003.
35. J. MacQueen, "Some methods for classification and analysis of multivariate observations," in Proceedings of the fifth Berkeley Symposium on Mathematical Statistics and Probability, pp. 281-297, 1967.
36. G. J. McLachlan and T. Krishnan, The EM Algorithm and Extensions. New York, NY: John Wiley & Sons, 2nd Edition, 2008.
37. G. J. McLachlan and D. Peel, Finite Mixture Models. New York, NY: John Wiley & Sons, 2000.
38. R. Mendes, J. Kennedy, and J. Neves, "The fully informed particle swarm: simpler, maybe better," IEEE Transactions on Evolutionary Computation, vol. 8, no. 3, pp. 204-210, June 2004.
39. X.-L. Meng and D. B. Rubin, "On the global and componentwise rates of convergence of the EM algorithm," Linear Algebra and its Applications, vol. 199, pp. 413-425, March 1994.
40. X.-L. Meng and D. A. van Dyk, "The EM algorithm - an old folk-song sung to a fast new tune," Journal ofthe Royal Statistical Society, Series B (Methodological), vol. 59, no. 3, pp. 511-567, 1997. (Pubitemid 127686967)
41. R. M. Neal and G. E. Hinton, "A view of the EM algorithm that justifies incremental, sparse, and other variants," in Learning in Graphical Models, (M. I. Jordan, ed.), MIT Press, November 1998.
42. J. K. Nelson and M. R. Gupta, "An EM technique for multiple transmitter localization," in Proceedings of the 41st Annual Conference on Information Sciences and Systems, pp. 610-615, 2007. (Pubitemid 351710489)
43. J. K. Nelson, M. R. Gupta, J. Almodovar, and W. H. Mortensen, "A quasi EM method for estimating multiple transmitter locations," IEEE Signal Processing Letters, vol. 16, no. 5, pp. 354-357, May 2009.
44. S. Newcomb, "A generalized theory of the combination of observations so as to obtain the best result," American Journal of Mathematics, vol. 8, no. 4, pp. 343-366, August 1886.
45. J. Nocedal and S. J. Wright, Numerical Optimization. New York, NY: Springer, 2nd Edition, 2006.
46. P. M. Pardalos and H. E. Romeijn, eds., Handbook of Global Optimization. Vol. 2, Norwell, MA: Kluwer, 2002.
47. K. B. Petersen and M. S. Pedersen, The Matrix Cookbook. November 2008.http://matrixcookbook.com/.
48. W. Qian and D. M. Titterington, "Stochastic relaxations and EM algorithms for Markov random fields," Journal of Statistical Computation and Simulation, vol. 40, pp. 55-69, 1992.
49. L. R. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proceedings of the IEEE, vol. 77, no. 2, pp. 257-286, February 1989.
50. R. A. Redner and H. F. Walker, "Mixture densities, maximum likelihood and the EM algorithm," SIAM Review, vol. 26, no. 2, pp. 195-239, April 1984.
51. W. H. Richardson, "Bayesian-based iterative method of image restoration," Journal of Optical Society ofAmerica, vol. 62, no. 1, pp. 55-59, 1972.
52. C. P. Robert and G. Casella, Monte Carlo Statistical Methods. New York, NY: Springer, 2nd Edition, 2004.
53. A. Roche, "EM algorithm and variants: An informal tutorial," Unpublished (available online at ftp://itp.cea.ir/pub/dsv/madic/publis/Roche-em. pdf), 2003.
54. G. Ronning, "Maximum Likelihood estimation of Dirichlet distributions," Journal ofStatistical Computation and Simulation, vol. 32, no. 4, pp. 215-221, 1989.
55. H. Stark and Y. Yang, Vector Space Projections: A Numerical Approach to Signal and Image Processing, Neural Nets, and Optics. New York, NY: John Wiley & Sons, 1998.
56. C. A. Sugar and G. M. James, "Finding the number of clusters in a dataset: An information-theoretic approach," Journal ofthe American Statistical Association, vol. 98, no. 463, pp. 750-763, September 2003. (Pubitemid 37395947)
57. M. A. Tanner and W. H. Wong, "The calculation of posterior distributions by data augmentation," Journal ofthe American Statistical Association, vol. 82, no. 398, pp. 528-540, June 1987.
58. D. A. van Dyk and X.-L. Meng, "The art of data augmentation," Journal of Computational and Graphical Statistics, vol. 10, no. 1, pp. 1-50, March 2001. (Pubitemid 33721606)
59. J. Wang, A. Dogandzic, and A. Nehorai, "Maximum likelihood estimation of compound-Gaussian clutter and target parameters," IEEE Transactions on Signal Processing, vol. 54, no. 10, pp. 3884-3898, October 2006. (Pubitemid 44500841)
60. G. C. G. Wei and M. A. Tanner, "A Monte Carlo implementation of the EM algorithm and the poor man's data augmentation algorithms," Journal of the American Statistical Association, vol. 85, no. 411, pp. 699-704, September 1990.
61. L. R. Welch, "Hidden Markov Models and the Baum-Welch Algorithm," IEEE Information Theory Society Newsletter, vol. 53, no. 4, pp. 1-13, December 2003.
62. C. F. J. Wu, "On the convergence properties of the EM algorithm," The Annals ofStatistics, vol. 11, no. 1, pp. 95-103, March 1983.
63. L. Xu and M. I. Jordan, "On convergence properties of the EM algorithm for Gaussian mixtures," Neural Computation, vol. 8, no. 1, pp. 129-151, January 1996. (Pubitemid 126449919)
64. R. W. Yeung, A First Course in Information Theory. New York, NY: Springer, 2002.
65. J. Zhang, "The mean field theory in EM procedures for Markov random fields," IEEE Transactions on Signal Processing, vol. 40, no. 10, pp. 2570-2583, October 1992. (Pubitemid 23555907)