Variational learning for Gaussian mixture models

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 36, Issue 4, 2006, Pages 849-862

  Nasios, Nikolaos ^a   Bors, Adrian G ^a  

^a UNIVERSITY OF YORK   (United Kingdom)

Author keywords

Bayesian inference; Expectation maximization algorithm; Gaussian mixtures; Maximum log likelihood estimation; Variational training

Indexed keywords

IMAGE SEGMENTATION; INFERENCE ENGINES; MATHEMATICAL MODELS; MAXIMUM LIKELIHOOD ESTIMATION; PARAMETER ESTIMATION; SIGNAL DETECTION; VARIATIONAL TECHNIQUES;

BAYESIAN INFERENCE; GAUSSIAN MIXTURE MODELS; VARIATIONAL EXPECTATION-MAXIMIZATION (VEM) ALGORITHM;

LEARNING SYSTEMS;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; COMPUTER ASSISTED DIAGNOSIS; COMPUTER SIMULATION; METHODOLOGY; NORMAL DISTRIBUTION; STATISTICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; IMAGE INTERPRETATION, COMPUTER-ASSISTED; MODELS, STATISTICAL; NORMAL DISTRIBUTION; PATTERN RECOGNITION, AUTOMATED;

EID: 33746813525     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCB.2006.872273     Document Type: Article

References (46)

1. D. M. Titterington, A. F. M. Smith, and U. E. Makov, Statistical Analysis of Finite Mixture Distributions. Hoboken, NJ: Wiley, 1985.
2. A. K. Jain, R. P. W. Duin, and J. C. Mao, "Statistical pattern recognition: A review," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 1, pp. 4-37, Jan. 2000.
3. R. A. Redner and H. F. Walker, "Mixture densities, maximum likelihood and the EM algorithm," SIAM Rev., vol. 26, no. 2, pp. 195-239, Apr. 1984.
4. A. G. Bors and M. Gabbouj, "Minimal topology for a radial basis functions neural network for pattern classification," Digit. Signal Process.: A Rev. J., vol. 4, no. 3, pp. 173-188, Jul. 1994.
5. A. G. Bors and I. Pitas, "Median radial basis function neural network," IEEE Trans. Neural Netw., vol. 7, no. 6, pp. 1351-1364, Nov. 1996.
6. I. Cha and S. A. Kassam, "RBFN restoration of nonlinearly degraded images," IEEE Trans. Image Process., vol. 5, no. 6, pp. 964-975, Jun. 1996.
7. M. W. Mak and S. Y. Kung, "Estimation of elliptical basis function parameters by the EM algorithm with application to speaker verification," IEEE Trans. Neural Netw., vol. 11, no. 4, pp. 961-969, Jul. 2000.
8. M. Lazaro, I. Santamaria, and C. Pantaleon, "A new EM-based training algorithm for RBF networks," Neural Netw., vol. 16, no. 1, pp. 69-77, Jan. 2003.
9. A. G. Bors and I. Pitas, "Object classification in 3-D images using alpha-trimmed mean radial basis function network," IEEE Trans. Image Process., vol. 8, no. 12, pp. 1730-1744, Dec. 1999.
10. G. Box and G. Tiao, Bayesian Inference in Statistical Models. Reading, MA: Addison-Wesley, 1992.
11. A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin, Bayesian Data Analysis. London, U.K.: Chapman & Hall, 1995.
12. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via EM algorithm," J. R. Statist. Soc., B, vol. 39, no. 1, pp. 1-38, 1977.
13. G. J. McLachlan and T. Krishnan, The EM Algorithm and Extensions (Wiley Series in Probability and Statistics). New York: Wiley, 1997.
14. R. S. Blum and J. Yang, "Image fusion using the expectation-maximization algorithm and a Gaussian mixture model," in Multisensor Surveillance Systems: The Fusion Perspective, G. L. Foresti, C. S. Regazzoni, and P. Varshney, Eds. Norwell, MA: Kluwer, 2003, pp. 81-96.
15. M. A. T. Figueiredo and A. K. Jain, "Unsupervised learning of finite mixture models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 3, pp. 381-396, Mar. 2002.
16. N. Ueda, R. Nakano, Z. Ghahramani, and G. E. Hinton, "SMEM algorithm for mixture models," Neural Comput., vol. 12, no. 9, pp. 2109-2118, Sep. 2000.
17. Z. H. Zhang, C. B. Chen, J. Sun, and K. L. Chan, "EM algorithms for Gaussian mixtures with split-and-merge operation," Pattern Recognit., vol. 36, no. 9, pp. 1973-1983, Sep. 2003.
18. M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul, "An introduction to variational methods for graphical models," in Learning in Graphical Models, M. I. Jordan, Ed. Cambridge, MA: MIT Press, 1999, pp. 105-161.
19. T. S. Jaakkola and M. I. Jordan, "Bayesian parameter estimation via variational methods," Stat. Comput., vol. 10, no. 1, pp. 25-37, Sep. 2000.
20. T. S. Jaakkola and M. I. Jordan, "Improving the mean field approximation via the use of mixture distributions," in Learning in Graphical Models, M. I. Jordan, Ed. Cambridge, MA: MIT Press, 1999, pp. 163-173.
21. S. J. Roberts, D. Husmeier, I. Rezek, and W. D. Penny, "Bayesian approaches to Gaussian mixture modeling," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 11, pp. 1133-1142, Nov. 1998.
22. D. Husmeier, "The Bayesian evidence scheme for regularizing probability-density estimating neural networks," Neural Comput., vol. 12, no. 11, pp. 2685-2717, Nov. 2000.
23. K. Chan, T.-W. Lee, and T. J. Sejnowski, "Variational Bayesian learning of ICA with missing data," Neural Comput., vol. 15, no. 8, pp. 1991-2011, Aug. 2003.
24. D. J. C. MacKay, "Introduction to Monte Carlo methods," in Learning in Graphical Models, M. I. Jordan, Ed. Cambridge, MA: MIT Press, 1999, pp. 175-204.
25. M. J. Beal and Z. Ghahramani, "The variational Bayesian EM algorithm for incomplete data: With application to scoring graphical model structures," in Bayesian Statistics 7, D. Heckerman, A. F. M. Smith, and M. West, Eds. London, U.K.: Oxford Univ. Press, 2003, pp. 453-464.
26. M. N. Gibbs and D. J. C. MacKay, "Variational Gaussian process classifiers," IEEE Trans. Neural Netw., vol. 11, no. 6, pp. 1458-1464, Nov. 2000.
27. H. Attias, "A variational Bayesian framework for graphical models," in Advances in Neural Information Processing Systems (NIPS), vol. 12. Cambridge, MA: MIT Press, 2000, pp. 209-215.
28. T. Minka and J. Lafferty, "Expectation-propagation for the generative aspect model," in Proc. 18th Conf. Uncertainty Arti. Intell., Edmonton, AB, Canada, Aug. 1-4, 2002, pp. 352-359.
29. Z. Ghahramani and M. J. Beal, "Propagation algorithms for variational Bayesian learning," in Advances in Neural Information Processing Systems (NIPS), vol. 13. Cambridge, MA: MIT Press, 2001, pp. 294-300.
30. S. Richardson and P. J. Green, "On Bayesian analysis of mixtures with an unknown number of components," J. Roy. Stat. Soc., Ser. B, vol. 59, no. 4, pp. 731-792, 1997.
31. C. Andrieu, N. de Freitas, and A. Doucet, "Robust full Bayesian learning for radial basis networks," Neural Comput., vol. 13, no. 10, pp. 2359-2407, Oct. 2001.
32. Z. Ghahramani and G. E. Hinton, "Variational learning for switching state-space methods," Neural Comput., vol. 12, no. 4, pp. 831-864, Apr. 2000.
33. S. J. Roberts and W. D. Penny, "Variational Bayes for generalized autoregressive models," IEEE Trans. Signal Process., vol. 50, no. 9, pp. 2245-2257, Sep. 2002.
34. N. de Freitas, P. Højen-Sørensen, M. Jordan, and S. Russell, "Variational MCMC," in Proc. 17th Conf. Uncertainty Artif. Intell., Seattle, WA, 2001, pp. 120-127.
35. Z. Ghahramani and M. J. Beal, "Variational inference for Bayesian mixtures of factor analysers," in Advances in Neural Information Processing Systems (NIPS), vol. 12. Cambridge, MA: MIT Press, 2000, pp. 449-455.
36. R. A. Choudrey and S. J. Roberts, "Variational mixture of Bayesian independent component analyzers," Neural Comput., vol. 15, no. 1, pp. 213-252, Jan. 2003.
37. H. Attias, "Inferring parameters and structure of latent variable models by variational Bayes," in Proc. 15th Conf. Uncertainty Artif. Intell., Stockholm, Sweden, 1999, pp. 21-30.
38. N. Nasios and A. G. Bors, "Blind source separation using variational expectation-maximization algorithm," in Proc. Int. Conf. Comput. Anal. Images Patterns, Lecture Notes in Computer Science 2756, Groningen, The Netherlands, 2003, pp. 442-450.
39. N. Nasios and A. G. Bors, "Variational segmentation of color images," in Proc. IEEE Int. Conf. Image Process., Genova, Italy, 2005, vol. II, pp. 614-617.
40. G. Schwarz, "Estimating the dimension of a model," Ann. Stat., vol. 7, no. 2, pp. 461-464, 1978.
41. J. Rissanen, Stochastic Complexity in Statistical Inquiry. Singapore: World Scientific, 1989.
42. G. Ronning, "Maximum-likelihood estimation of Dirichlet distributions," J. Statist. Comput. Simul., vol. 32, no. 4, pp. 215-221, 1989.
43. M. C. Jeruchim, P. Balaban, and K. S. Shanmungan, Simulation of Communication Systems. New York: Plenum, 1992.
44. A. G. Bors and M. Gabbouj, "Quadrature modulated signal detection based on Gaussian neural networks," in Proc. IEEE Workshop Vis. Signal Process. Commun., Melbourne, Australia, 1993, pp. 113-116.
45. J. B. Gao, J. Zhang, and M. G. Fleming, "Novel technique for multiresolution color image segmentation," Opt. Eng., vol. 41, no. 3, pp. 608-614, Mar. 2002.
46. D.A. Forsyth and J. Ponce, Computer Vision a Modern Approach. Upper Saddle River, NJ: Prentice-Hall, 2003.