Learning Gaussian mixture with automatic model selection: A comparative study on three Bayesian related approaches

Frontiers of Electrical and Electronic Engineering in China

Volumn 6, Issue 2, 2011, Pages 215-244

  Shi, Lei ^a   Tu, Shikui ^a   Xu, Lei ^a  

^a CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

automatic model selection; Bayesian Ying Yang (BYY) harmony learning; conjugate distributions; Dirichlet; empirical comparison; Gaussian mixture model (GMM); Jeffreys prior; joint Normal Wishart (NW); marginalized student's T distribution; minimum message length (MML); variational Bayesian (VB)

Indexed keywords

EID: 79958731437     PISSN: 16733460     EISSN: 16733584     Source Type: Journal    
DOI: 10.1007/s11460-011-0153-z     Document Type: Article

References (43)

1. Constantinopoulos C, Titsias M K. Bayesian feature and model selection for Gaussian mixture models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(6): 1013-1018.
2. Redner R, Walker H. Mixture densities, maximum likelihood and the EM algorithm. SIAM Review, 1984, 26(2): 195-239.
3. Engel A, den Broeck C P L V. Statistical Mechanics of Learning. New York: Cambridge University Press, 2001.
4. Constantinopoulos C, Likas A. Unsupervised learning of Gaussian mixtures based on variational component splitting. IEEE Transactions on Neural Networks, 2007, 18(3): 745-755.
5. Verbeek J J, Vlassis N, Krose B. Efficient greedy learning of Gaussian mixture models. Neural Computation, 2003, 15(2): 469-485.
6. Xu L, Jordan M I. On convergence properties of the EM algorithm for Gaussian mixtures. Neural Computation, 1996, 8(1): 129-151.
7. McLachlan G J, Krishnan T. The EM Algorithm and Extensions (Wiley Series in Probability and Statistics). New York: Wiley-Interscience, 2007.
8. Akaike H. A new look at the statistical model identification. IEEE Transactions on Automatic Control, 1974, 19(6): 716-723.
9. Schwarz G. Estimating the dimension of a model. The Annals of Statistics, 1978, 6(2): 461-464.
10. Barron A R, Rissanen J, Yu B. The minimum description length principle in coding and modeling. IEEE Transactions on Information Theory, 1998, 44(6): 2743-2760.
11. Rissanen J. Modelling by the shortest data description. Automatica, 1978, 14(5): 465-471.
12. Xu L. Bayesian Ying-Yang system, best harmony learning, and five action circling. Frontiers of Electrical and Electronic Enginnering in China, 2010, 5(3): 281-328.
13. Xu L, Krzyzak A, Oja E. Unsupervised and supervised classifications by rival penalized competitive learning. In: Proceedings of the 11th International Conference on Pattern Recognition. 1992, I: 672-675.
14. Xu L, Krzyzak A, Oja E. Rival penelized competitive learning for clustering analysis, RBF net, and curve detection. IEEE Transactions on Neural Networks, 1993, 4(4): 636-649.
15. Xu L. Rival penalized competitive learning, finite mixture, and multisets clustering. In: Proceedings of IEEE Intentional Joint Conference on Neural Networks. 1998, 2: 2525-2530.
16. Xu L. Bayesian-Kullback coupled Ying-Yang machines: unified learnings and new results on vector quantization. In: Proceedings of International Conference on Neural Information Processing. 1995, 977-988.
17. Xu L. Bayesian Ying Yang learning. Scholarpedia, 2007, 2(3): 1809.
18. Figueiredo M A F, Jain A K. Unsupervised learning of finite mixture models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(3): 381-396.
19. Neal R, Hinton G E. A view of the EM algorithm that justifies incremental, sparse, and other variants. Learning in Graphical Models. Norwell: Kluwer Academic Publishers, 1998, 355-368.
20. Corduneanu A, Bishop C M. Variational Bayesian model selection for mixture distributions. In: Proceedings of the 8th International Conference on Artificial Intelligence and Statistics. 2001, 27-34.
21. Jaakkola T, Jordan M. Bayesian parameter estimation via variational methods. Statistics and Computing, 2000, 10(1): 25-37.
22. Wallace C, Boulton D. An information measure for classification. The Computer Journal, 1968, 11(2): 185-194.
23. Wallace C S, Dowe D L. Minimum message length and Kolmogorov complexity. The Computer Journal, 1999, 42(4): 270-283.
24. Attias H. A variational Bayesian framework for graphical models. Advances in Neural Information Processing Systems, 2000, 12: 209-215.
25. Gelman A, Carlin J B, Stern H S, Rubin D B. Bayesian Data Analysis. 2nd ed. Texts in Statistical Science. Boca Raton: Chapman & Hall/CRC, 2003.
26. Xu L. Machine learning problems from optimization perspective. Journal of Global Optimization, 2010, 47(3): 369-401.
27. Unnikrishnan R, Pantofaru C, Hebert M. Toward objective evaluation of image segmentation algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 929-944.
28. Xu L. Bayesian Ying Yang system, best harmony learning and Gaussian manifold based family. In: Zurada J, Yen G, Wang J, eds. Computational Intelligence: Research Frontiers. Berlin-Heidelberg: Springer-Verlag, 2008, 5050: 48-78.
29. Xu L. Multisets modeling learning: a unified theory for supervised and unsupervised learning. In: Proceedings of IEEE Intentional Joint Conference on Neural Networks. 1994, I: 315-320.
30. Xu L. A unified learning framework: multisets modeling learning. In: Proceedings of World Congress on Neural Networks. 1995, 1: 35-42.
31. Xu L. BYY harmony learning, structural RPCL, and topological self-organizing on unsupervised and supervised mixture models. Neural Networks, 2002, 15(8-9): 1125-1151.
32. Xu L. Data smoothing regularization, multi-sets-learning, and problem solving strategies. Neural Networks, 2003, 16(5-6): 817-825.
33. Xu L. A unified perspective and new results on RHT computing, mixture based learning, and multi-learner based problem solving. Pattern Recognition, 2007, 40(8): 2129-2153.
34. Xu L. Learning algorithms for RBF functions and subspace based functions. Handbook of Research on Machine Learning, Applications and Trends: Algorithms, Methods and Techniques. Hershey: IGI Global, 2009, 60-94.
35. Xu L. Best harmony, unified RPCL and automated model selection for unsupervised and supervised learning on Gaussian mixtures, ME-RBF models and three-layer nets. International Journal of Neural Systems, 2001, 11(1): 3-69.
36. Bartlett P L, Boucheron S, Lugosi G. Model selection and error estimation. Machine Learning, 2002, 48(1-3): 85-113.
37. Kearns M, Mansour Y, Ng A Y, Ron D. An experimental and theoretical comparison of model selection methods. Machine Learning, 1997, 27(1): 7-50.
38. Wallace C S, Dowe D L. Refinements of MDL and MML coding. The Computer Journal, 1999, 42(4): 330-337.
39. Kotz S, Nadarajah S. Multivariate t Distributions and Their Applications. Cambridge: Cambridge University Press, 2004.
40. Varma M, Zisserman A. Texture classification: are filter banks necessary? In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2003, 2: 691-698.
41. Nikou C, Likas A, Galatsanos N. A Bayesian framework for image segmentation with spatially varying mixtures. IEEE Transactions on Image Processing, 2010, 19(9): 2278-2289.
42. Shi J, Malik J. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(8): 888-905.
43. Rother C, Kolmogorov V, Blake A. "GrabCut": interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics, 2004, 23(3): 309-314.