Learning Gaussian mixture models with entropy-based criteria

IEEE Transactions on Neural Networks

Volumn 20, Issue 11, 2009, Pages 1756-1771

  Benavent, Antonio Peñalver ^a   Ruiz, Francisco Escolano ^b   Sáez, Juan Manuel ^b  

^a MIGUEL HERNÁNDEZ UNIVERSITY   (Spain)

^b UNIVERSITY OF ALICANTE   (Spain)

Author keywords

Clustering; EM algorithm; Entropy estimation; Minimum description length (MDL) criterion; Mixture models; Model order selection

Indexed keywords

ALGORITHMS; COLOR IMAGE PROCESSING; ENTROPY; IMAGE SEGMENTATION; MAXIMUM LIKELIHOOD; MAXIMUM LIKELIHOOD ESTIMATION; MAXIMUM PRINCIPLE; PARAMETER ESTIMATION; PROBABILITY DENSITY FUNCTION;

CLUSTERING; EM ALGORITHMS; ENTROPY ESTIMATION; MINIMUM DESCRIPTION LENGTH; MIXTURE MODEL; MODEL-ORDER SELECTION;

CLUSTERING ALGORITHMS;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; ENTROPY; NORMAL DISTRIBUTION; STATISTICAL ANALYSIS; STATISTICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; DATA INTERPRETATION, STATISTICAL; ENTROPY; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER); NORMAL DISTRIBUTION;

EID: 75549089343     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2009.2030190     Document Type: Article

References (59)

1. A. Jain, R. Dubes, and J. Mao, "Statistical pattern recognition: A review, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 1, pp. 4-38, Jan. 2000.
2. D. Titterington, A. Smith, and U. Makov, Statistical Analysis of Finite Mixture Distributions. Chichester, U.K.: Wiley, 2002.
3. A. Jain and R. Dubes, Algorithms for Clustering Data. Englewood Cliffs, NJ: Prentice-Hall, 1988.
4. T. Hastie and R. Tibshirani, "Discriminant analysis by Gaussian mixtures, " J. R. Statist. Soc. B, vol. 58, no. 1, pp. 155-176, 1996.
5. P. D. G. Hinton and M. Revow, "Modeling the manifolds of images of handwriting digits, " IEEE Trans. Neural Netw., vol. 8, no. 1, pp. 65-74, Jan. 1997.
6. S. Dalal and W. Hall, "Approximating priors by mixtures of natural conjugate priors, " J. R. Statist. Soc. B, vol. 45, no. 1, 1983.
7. G. Box and G. Tiao, Bayesian Inference in Statistical Models. Reading, MA: Addison-Wesley, 1992.
8. A. Dempster, N. Laird, and D. Rubin, "Maximum likelihood estimation from incomplete data via the EM algorithm, " J. R. Statist. Soc., vol. 39, no. 1, pp. 1-38, 1977.
9. M. Figueiredo and A. Jain, "Unsupervised learning of finite mixture models, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 3, pp. 381-399, Mar. 2002.
10. D. Husmeier, "The Bayesian evidence scheme for regularizing probability-density estimating neural networks, " Neural Comput., vol. 12, no. 11, pp. 2685-2717, 2000.
11. D. MacKay, "Introduction to Monte Carlo methods, " in Learning in Graphical Models, M. Jordan, Ed. Cambridge, MA: MIT Press, 1999.
12. Z. Ghahramani and M. Beal, "Variational inference for Bayesian mixture of factor analysers, " in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 1999, vol. 12.
13. N. Nasios and A. Bors, "Variational learning for Gaussian mixtures, " IEEE Trans. Syst. Man Cybern. B, Cybern., vol. 36, no. 4, pp. 849-862, Aug. 2006.
14. N. Nasios and A. Bors, "Blind source separation using variational expectation-maximization algorithm, " in Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2003, vol. 2756, pp. 442-450.
15. G. McLachlan and D. Peel, Finite Mixture Models. New York:Wiley, 2000.
16. R. Redner and H. Walker, "Mixture densities, maximum likelihood, and the EM algorithm, " SIAM Rev., vol. 26, no. 2, pp. 195-239, 1984.
17. N. Vlassis and A. Likas, "A kurtosis-based dynamic approach to Gaussian mixture modeling, " IEEE Trans. Syst. Man Cybern. A, Syst. Humans, vol. 29, no. 4, pp. 393-399, Jul. 1999.
18. N. Vlassis, A. Likas, and B. Krose, "A multivariate kurtosis-based dynamic approach to Gaussian mixture modeling, intelligent autonomous systems, " Comput. Sci. Inst., Faculty Sci., Univ. Amsterdam, Amsterdam, The Netherlands, 2000.
19. N. Vlassis and A. Likas, "A greedyEMalgorithm for Gaussian mixture learning, " Neural Process. Lett., vol. 15, no. 1, pp. 77-87, 2000.
20. J. Verbek, N. Vlassis, and B. Krose, "Efficient greedy learning of Gaussian mixture models, " Neural Comput., vol. 15, no. 2, pp. 469-485, 2003.
21. M. Figueiredo, J. Leitao, and A. Jain, "On fitting mixture models, " in Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 1999, vol. 1654, pp. 54-69.
22. M. A. T. Figueiredo and A. K. Jain, "Unsupervised selection and estimation of finite mixture models, " in Proc. IEEE Int. Conf. Pattern Recognit., 2000, pp. 87-90.
23. J. Rissanen, Stochastic Complexity in Statistical Inquiry. Singapore: World Scientific, 1989.
24. C. Wallace and P. Freeman, "Estimation and inference via compact coding, " J. R. Statist. Soc. B, vol. 49, no. 3, pp. 241-252, 1987.
25. N. Ueda, R. Nakano, Z. Ghahramani, and G. E. Hinton, "SMEM algorithm for mixture models, " Neural Comput., vol. 12, no. 1, pp. 2109-2128, 2000.
26. L. Xu, "Bayesian Ying-Yang machine, clustering and number of clusters, " Pattern Recognit. Lett., vol. 18, no. 1, pp. 1167-1178, 1997.
27. A. M. Martinez and J. Vitria, "Learning mixture models using a genetic version of theEMalgorithm, " Pattern Recognit. Lett., vol. 21, no. 1, pp. 759-769, 2000.
28. F. Pernkopf and D. Bouchaffra, "Genetic-based EM algorithm for learning Gaussian mixture models, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 8, pp. 1344-1348, Aug. 2006.
29. C. Constantinopoulos and A. Likas, "Unsupervised learning of Gaussian mixtures based on variational component splitting, " IEEE Trans. Neural Netw., vol. 18, no. 3, pp. 745-755, May 2007.
30. A. Penalver, F. Escolano, and J. Sáez, "EBEM: An entropy-based EM algorithm for Gaussian mixture models, " in Proc. 18th Int. Conf. Pattern Recognit., 2006, pp. 451-455.
31. T. Cover and J. Thomas, Elements of Information Theory. New York: Wiley, 1991.
32. M. Jones and R. Sibson, "What is projection pursuit, " J. R. Statist. Soc. A, vol. 150, pp. 1-36, 1987.
33. E. Beirlant, E. Dudewicz, L. Gyorfi, and E. V. der Meulen, "Nonparametric entropy estimation, " Int. J. Math. Statist. Sci., vol. 6, no. 1, pp. 17-39, 1996.
34. I. Paninski, "Estimation of entropy and mutual information, " Neural Comput., vol. 15, no. 1, pp. 1191-125, 2003.
35. P. Viola and W. M. Wells, III, "Alignment by maximization of mutual information, " in Proc. 5th Int. Conf. Comput. Vis., 1995, pp. 469-485.
36. P. Viola, N. N. Schraudolph, and T. J. Sejnowski, "Empirical entropy manipulation for real-world problems, " in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 1996, vol. 8.
37. A. Hyvarinen and E. Oja, "Independent component analysis: Algorithms and applications, " Neural Netw., vol. 13, no. 4-5, pp. 411-430, 2000.
38. A. Hero and O. Michel, "Applications of spanning entropic graphs, " IEEE Signal Process. Mag., vol. 19, no. 5, pp. 85-95, Sep. 2002.
39. E. Parzen, "On estimation of a probability density function and mode, " Ann. Math. Statist., vol. 33, no. 1, pp. 1065-1076, 1962.
40. R. Hoffman and A. Jain, "A test of randomness based on the minimal spanning tree, " Pattern Recognit. Lett., vol. 1, no. 1, pp. 175-180, 1983.
41. A. Hero and O. Michel, "Asymptotic theory of greedy aproximations to minimal k-point random graphs, " IEEE Trans. Inf. Theory, vol. 45, no. 6, pp. 1921-1939, Sep. 1999.
42. D. Bertsimas and G. V. Ryzin, "An asymptotic determination of the minimum spanning tree and minimum matching constants in geometrical probability, " Oper. Res. Lett., vol. 9, no. 1, pp. 223-231, 1990.
43. K. Zyczkowski, "Renyi extrapolation of shannon entropy, " Open Syst. Inf. Dyn., vol. 10, no. 3, pp. 297-310, 2003.
44. A. Mokkadem, "Estimation of the entropy and information of absolutely continuous random variables, " IEEE Trans. Inf. Theory, vol. 35, no. 1, pp. 193-196, Jan. 1989.
45. P. Viola, "Alignment by maximization of mutual information, " Artif. Intell. Lab., Massachusetts Inst. Technol., Cambridge, MA, Tech. Rep. 1548, 1995.
46. A. Hero and O. Michel, "Asymptotic theory of greedy aproximations to minimal k-point random graphs, " Commun. Signal Process. Lab. (CCSPL), Dept. Electr. Eng. Comput. Sci., Univ. Michigan, Ann Arbor, MI, Tech. Rep. 315, 1998.
47. G. Schwarz, "Estimating the dimension of a model, " Ann. Statist., vol. 6, pp. 461-464, 1978.
48. H. Akaike, "Information theory and an extension of the maximum likelihood principle, " in Proc. 2nd Int. Symp. Inf. Theory, Budapest, Hungary, 1973, pp. 267-281.
49. S. Richardson and P. Green, "On Bayesian analysis of mixtures with unknown number of components (with discussion), " J. R. Statist. Soc. B, vol. 59, no. 1, pp. 731-792, 1997.
50. Z. Zhang, K. Chan, Y. Wu, and C. Chen, "Learning a multivariate Gaussian mixture models with the reversible jump MCMC algorithm, " Statist. Comput., vol. 14, no. 1, pp. 343-355, 2004.
51. P. Dellaportas and I. Papageorgiou, "Multivariate mixtures of normals with unknown number of components, " Statist. Comput., vol. 16, no. 1, pp. 57-68, 2006.
52. T. M. Mitchell, Machine Learning. Boston, MA: McGraw-Hill, 1997.
53. S. Zhu, R. Zhang, and Z. Tu, "Integrating bottom-up for object recognition by data driven Markov chain Monte Carlo, " in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2000, pp. 738-745.
54. S. Li, Markov Random Field Modeling in Computer Vision. London, U.K.: Springer-Verlag, 1995.
55. Z. W. Tu and S. C. Zhu, "Image segmentation by data driven Markov chain Monte Carlo, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 1, pp. 657-673, Jan. 2002.
56. Z. Zhang, C. Chen, J. Sun, and K. Chan, "EM algorithms for Gaussian mixtures with split-and-merge operation, " Pattern Recognit., vol. 36, no. 1, pp. 1973-1983, 2003.
57. C. Blake and C. Merz, UCI Repository of Machine Learning Databases, Dept. Inf. Comput. Sci., Univ. California, Irvine, CA, 1998.
58. S. Aeberhard, D. Coomans, and O. de Vel, "The performance of statistical pattern recognition methods in high dimensional settings, " in Proc. IEEE Signal Process. Workshop Higher Order Statist., Ceasarea, Israel, 1994, pp. 14-16.
59. C. Archambeau, F. Vrins, and M. Verleysen, M. Verleysen, Ed., "Flexible and robust Bayesian classification by finite mixture models, " in Proc. 12th Eur. Symp. Artif. Neural Netw., Bruges, Belgium, 2004, pp. 75-80.