Unsupervised learning of categorical data with competing models

IEEE Transactions on Neural Networks and Learning Systems

Volumn 23, Issue 11, 2012, Pages 1726-1737

  Ilin, Roman ^a  

^a AIR FORCE RESEARCH LABORATORY   (United States)

Author keywords

Bernoulli mixture; dynamic logic; maximum a posteriori (MAP); model selection; modeling fields theory; regularization; text categorization; vague to crisp process

Indexed keywords

BERNOULLI MIXTURES; DYNAMIC LOGIC; MAXIMUM A POSTERIORI; MODEL SELECTION; REGULARIZATION; TEXT CATEGORIZATION; VAGUE TO CRISPS;

TEXT PROCESSING; UNSUPERVISED LEARNING;

CLUSTERING ALGORITHMS;

EID: 84876106588     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2012.2213266     Document Type: Article

References (48)

1. L. Perlovsky, Neural Networks and Intellect, Using Model-Based Concepts. New York: Oxford Univ. Press, 2001.
2. L. Perlovsky, "'Vague-to-crisp' neural mechanism of perception," IEEE Trans. Neural Netw., vol. 20, no. 8, pp. 1363-1367, Aug. 2009.
3. S. Streltsov, S. Petrov, I. Muchnick, and L. Perlovsky, "Computational analysis of dynamic logic GMTI tracker," in Proc. Integr. Knowl. Intens. Multiagent Syst., Waltham, MA, 2007, pp. 190-194.
4. M. Bar, K. Kassam, A. Ghuman, J. Boshyan, A. Schmid, A. Dale, M. Hämäläinen, L. Marinkovic, D. Schacter, B. Rosen, and E. Halgren, "Top-down facilitation of visual recognition," Proc. Nat. Acad. Sci., vol. 103, no. 2, pp. 449-454, 2006.
5. L. Perlovsky, "Modeling field theory of higher cognitive functions," in Artificial Cognition Systems, A. Loula, R. Gudwin, and J. Queiroz, Eds. Hershey, PA: Idea Group, 2006, pp. 64-105.
6. A. K. Jain, "Data clustering: 50 years beyond K-means," Pattern Recognit. Lett., vol. 31, no. 8, pp. 651-666, 2010.
7. A. Topchy, A. K. Jain, and W. Punch, "Clustering ensembles: Models of consensus and weak partitions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 12, pp. 1866-1881, Dec. 2005.
8. S. Decherchi, S. Ridella, R. Zunino, P. Gastaldo, and D. Anguita, "Using unsupervised analysis to constrain generalization bounds for support vector classifiers," IEEE Trans. Neural Netw., vol. 21, no. 3, pp. 424-438, Mar. 2010.
9. F. Wang, B. Zhao, and C. Zhang, "Unsupervised large margin discriminative projection," IEEE Trans. Neural Netw., vol. 22, no. 9, pp. 1446-1456, Sep. 2011.
10. S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang, "Domain adaptation via transfer component analysis," IEEE Trans. Neural Netw., vol. 22, no. 2, pp. 199-210, Feb. 2011.
11. R. Xu and D. Wunsch, Clustering. New York: Wiley, 2008.
12. K. P. Burnham and D. R. Anderson, Model Selection and Multimodel Inference: A Practical Infromation-Theoretic Approach. New York: Springer-Verlag, 2002.
13. G. J. McLachlan and D. Peel, Finite Mixture Models. New York: Wiley, 2000.
14. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. Royal Stat. Soc. B, vol. 39, no. 1, pp. 1-38, 1977.
15. G. McLachlan and G. J. Krishnan, The EM Algorithm and Extensions. New York: Wiley, 1997.
16. S. Geman and D. Geman, "Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images," IEEE Trans. Pattern Anal. Mach. Intell., vol. 6, no. 6, pp. 721-741, Nov. 1984.
17. J. Diebolt and C. P. Robert, "Estimation of finite mixture distributions through Bayesian sampling," J. Royal Stat. Soc. B, vol. 56, no. 2, pp. 363-375, 1994.
18. H. Akaike, "Information theory and the maximum likelihood principle," in Proc. 2nd Int. Symp. Inf. Theory, 1973, pp. 599-609.
19. G. E. Schwarz, "Estimating the dimension of a model," Ann. Stat., vol. 6, no. 2, pp. 461-464, 1978.
20. C. Biernacki, G. Celeux, and G. Govaert, "Assessing a mixture model for clustering with the integrated completed likelihood," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 7, pp. 719-725, Jul. 2000.
21. G. Bouchard and G. Celeux, "Selection of generative models in classification," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 4, pp. 544-554, Apr. 2006.
22. R. E. Kass and A. E. Raftery, "Bayes factor," J. Amer. Stat. Assoc., vol. 90, no. 430, pp. 773-795, 1995.
23. S. Chib and I. Jeliazkov, "Marginal likelihood from the Metropolis-Hastings output," J. Amer. Stat. Assoc., vol. 96, no. 453, pp. 271-281, 2001.
24. P. J. Green, "Reversible jump Markov chain Monte Carlo computation and Bayesian model determination," Biometrika, vol. 82, no. 4, pp. 711-732, 1995.
25. S. Richardson and P. J. Green, "On Bayesian analysis of mixtures with an unknown number of components," J. Royal Stat. Soc. B, vol. 59, no. 4, pp. 731-792, 1997.
26. A. N. Tychonoff, "Solution of incorrectly formulated problems and the regularization method," Doklady Akademii Nauk SSSR, vol. 151, pp. 501-504, Jan. 1963.
27. T. Poggio and F. Girosi, "Networks for approximation and learning," Proc. IEEE, vol. 78, no. 9, pp. 1481-1497, Sep. 1990.
28. M. A. T. Figueiredo and A. K. Jain, "Unsupervised learning of finite mixture models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 3, pp. 381-396, Mar. 2002.
29. Z. Lu, Y. Peng, and J. Xiao, "Unsupervised learning of finite mixtures using entropy regularization and its application to image segmentation," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., Anchorage, AK, Jun. 2008, pp. 1-8.
30. J. Ma, J. Liu, and Z. Ren, "Parameter estimation of Poisson mixture with automated model selection through BYY harmony learning," Pattern Recognit., vol. 42, no. 11, pp. 2659-2670, 2009.
31. R. Tibshirani, G. Walther, and T. Hastie, "Estimating the number of clusters in a data set via the gap statistic," J. Royal Stat. Soc. B, vol. 63, no. 2, pp. 411-423, 2001.
32. U. V. Luxburg, "A tutorial on spectral clustering," Stat. Comput., vol. 17, no. 4, pp. 395-416, 2007.
33. L. I. Perlovsky, "Conundrum of combinatorial complexity," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 6, pp. 666-670, Jun. 1998.
34. L. Perlovsky and R. Deming, "Neural networks for improved tracking," IEEE Trans. Neural Netw., vol. 18, no. 6, pp. 1854-1857, Nov. 2007.
35. R. Ilin and R. Deming, "Simultaneous detection and tracking of multiple objects in noisy and cluttered environment using maximum likelihood estimation framework," in Proc. OCEANS, Sydney, Australia, May 2010, pp. 1-7.
36. R. Ilin and L. Perlovsky, "Dynamic Logic learning in cognitive-based situation models," in Proc. IEEE 1st Int. Multidiscip. Conf. Cognit. Methods Situat. Aware. Decision Support, Miami Beach, FL, Feb. 2011, pp. 32-39.
37. R. Ilin and L. Perlovsky, "Cognitively inspired neural network for recognition of situations," Int. J. Natural Comput. Res., vol. 1, no. 1, pp. 1-20, Jan. 2010.
38. C. Bishop, "Regularization in neural networks," in Patern Recognition and Machine Learning. New York: Springer-Verlag, 2006, ch. 5.5.
39. R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis. New York: Wiley, 1973.
40. A. Ng, M. Jordan, and Y. Weiss, "On spectral clustering: Analysis and an algorithm," in Advances in Neural Information Processing Systems, T. Dietterich, S. Becker, and Z. Ghahramani, Eds. Cambridge, MA: MIT Press, 2002, pp. 849-856.
41. C. A. Sugar and G. M. James, "Finding the number of clusters in a dataset: An information-theoretical approach," J. Amer. Stat. Assoc., vol. 98, no. 463, pp. 750-763, Sep. 2003.
42. B. Zhang and S. N. Srihari, "Properties of binary vector dissimilarity measures," in Proc. JCIS Int. Conf. Comput. Vis., Pattern Recognit., Image Process., 2003, pp. 1-4.
43. W. M. Rand, "Objective criteria for the evaluation of clustering methods," J. Amer. Stat. Assoc., vol. 66, no. 336, pp. 846-850, 1971.
44. UCI Machine Learning Repository, Congressional Voting Data Set. (2011, Mar.) [Online]. Available: http://archive.ics.uci.edu/ml/datasets/ Congressional+Voting+Records
45. C. C. Aggarwal, "A survey of text clustering algorithms," in Mining Text Data, C. C. Aggarwal and C. Zhai, Eds. New York: Springer-Verlag, 2012, ch. 4.
46. A. Juan and E. Vidal, "On the use of Bernoulli mixture models for text classification," Pattern Recognit., vol. 35, no. 12, pp. 2705-2710, 2001.
47. 20 Newsgroups Data Set. (2011, Mar.) [Online]. Available: http://people.csail.mit.edu/jrennie/20Newsgroups/
48. Princeton Univ. (2010). About WordNet, Princeton, NJ [Online]. Available: http://wordnet.princeton.edu/wordnet