Unsupervised hybrid feature extraction selection for high-dimensional non-Gaussian data clustering with variational inference

IEEE Transactions on Knowledge and Data Engineering

Volumn 25, Issue 7, 2013, Pages 1670-1685

  Fan, Wentao ^a   Bouguila, Nizar ^a   Ziou, Djemel ^b  

^a Concordia University ^*  (Canada)

^b UNIVERSITÉ DE SHERBROOKE   (Canada)

Author keywords

Bayesian estimation; feature selection; generalized Dirichlet; human action videos; Mixture models; model selection; unsupervised learning; variational inference

Indexed keywords

BAYESIAN ESTIMATIONS; GENERALIZED DIRICHLET; HUMAN ACTIONS; MIXTURE MODEL; MODEL SELECTION; VARIATIONAL INFERENCE;

BAYESIAN NETWORKS; CLUSTERING ALGORITHMS; FEATURE EXTRACTION; GAUSSIAN NOISE (ELECTRONIC); MIXTURES; UNSUPERVISED LEARNING;

VARIATIONAL TECHNIQUES;

EID: 84878296503     PISSN: 10414347     EISSN: None     Source Type: Journal    
DOI: 10.1109/TKDE.2012.101     Document Type: Article

References (71)

1. J.H. Friedman and J.W. Tukey, "A Projection Pursuit Algorithm for Exploratory Data Analysis," IEEE Trans. Computers, vol. 23, no. 9, pp. 881-890, Sept. 1974.
2. P.E. Green, F.J. Carmone, and J. Kim, "A Preliminary Study of Optimal Variable Weighting in K-Means Clustering," J. Classification, vol. 7, no. 2, pp. 271-285, 1990.
3. F.Z. Brill, D.E. Brown, and W.N. Martin, "Fast Genetic Selection of Features for Neural Network Classifiers," IEEE Trans. Neural Networks, vol. 3, no. 2, pp. 324-328, Mar. 1992.
4. F.E. Shaudys and T.K. Leen, "Feature Selection for Improved Classification," Proc. IEEE Int'l Joint Conf. Neural Networks (IJCNN), pp. 697-702, 1992.
5. P.V. Verveer and R.P.W. Duin, "An Evaluation of Intrinsic Dimensionality Estimators," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 1, pp. 81-86, Jan. 1995.
6. C.E. Brodley and P.E. Utgoff, "Multivariate Decision Trees," Machine Learning, vol. 19, no. 1, pp. 45-77, 1995.
7. Y. Kim, W.N. Street, and F. Menczer, "Feature Selection in Unsupervised Learning via Evolutionary Search," Proc. Int'l Conf. Knowledge Discovery and Data Mining (KDD), pp. 365-369, 2000.
8. H. Liu and H. Motoda, Computational Methods of Feature Selection. Chapman & Hall, 2008.
9. M.H.C. Law, M.A.T. Figueiredo, and A.K. Jain, "Simultaneous Feature Selection and Clustering Using Mixture Models," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 9, pp. 1154-1166, Sept. 2004.
10. S. Boutemedjet, N. Bouguila, and D. Ziou, "A Hybrid Feature Extraction Selection Approach for High-Dimensional Non-Gaussian Data Clustering," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 8, pp. 1429-1443, Aug. 2009.
11. M.W. Graham and D.J. Miller, "Unsupervised Learning of Parsimonious Mixtures on Large Spaces with Integrated Feature and Component Selection," IEEE Trans. Signal Processing, vol. 54, no. 4, pp. 1289-1303, Apr. 2006.
12. A.E. Raftery and N. Dean, "Variable Selection for Model-Based Clustering," J. Am. Statistical Assoc., vol. 101, no. 473, pp. 168-178, 2006. (Pubitemid 43500033)
13. C. Constantinopoulos, M.K. Titsias, and A. Likas, "Bayesian Feature and Model Selection for Gaussian Mixture Models," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 6, pp. 1013-1018, June 2006.
14. N. Bouguila, "A Model-Based Approach for Discrete Data Clustering and Feature Weighting Using MAP and Stochastic Complexity," IEEE Transactions Knowledge and Data Eng., vol. 21, no. 12, pp. 1649-1664, Dec. 2009.
15. C. Maugis, G. Celeux, and M-L. Martin-Magniette, "Variable Selection for Clustering with Gaussian Mixture Models," Biometrics, vol. 65, no. 3, pp. 701-709, 2009.
16. H. Lian, "Sparse Bayesian Hierarchical Modeling of High-Dimensional Clustering Problems," J. Multivariate Analysis, vol. 101, no. 7, pp. 1728-1737, 2010.
17. Y. Li, M. Dong, and J. Hua, "Simultaneous Localized Feature Selection and Model Detection for Gaussian Mixtures," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 5, pp. 953-960, May 2009.
18. C. Fraley and A.E. Raftery, "Model-Based Clustering, Discriminant Analysis, and Density Estimation," J. Am. Statistical Assoc., vol. 97, no. 458, pp. 611-631, 2002.
19. N. Bouguila and D. Ziou, "A Hybrid SEM Algorithm for High-Dimensional Unsupervised Learning Using a Finite Generalized Dirichlet Mixture," IEEE Trans. Image Processing, vol. 15, no. 9, pp. 2657-2668, Sept. 2006. (Pubitemid 44270206)
20. N. Bouguila and D. Ziou, "High-Dimensional Unsupervised Selection and Estimation of a Finite Generalized Dirichlet Mixture Model Based on Minimum Message Length," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 10, pp. 1716-1731, Oct. 2007.
21. C.S. Wallace, Statistical and Inductive Inference by Minimum Message Length. Springer-Verlag, 2005.
22. G.J. McLachlan and T. Krishnan, The EM Algorithm and Extensions. Wiley-Interscience, 1997.
23. A.L. Yuille, P. Stolorz, and J. Utans, "Statistical Physics, Mixtures of Distributions, and the EM Algorithm," Neural Computation, vol. 6, no. 2, pp. 334-340, 1994.
24. Z. Lu and H.H.S. Ip, "Generalized Competitive Learning of Gaussian Mixture Models," IEEE Trans. Systems, Man, and Cybernetics-Part B: Cybernetics, vol. 39, no. 4, pp. 901-909, Aug. 2009.
25. C.P. Robert and G. Casella, Monte Carlo Statistical Methods. Springer-Verlag, 1999.
26. O. Cappé and C.P. Robert, "Markov Chain Monte Carlo: 10 Years and Still Running!," J. Am. Statistical Assoc., vol. 95, no. 452, pp. 1282-1286, 2000.
27. T.S. Jaakkola and M.I. Jordan, "Bayesian Parameter Estimation via Variational Methods," Statistics and Computing, vol. 10, no. 1, pp. 25-37, 2000.
28. P. Hall, K. Humphreys, and D.M. Titterington, "On the Adequacy of Variational Lower Bound Functions for Likelihood-Based Inference in Markovian Models with Missing Values," J. Royal Statistical Soc. Series B (Statistical Methodology), vol. 64, no. 3, pp. 549-564, 2002.
29. N. de Freitas, P. Højen-Sørensen, M.I. Jordan, and S. Russel, "Variational MCMC," Proc. Conf. Uncertainty in Artificial Intelligence (UAI), pp. 120-127, 2001.
30. C.M. Bishop, D. Spiegelhalter, and J. Winn, "VIBES: A Variational Inference Engine for Bayesian Networks," Proc. Advances in Neural Information Processing Systems (NIPS), pp. 777-784, 2002.
31. T.S. Jaakkola and M.I. Jordan, "Variational Probabilistic Inference and the QMR-DT Network," J. Artificial Intelligence Research, vol. 10, pp. 291-322, 1999. (Pubitemid 30573907)
32. S.J. Roberts and W.D. Penny, "Variational Bayes for Generalized Autoregressive Models," IEEE Trans. Signal Processing, vol. 50, no. 9, pp. 2245-2257, Sept. 2002. (Pubitemid 34974216)
33. N. Nasios and A.G. Bors, "Variational Expectation-Maximization Training for Gaussian Networks," Proc. IEEE Workshop Neural Networks for Signal Processing (NNSP), pp. 339-348, 2003.
34. N. Nasios and A.G. Bors, "Variational Segmentation of Color Images," Proc. IEEE Int'l Conf. Image Processing (ICIP), pp. 614-617, 2005. (Pubitemid 44539014)
35. G.J. McLachlan and D. Peel, Finite Mixture Models. Wiley, 2000.
36. G.W. Milligan, "A Validation Study of a Variable Weighting Algorithm for Cluster Analysis," J. Classification, vol. 6, no. 1, pp. 53-71, 1989.
37. M.J. Brusco and J.D. Cradit, "A Variable-Selection Heuristic for K-Means Clustering," Psychometrika, vol. 66, no. 2, pp. 249-270, 2001.
38. E.B. Fowlkes, R. Gnanadesikan, and J.R. Kettenring, "Variable Selection in Clustering," J. Classification, vol. 5, no. 2, pp. 205-228, 1988.
39. J.Z. Huang, M.K. Ng, H. Rong, and Z. Li, "Automated Variable Weighting in K-Means Type Clustering," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 5, pp. 657-668, May 2005. (Pubitemid 40608400)
40. W.S. DeSarbo, J.D. Carroll, L.A. Clark, and P.E. Green, "Synthesized Clustering: A Method for Amalgamating Alternative Clustering Based with Differential Weighting of Variables," Psychometrika, vol. 49, no. 1, pp. 57-78, 1984.
41. D.A. Bell and H. Wang, "A Formalism for Relevance and Its Application in Feature Subset Selection," Machine Learning, vol. 41, no. 2, pp. 175-195, 2000.
42. N. Bouguila, D. Ziou, and E. Monga, "Practical Bayesian Estimation of a Finite Beta Mixture through Gibbs Sampling and its Applications," Statistics and Computing, vol. 16, no. 2, pp. 215-225, 2006. (Pubitemid 43742298)
43. E. Castillo, A.S. Hadi, C. Solares, "Learning and Updating of Uncertainty in Dirichlet Models," Machine Learning, vol. 26, no. 1, pp. 43-63, 1997. (Pubitemid 127722999)
44. N. Bouguila, D. Ziou, and R.I. Hammoud, "On Bayesian Analysis of a Finite Generalized Dirichlet Mixture via a Metropolis-within-Gibbs Sampling," Pattern Analysis and Applications, vol. 12, no. 2, pp. 151-166, 2009.
45. Z. Ma and A. Leijon, "Bayesian Estimation of Beta Mixture Models with Variational Inference," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 33, no. 11, pp. 2160-2173, Nov. 2011.
46. A. Corduneanu and C.M. Bishop, "Variational Bayesian Model Selection for Mixture Distributions," Proc. Eighth Int'l Conf. Artificial Intelligence and Statistics (AISTAT), pp. 27-34, 2001.
47. G. Parisi, Statistical Field Theory. Addison-Wesley, 1988.
48. L.K. Saul and M.I. Jordan, "Exploiting Tractable Substructures in Intractable Networks," Proc. Advances in Neural Information Processing Systems (NIPS), pp. 486-492, 1995.
49. T.S. Jaakkola and M.I. Jordan, "Computing Upper and Lower Bounds on Likelihoods in Intractable Networks," Proc. Conf. Uncertainty in Artificial Intelligence (UAI), pp. 340-348, 1996.
50. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge Univ. Press, 2004.
51. N. Bouguila and D. Ziou, "Unsupervised Selection of a Finite Dirichlet Mixture Model: An MML-Based Approach," IEEE Trans. Knowledge and Data Eng., vol. 18, no. 8, pp. 993-1009, Aug. 2006. (Pubitemid 44109531)
52. J.H. Friedman and J.J. Meulman, "Clustering Objects on Subsets of Attributes (with Discussions)," J. Royal Statistical Soc., Series B, vol. 66, no. 4, pp. 815-849, 2004.
53. X. Zhou, X. Zhuang, S. Yan, S-F. Chang, M. Hasegawa-Johnson, and T.S. Huang, "SIFT-Based Kernel for Video Event Analysis," Proc. ACM Int'l Conf. Multimedia (MM), pp. 229-238, 2008.
54. D. Zhong, H. Zhang, and S-F. Chang, "Clustering Methods for Video Browsing and Annotation," Proc. SPIE Conf. Storage and Retrieval for Video and Image Databases, pp. 239-246, 1997.
55. P. Dollár, V. Rabaud, G. Cottrell, and S. Belongie, "Behavior Recognition via Sparse Spatio-Temporal Feature," Proc. IEEE Int'l Workshop Visual Surveillance and Performance Evaluation of Tracking and Surveillance (VS-PETS), pp. 65-72, 2005. (Pubitemid 46178025)
56. J.C. Niebles, H. Wang, and L. Fei-Fei, "Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words," Proc. British Machine Vision Conference (BMVC), pp. 1249-1258, 2006.
57. I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, "Learning Realistic Human Actions from Movies," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1-8, 2008.
58. H. Jhuang, T. Serre, L. Wolf, and T. Poggio, "A Biologically Inspired System for Action Recognition," Proc. IEEE Int'l Conf. Computer Vision (ICCV), pp. 1-8, 2007.
59. Y. Ke, R. Sukthankar, and M. Hebert, "Efficient Visual Event Detection Using Volumetric Features," Proc. IEEE Int'l Conf. Computer Vision (ICCV), pp. 166-173, 2005. (Pubitemid 44054955)
60. P. Turaga, R. Chellappa, V.S. Subrahmanian, and O. Udrea, "Machine Recognition of Human Activities: A Survey," IEEE Trans. Circuits and Systems for Video Technology, vol. 18, no. 11, pp. 1473-1488, Nov. 2008.
61. C. Schüldt, I. Laptev, and B. Caputo, "Recognizing Human Actions: A Local SVM Approach," Proc. 17th Int'l Conf. Pattern Recognition (ICPR), pp. 32-36, 2004.
62. S. Becker, "Learning to Categorize Objects Using Temporal Coherence," Proc. Advances in Neural Information Processing Systems (NIPS), pp. 361-368, 1992.
63. A. Frome, Y. Singer, and J. Malik, "Image Retrieval and Classification Using Local Distance Functions," Proc. Advances in Neural Information Processing Systems (NIPS), pp. 417-424, 2006.
64. G. Csurka, C.R. Dance, L. Fan, J. Willamowski, and C. Bray, "Visual Categorization with Bags of Keypoints," Proc. Workshop Statistical Learning in Computer Vision, Eighth European Conf. Computer Vision (ECCV), 2004.
65. I. Laptev, "On Space-Time Interest Points," Int'l J. Computer Vision, vol. 64, no. 2/3, pp. 107-123, 2005. (Pubitemid 41314645)
66. S.F. Wong and R. Cipolla, "Extracting Spatiotemporal Interest Points Using Global Information," Proc. IEEE Int'l Conf. Computer Vision (ICCV), pp. 1-8, 2007.
67. A. Oikonomopoulos, I. Patras, and M. Pantic, "Spatiotemporal Salient Points for Visual Recognition of Human Actions," IEEE Trans. Systems, Man, and Cybernetics-Part B: Cybernetics, vol. 36, no. 3, pp. 710-719, June 2005.
68. G. Willems, T. Tuytelaars, and L. Van Gool, "An Efficient Dense and Scale-invariant Spatio-Temporal Interest Point Detector," Proc. Ninth European Conf. Computer Vision (ECCV), pp. 650-663, 2008.
69. T. Hofmann, "Unsupervised Learning by Probabilistic Latent Semantic Analysis," Machine Learning, vol. 42, no. 1/2, pp. 177-196, 2001. (Pubitemid 32872403)
70. A. Bosch, A. Zisserman, and X. Munoz, "Scene Classification via pLSA," Proc. Ninth European Conf. Computer Vision (ECCV), pp. 517-530, 2006. (Pubitemid 44029655)
71. N. Bouguila and D. Ziou, "A Dirichlet Process Mixture of Generalized Dirichlet Distributions For Proportional Data Modeling," IEEE Trans. Neural Networks, vol. 21, no. 1, pp. 107-122, Jan. 2010.