A semidefinite programming based search strategy for feature selection with mutual information measure

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 37, Issue 8, 2015, Pages 1529-1540

  Naghibi, Tofigh ^a   Hoffmann, Sarah ^a   Pfister, Beat ^a  

^a ETH ZURICH   (Switzerland)

Author keywords

Approximation ratio; Convex objective; Feature selection; Mutual information

Indexed keywords

ALGORITHMS; APPROXIMATION ALGORITHMS; CLUSTERING ALGORITHMS; COMPUTATIONAL COMPLEXITY; DATA MINING; GRAPH THEORY; OPTIMIZATION; POLYNOMIAL APPROXIMATION; SET THEORY;

APPROXIMATION RATIOS; CONVEX OBJECTIVES; DATA MINING APPLICATIONS; FEATURE SUBSET SELECTION; MUTUAL INFORMATION MEASURES; MUTUAL INFORMATIONS; SEMI-DEFINITE PROGRAMMING; SEQUENTIAL SEARCH ALGORITHM;

FEATURE EXTRACTION;

EID: 84937811946     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2014.2372791     Document Type: Article

References (43)

1. N. Abramson, Information Theory and Coding. New York, NY, USA: McGraw-Hill, 1963.
2. D. W. Aha and R. L. Bankert, "A comparative evaluation of sequential feature selection algorithms", in Learning from Data: Artificial Intelligence and Statistics V. New York, NY, USA: Springer-Verlag, 1996.
3. Y. Asahiro, K. Iwama, H. Tamaki, and T. Tokuyama, "Greedily finding a dense subgraph", J. Algorithms, vol. 34, no. 2, pp. 203-221, 2000.
4. K. S. Balagani and V. V. Phoha, "On the feature selection criterion based on an approximation of multidimensional mutual information", IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 7, pp. 1342-1343, Jul. 2010.
5. R. Battiti, "Using mutual information for selecting features in supervised neural net learning", IEEE Trans. Neural Netw., vol. 5, no. 4, pp. 537-550, Jul. 1994.
6. V. I. Bogachev, Measure Theory. New York, NY, USA: Springer, 2007.
7. S. Boyd and L. Vandenberghe, Convex Optimization. New York, NY, USA: Cambridge Univ. Press, 2004.
8. G. Brown, "A new perspective for information theoretic feature selection", in Proc. 12th Int. Conf. Artif. Intell. Statist., 2009, pp. 49-56.
9. G. Brown, A. Pocock, M. Zhao, and M. Lujan, "Conditional likelihood maximisation: A unifying framework for information theoretic feature selection", J. Mach. Learn., vol. 13, no. 1, pp. 27-66, 2012.
10. P. M. Ciarelli, E. O. T. Salles, and E. Oliveira, "An evolving system based on probabilistic neural network", in Proc. 11th Brazilian Symp. Neural Netw., 2010, pp. 182-187.
11. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York, NY, USA: Wiley-Interscience, 1991.
12. J. Doak, "An evaluation of feature selection methods and their application to computer security", Dept. of Comput. Sci., Univ. of California at Davis, Davis, CA, USA, Technical Report CSE-92-18, 1992.
13. R. Fano, Transmission of Information: A Statistical Theory of Communications. Cambridge, MA, USA: MIT Press, 1961.
14. A. Frank and A. Asuncion, "UCI machine learning repository", 2010.
15. J. H. Friedman and J. W. Tukey, "A projection pursuit algorithm for exploratory data analysis", IEEE Trans. Comput., vol. C-23, no. 9, pp. 881-890, Sep. 1974.
16. A. M. Frieze and J. Yadegar, "On the quadratic assignment problem", Discrete Appl. Math., vol. 5, no. 1, pp. 89-98, 1983.
17. B. Frenay, G. Doquire, and M. Verleysen, "On the potential inadequacy of mutual information for feature selection", in Proc. Eur. Symp. Artif. Neural Netw., 2012, pp. 120-126.
18. M. X. Goemans and D. P. Williamson, "Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming", J. ACM, vol. 42, no. 6, pp. 1115-1145, 1995.
19. L. Grippo, L. Palagi, M. Piacentini, V. Piccialli, and G. Rinaldi, "SpeeDP: An algorithm to compute SDP bounds for very large max-cut instances", Math. Programm., vol. 136, pp. 353-373, 2012.
20. M. Gurban, "Multimodal feature extraction fusion audio-visual speech recognition", PhD thesis, 4292, STI, EPF Lausanne, Lausanne, Switzerland, 2009.
21. T. S. Han, "Multiple mutual informations and multiple interactions in frequency data", Inf. Control, vol. 46, no. 1, pp. 26-45, 1980.
22. M. Hellman and J. Raviv, "Probability of error, equivocation and the Chernoff bound", IEEE Trans. Inf. Theory, vol. 16, no. 4, pp. 368-372, Jul. 1970.
23. M. Hollander and D. A. Wolfe, Nonparametric Statistical Methods, 2nd ed. New York, NY, USA: Wiley-Interscience, 1999.
24. B. G. Hu, "What are the differences between Bayesian classifiers and mutual-information classifiers?" IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 2, pp. 249-264, Feb. 2014.
25. D. Karger and N. Srebro, "Learning Markov networks: Maximum bounded tree-width graphs", in Proc. 12th Annu. Symp. Discrete Algorithms, 2001, pp. 392-401.
26. B. J. Killian, J. Y. Kravitz, and M. K. Gilson, "Extraction of configurational entropy from molecular simulations via an expansion approximation", J. Chem. Phys., vol. 127, no. 2, p. 024107, 2007.
27. R. Kohavi, "Wrappers for performance enhancement and oblivious decision graphs", PhD thesis, Dept. Comput. Sci., Stanford Univ., Stanford, CA, USA, 1996.
28. N. Kwak and C. Choi, "Input feature selection for classification problems", IEEE Trans. Neural Netw., vol. 13, no. 1, pp. 143-159, Jan. 2002.
29. J. Lofberg, "Yalmip: A toolbox for modeling and optimization in matlab", in Proc. Comput. Aided Control Syst. Des., 2004, pp. 284-289.
30. W. McGill, "Multivariate information transmission", IRE Professional Group Inf. Theory, vol. 4, no. 4, pp. 93-111, 1954.
31. P. Meyer and G. Bontempi, On the Use of Variable Complementarity for Feature Selection in Cancer Classification. New York, NY, USA: Springer, 2006.
32. T. Naghibi, S. Hoffmann, and B. Pfister, "Convex approximation of the NP-hard search problem in feature subset selection", in Proc. Int. Conf. Acoust., Speech Signal Process., 2013, pp. 3273-3277.
33. P. M. Narendra and K. Fukunaga, "A branch and bound algorithm for feature subset selection", IEEE Trans. Comput., vol. C-26, no. 9, pp. 917-922, Sep. 1977.
34. J. Neumann, C. Schnörr, and G. Steidl, "SVM-based feature selection by direct objective minimisation", in Proc. 26th DAGM Symp., 2004, pp. 212-219.
35. H. Peng, F. Long, and C. Ding, "Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy", IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 8, pp. 1226-1238, Aug. 2005.
36. S. Poljak, F. Rendl, and H. Wolkowicz, "A recipe for semidefinite relaxation for (0, 1)-quadratic programming", J. Global Optim., vol. 7, no. 1, pp. 51-73, 1995.
37. P. Raghavan, "Probabilistic construction of deterministic algorithms: Approximating packing integer programs", J. Comput. Syst. Sci., vol. 37, no. 2, pp. 130-143, 1988.
38. F. M. Reza, An Introduction to Information Theory. New York, NY, USA: Dover Publications, Inc., 1961.
39. I. Rodriguez-Lujan, R. Huerta, Ch. Elkan, and C. S. Cruz, "Quadratic programming feature selection", J. Mach. Learn. Res., vol. 11, pp. 1491-1516, 2010.
40. A. Srivastav and K. Wolf, "Finding dense subgraphs with semidefinite programming", in Approximation Algorithms for Combinatiorial Optimization. New York, NY, USA: Springer, 1998.
41. H. Vafaie and K. De Jong, "Robust feature selection algorithms", in Proc. 5th Int. Conf. Tools Artif. Intell., 1993, pp. 356-363.
42. R. W. Yeung, "A new outlook on Shannon's information measures", IEEE Trans. Inf. Theory, vol. 37, no. 3, pp. 466-474, May 1991.
43. X. Zhao, D. Sun, and K. Toh, "A Newton-CG augmented Lagrangian method for semidefinite programming", SIAM J. Optim., vol. 20, no. 4, pp. 1737-1765, 2010.