Theoretical and empirical study on the potential inadequacy of mutual information for feature selection in classification

Neurocomputing

Volumn 112, Issue , 2013, Pages 64-78

  Frénay, Benoît ^a   Doquire, Gauthier ^a   Verleysen, Michel ^a  

^a UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

Author keywords

Classification; Feature selection; Hellman Raviv and Fano bounds; Mutual information; Probability of misclassification

Indexed keywords

CLASSIFICATION ACCURACY; EMPIRICAL STUDIES; FEA-TURE SELECTIONS; HELLMAN-RAVIV AND FANO BOUNDS; MISCLASSIFICATION PROBABILITY; MUTUAL INFORMATIONS; PERFORMANCE CRITERION; PROBABILITY OF MISCLASSIFICATION;

CLASSIFICATION (OF INFORMATION); FEATURE EXTRACTION; NEURAL NETWORKS;

COMPUTER APPLICATIONS;

ACCURACY; ARTICLE; CLASSIFIER; COMPARATIVE STUDY; CRITERION VARIABLE; DATA ANALYSIS; DATA MINING; FEATURE SELECTION PROCESS; MATHEMATICAL ANALYSIS; MATHEMATICAL AND STATISTICAL PROCEDURES; MATHEMATICAL MODEL; MATHEMATICAL PARAMETERS; MUTUAL INFORMATION; PRIORITY JOURNAL; PROBABILITY; PROCESS MODEL; PROCESS OPTIMIZATION; STATISTICAL DISTRIBUTION;

EID: 84877634882     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2012.12.051     Document Type: Article

References (18)

1. Bellman R.E. Adaptive Control Processes-A Guided Tour 1961, Princeton University Press, Princeton, New Jersey, USA.
2. Verleysen M. Learning high-dimensional data. Limitations Future Trends Neural Comput. 2003, 186:141-162.
3. Guyon I., Elisseeff A. An introduction to variable and feature selection. J. Mach. Learn. Res. 2003, 3:1157-1182.
4. Battiti R. Using mutual information for selecting features in supervised neural net learning. IEEE Trans. Neural Networks 1994, 5:537-550.
5. Cover T.M., Thomas J.A. Elements of Information Theory 1991, Wiley-Interscience. 99th edition.
6. Peng H., Long F., Ding C. Feature selection based on mutual information. criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans. Pattern Anal. Mach. Intell. 2005, 27:1226-1238.
7. Fleuret F. Fast binary feature selection with conditional mutual information. J. Mach. Learn. Res. 2004, 5:1531-1555.
8. Rossi F., Lendasse A., Francoi D., Wertz V., Verleysen M. Mutual information for the selection of relevant variables in spectrometric nonlinear modelling. Chemom. Intell. Lab. Syst. 2006, 80:215-226.
9. B. Frénay, G. Doquire, M. Verleysen, On the potential inadequacy of mutual information for feature selection, in: Proceedings of ESANN 2012, 2012.
10. Shannon C.E. A mathematical theory of communication. Bell Syst. Tech. J. 1948, 27:379-423. 623-656.
11. Fano R. Transmission of Information. A Statistical Theory of Communications 1961, The MIT Press, Cambridge, MA.
12. J.W. Fisher, M. Siracusa, T. Kihn, Estimation of signal information content for classification, in: Proceedings of DSP/SPE 2009, 2009.
13. Hellman M.E., Raviv J. Probability of error, equivocation and the Chernoff bound. IEEE Trans. Inf. Theory 1970, 16:368-372.
14. G. Brown, An information theoretic perspective on multiple classifier systems, in: Proceedings of MCS 2009, Springer-Verlag, Berlin, Heidelberg, 2009, pp. 344-353.
15. Ozertem U., Erdogmus D., Jenssen R. Spectral feature projections that maximize Shannon mutual information with class labels. Pattern Recognition 2006, 39:1241-1252.
16. Francois D., Rossi F., Wertz V., Verleysen M. Resampling methods for parameter-free and robust feature selection with mutual information. Neurocomputing 2007, 70(7-9):1276-1288.
17. D.N.A. Asuncion, UCI machine learning repository, 2007 URL 〈〉. http://www.ics.uci.edu/~mlearn/MLRepository.html.
18. Kozachenko L.F., Leonenko N. Sample estimate of the entropy of a random vector. Probl. Inf. Transm. 1987, 23:95-101.