Is mutual information adequate for feature selection in regression?

Neural Networks

Volumn 48, Issue , 2013, Pages 1-7

  Frénay, Benoît ^a   Doquire, Gauthier ^a   Verleysen, Michel ^a  

^a UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

Author keywords

Feature selection; MAE; MSE; Mutual information; Regression

Indexed keywords

HIGH-DIMENSIONAL REGRESSION PROBLEMS; MACHINE LEARNING LITERATURE; MAE; MEAN ABSOLUTE ERROR; MSE; MUTUAL INFORMATIONS; REGRESSION; THEORETICAL DEVELOPMENT;

FEATURE EXTRACTION; LEARNING ALGORITHMS;

REGRESSION ANALYSIS;

ANALYTICAL ERROR; ANALYTICAL PARAMETERS; ARTICLE; ARTIFICIAL NEURAL NETWORK; ENTROPY; FEATURE SELECTION; MATHEMATICAL ANALYSIS; MEAN ABSOLUTE ERROR; MEAN SQUARED ERROR; MUTUAL INFORMATION; PRIORITY JOURNAL; REGRESSION ANALYSIS; STATISTICAL PARAMETERS; THEORETICAL STUDY;

FEATURE SELECTION; MAE; MSE; MUTUAL INFORMATION; REGRESSION;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; ENTROPY; IMAGE PROCESSING, COMPUTER-ASSISTED; INFORMATICS; INFORMATION STORAGE AND RETRIEVAL; NEURAL NETWORKS (COMPUTER); NORMAL DISTRIBUTION; REGRESSION ANALYSIS; SIGNAL-TO-NOISE RATIO;

EID: 84880930604     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2013.07.003     Document Type: Article

References (26)

1. Archambeau C., Delannay N., Verleysen M. Mixtures of robust probabilistic principal component analyzers. Neurocomputing 2008, 71(7-9):1274-1282.
2. Ash R. Information theory 1990, Dover Publications.
3. Battiti R. Using mutual information for selecting features in supervised neural net learning. IEEE Transactions on Neural Networks 1994, 5:537-550.
4. Bellman R.E. Adaptive control processes-a guided tour 1961, Princeton University Press.
5. Chen B., Hu J., Li H., Sun Z. Adaptive filtering under maximum mutual information criterion. Neurocomputing 2008, 71(16-18):3680-3684.
6. Cover T.M., Thomas J.A. Elements of information theory 1991, Wiley-Interscience. 1st ed.
7. Dijck G.V., Hulle M.M.V. Speeding up the wrapper feature subset selection in regression by mutual information relevance and redundancy analysis. Proceedings of the 16th international conference on artificial neural networks 2006, 31-40. Springer.
8. Emmert-Streib F., Dehmer M. Information theory and statistical learning 2008, Springer.
9. Fleuret F. Fast binary feature selection with conditional mutual information. Journal of Machine Learning Research 2004, 5:1531-1555.
10. Frénay, B., Doquire, G., & Verleysen, M. (2012). On the potential inadequacy of mutual information for feature selection. In Proceedings of ESANN.
11. Frénay B., Doquire G., Verleysen M. Theoretical and empirical study on the potential inadequacy of mutual information for feature selection in classification. Neurocomputing 2013, 112:64-78.
12. Globerson A., Tishby N. Sufficient dimensionality reduction. Journal of Machine Learning Research 2003, 3:1307-1331.
13. Guo D., Shamai S., Verdú S. Mutual information and minimum mean-square error in Gaussian channels. IEEE Transactions on Information Theory 2005, 51(4):1261-1282.
14. Guyon I., Elisseeff A. An introduction to variable and feature selection. Journal of Machine Learning Research 2003, 3:1157-1182.
15. Ihara S. Information theory for continuous system 1993, World Scientific.
16. Kohavi R., John G.H. Wrappers for feature subset selection. Artificial Intelligence 1997, 97:273-324.
17. Kojadinovic, I., & Wottka, T. (2000). Comparison between a filter and a wrapper approach to variable subset selection in regression problems. In Proceedings of ESIT.
18. Kotz S., Kozubowski T., Podgórski K. The Laplace distribution and generalizations: a revisit with applications to communications, economics, engineering, and finance 2001, Birkhäuser.
19. Kozachenko L.F., Leonenko N. Sample estimate of the entropy of a random vector. Problems of Information Transmission 1987, 23:95-101.
20. Kraskov A., Stögbauer H., Grassberger P. Estimating mutual information. Physical Review E 2004, 69:066138.
21. Peel D., McLachlan G.J. Robust mixture modelling using the t distribution. Statistics and Computing 2000, 10(4):339-348.
22. Psarakis S., Panaretos J. The folded t distribution. Communications in Statistics: A Theory and Methods 1990, 19(7):2717-2734.
23. Rossi F., François D., Wertz V., Meurens M., Verleysen M. Fast selection of spectral variables with b-spline compression. Chemometrics and Intelligent Laboratory Systems 2007, 86(2):208-218. 4.
24. Rossi F., Lendasse A., Francois D., Wertz V., Verleysen M. Mutual information for the selection of relevant variables in spectrometric nonlinear modelling. Chemometrics and Intelligent Laboratory Systems 2006, 80:215-226.
25. Shannon C.E. A mathematical theory of communication. The Bell System Technical Journal 1948, 27:379-423.
26. Verleysen M. Learning high-dimensional data. Limitations and future trends in neural computation, Vol. 186 2003, 141-162.