Increasing and decreasing returns and losses in mutual information feature subset selection

Entropy

Volumn 12, Issue 10, 2010, Pages 2144-2170

  Van Dijck, Gert ^a   Van Hulle, Marc M ^a  

^a KU LEUVEN UNIVERSITY OF LEUVEN   (Belgium)

Author keywords

Bayesian networks; Bit parity; Conditional entropy; Conditional mutual information; Decreasing losses; Decreasing returns; Feature subset selection; Increasing losses; Increasing returns

Indexed keywords

EID: 77958167275     PISSN: None     EISSN: 10994300     Source Type: Journal    
DOI: 10.3390/e12102144     Document Type: Article

References (35)

1. Liu, H.; Motoda, H. Computational Methods of Feature Selection; Chapman & Hall/CRC: Boca Raton, FL, USA, 2007.
2. Van Dijck, G.; Van Vaerenbergh, J.; Van Hulle, M.M. Posterior probability profiles for the automated assessment of the recovery of patients with stroke from activity of daily living tasks. Artif. Intell. Med. 2009, 46, 233-249.
3. Duda, R.O.; Hart, P.E.; Stork, D.G. Pattern Classification, second ed.; John Wiley & Sons: New York, NY, USA, 2001.
4. Lewis II, P.M. The characteristic selection problem in recognition systems. IEEE Trans. Inf. Theory 1962, 8, 171-178.
5. Guyon, I.; Elisseeff, A. An introduction to variable and feature selection. J. Mach. Learn. Res. 2003, 3, 1157-1182.
6. Wang, G.; Lochovsky, F.H.; Yang, Q. Feature selection with conditional mutual information maximin in text categorization. Proceedings of the 13th ACM International Conference on Information and Knowledge Management (CIKM'04); Evans, D.A.; Gravano, L.; Herzog, O.; Zhai, C.; Ronthaler, M., Eds.; ACM Press: New York, NY, USA, 2004; pp. 342-349.
7. Guo, B.; Nixon, M.S. Gait feature subset selection by mutual information. IEEE Trans. Syst. Man Cybern. Part A-Syst. Hum. 2009, 29, 36-46.
8. Huang, D.; Chow, T.W.S.; Wa, E.W.M.; Li, J. Efficient selection of discriminative genes from microarray gene expression data for cancer diagnosis. IEEE Trans. Circuits Syst. I-Regul. Pap. 2005, 52, 1909-1918.
9. Kamentsky, L.A.; Liu, C.N. Computer-automated design of multifont print recognition logic. IBM J. Res. Dev. 1963, 7, 2-13.
10. Liu, C.N. A programmed algorithm for designing multifont character recognition logics. IEEE Trans. Electron. 1964, EC-13, 586-593.
11. Battiti, R. Using mutual information for selecting features in supervised neural net learning. IEEE Trans. Neural Netw. 1994, 5, 537-550.
12. Cover, T.M.; Thomas, J.A. Elements of Information Theory, second ed.; John Wiley & Sons: Hoboken, NJ, USA, 2006.
13. Liu, D.; Chang, T.; Zhang, Y. A constructive algorithm for feedforward neural networks with incremental training. IEEE Trans. Circuits Syst. I-Regul. Pap. 2002, 49, 1876-1879.
14. McGill, W.J. Multivariate information transmission. IEEE Trans. Inf. Theory 1954, 4, 93-111.
15. Matsuda, H. Physical nature of higher-order mutual information: Intrinsic correlations and frustration. Phys. Rev. E 2000, 62, 3096-3102.
16. Shiono, S.; Yamada, S.; Nakashima, M.; Matsumoto, K., Information theoretic analysis of connection structure from spike trains. In Advances in Neural Information Processing Systems 5; Hanson, S.J.; Cowan, J.D.; Giles, C.L., Eds.; Morgan Kaufmann Publishers Inc.: San Francisco, CA, USA, 1993; pp. 515-522.
17. Bontempi, G.; Meyer, P.E., Causal filter selection in microarray data. In Proceedings of the 27th International Conference on Machine Learning; Fürnkranz, J.; Joachims, T., Eds.; Omnipress, 2010; pp. 95-102.
18. Kotz, S.; Nadarajah, S. Multivariate t Distributions and Their Applications; Cambridge University Press: Cambridge, UK, 2004; pp. 15-16.
19. Pearl, J. Probabilistic Reasoning in Intelligent Systems; Morgan Kaufmann: San Francisco, California, USA, 1988.
20. Neapolitan, R.E. Learning Bayesian Networks; Prentice-Hall, Inc.: Upper Saddle River, NJ, USA, 2003.
21. Van Dijck, G.; Van Hulle, M.M. Speeding up feature subset selection through mutual information relevance filtering. In Knowledge Discovery in Databases: PKDD 2007; Kok, J.; Koronacki, J.; Lopez de Mantaras, R.; Matwin, S.; Mladenic, D.; Skowron, A., Eds.; Springer Berlin/Heidelberg, 2007; Vol. 4702, Lecture Notes in Computer Science, pp. 277-287.
22. Van Dijck, G. Information Theoretic Approach to Feature Selection and Redundancy Assessment. PhD dissertation, Katholieke Universiteit Leuven, 2008.
23. Kwak, N.; Choi, C.H. Input feature selection for classification problems. IEEE Trans. Neural Netw. 2002, 13, 143-159.
24. Peng, H.; Long, F.; Ding, C. Feature selection based on mutual information: Criteria of max-dependency, max-relevance and min-redundancy. IEEE Trans. Pattern Anal. Mach. Intell. 2005, 27, 1226-1238.
25. Estévez, P.A.; Tesmer, M.; Perez, C.A.; Zurada, J.M. Normalized mutual information feature selection. IEEE Trans. Neural Netw. 2009, 20, 189-201.
26. Fleuret, F. Fast binary feature selection with conditional mutual information. J. Mach. Learn. Res. 2004, 5, 1531-1555.
27. Kwak, N.; Choi, C.H. Input feature selection by mutual information based on Parzen window. IEEE Trans. Pattern Anal. Mach. Intell. 2002, 24, 1667-1671.
28. Bonev, B.; Escolano, F.; Cazorla, M. Feature selection, mutual information, and the classification of high-dimensional patterns: applications to image classification and microarray data analysis. Pattern Anal. Appl. 2008, 11, 309-319.
29. François, D.; Rossi, F.; Wertz, V.; Verleysen, M. Resampling methods for parameter-free and robust feature selection with mutual information. Neurocomputing 2007, 70, 1276-1288.
30. Hellman, M.E.; Raviv, J. Probability of error, equivocation, and the Chernoff bound. IEEE Trans. Inf. Theory 1970, IT-16, 368-372.
31. Kovalevsky, V.A. The problem of character recognition from the point of view of mathematical statistics. In Character Readers and Pattern Recognition; Kovalevsky, V.A., Ed.; Spartan: New York, NY, USA, 1968.
32. Tebbe, D.L.; Dwyer III, S.J. Uncertainty and the probability of error. IEEE Trans. Inf. Theory 1968, IT-14, 516-518.
33. Feder, M.; Merhav, N. Relations between entropy and error probability. IEEE Trans. Inf. Theory 1994, 40, 259-266.
34. Golić, J.D. Comment on "Relations between entropy and error probability". IEEE Trans. Inf. Theory 1999, 45, 372-372.
35. Fano, R.M. Transmission of Information: A Statistical Theory of Communication; John Wiley & Sons: New York, NY, USA, 1961.