Feature Selection via l1-Penalized Squared-Loss Mutual Information

IEICE Transactions on Information and Systems

Volumn E96-D, Issue 7, 2013, Pages 1513-1524

  Jitkrittum, Wittawat ^a   Hachiya, Hirotaka ^a   Sugiyama, Masashi ^a  

^a Tokyo Institute of Technology ^*  (Japan)

Author keywords

Density ratio estimation; Dimensionality reduction; Feature selection; L1 regularization; Squared loss mutual information

Indexed keywords

FEATURE EXTRACTION; INVERSE PROBLEMS; REDUNDANCY;

DENSITY-RATIO ESTIMATIONS; DIMENSIONALITY REDUCTION; FEATURE INTERACTIONS; FEATURE SELECTION ALGORITHM; IMPORTANT FEATURES; KEY CHARACTERISTICS; L1-REGULARIZATION; MUTUAL INFORMATIONS;

ALGORITHMS;

EID: 84877698311     PISSN: 09168532     EISSN: 17451361     Source Type: Journal    
DOI: 10.1587/transinf.E96.D.1513     Document Type: Article

References (39)

1. S.M. Ali and S.D. Silvey, "A general class of coefficients of divergence of one distribution from another," J. Royal Statistical Society (Series B), vol.28, no.1, pp.131-142, 1966.
2. N. Aronszajn, "Theory of reproducing kernels," Trans. American Mathematical Society, vol.68, pp.337-404, 1950.
3. C.C. Chang and C.J. Lin, "LIBSVM: A library for support vector machines," Tech. Rep., Department of Computer Science, National Taiwan University, 2001. http://www.csie.ntu.edu.tw/?cjlin/libsvm/
4. T.M. Cover and J.A. Thomas, Elements of Information Theory, 2nd ed., John Wiley & Sons, Hoboken, NJ, USA, 2006.
5. I. Csisźar, "Information-type measures of difference of probability distributions and indirect observation," Studia Scientiarum Mathematicarum Hungarica, vol.2, pp.229-318, 1967.
6. J. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra, "Efficient projections onto the-1-ball for learning in high dimensions," Proc. 25th Annual International Conference on Machine Learning (ICML 2008), ed. A. McCallum and S. Roweis, pp.272-279, 2008.
7. K. Fukumizu, F.R. Bach, and M.I. Jordan, "Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces," J. Machine Learning Research, vol.5, no.1, pp.73-99, 2004.
8. A. Gretton, O. Bousquet, A. Smola, and B. Scholkopf, "Measuring statistical dependence with Hilbert-Schmidt norms," Algorithmic Learning Theory, ed. S. Jain, H.U. Simon, and E. Tomita, Lecture Notes in Artificial Intelligence, Berlin, Germany, pp.63-77, Springer-Verlag, 2005.
9. I. Guyon and A. Elisseeff, "An introduction to variable and feature selection," J.Machine Learning Research, vol.3, no.3, pp.1157-1182, 2003.
10. H. Hachiya and M. Sugiyama, "Feature selection for reinforcement learning: Evaluating implicit state-reward dependency via conditional mutual information," Machine Learning and Knowledge Discovery in Databases, Part I, ed. J.L. Balc'azar, A.G.F. Bonchi, and M. Sebag, Lect. Notes Comput. Sci., vol.6321, Berlin, pp.474-489, Springer, 2010.
11. M.A. Hall, "Correlation-based feature selection for discrete and numeric class machine learning," Proc. Seventeenth International Conference on Machine Learning, pp.359-366, San Francisco, CA, USA, 2000.
12. X. He, D. Cai, and P. Niyogi, "Laplacian score for feature selection," in Advances in Neural Information Processing Systems 18, ed. Y. Weiss, B. Scholkopf, and J. Platt, pp.507-514, MIT Press, Cambridge, MA, 2006.
13. K. Kira and L.A. Rendell, "A practical approach to feature selection," Proc. Ninth International Workshop on Machine Learning, pp.249-256, San Francisco, CA, USA, 1992.
14. R. Kohavi and G.H. John, "Wrappers for feature subset selection," Artif. Intell., vol.97, no.1, pp.273-324, 1997.
15. I. Kononenko, "Estimating attributes: Analysis and extensions of RELIEF," European Conference on Machine Learning, ed. F. Bergadano and L.D. Raedt, pp.171-182, New York, NY, USA, 1994.
16. S. Kullback and R.A. Leibler, "On information and sufficiency," The Annals of Mathematical Statistics, vol.22, pp.79-86, 1951.
17. P. Langley, "Selection of relevant features in machine learning," Proc. AAAI Fall Symposium on Relevance, pp.140-144, Menlo Park, CA, USA, 1994.
18. S.I. Lee, H. Lee, P. Abbeel, and A.Y. Ng, "Efficient L1 regularized logistic regression," Proc. 21st National Conference on Artificial Intelligence (AAAI), pp.401-408, 2006.
19. F. Li, Y. Yang, and E. Xing, "From lasso regression to feature vector machine," in Advances in Neural Information Processing Systems 18, ed. Y.Weiss, B. Scholkopf, and J. Platt, pp.779-786, MIT Press, Cambridge, MA, 2006.
20. F. Liese and I. Vajda, "On divergences and informations in statistics and information theory," IEEE Trans. Inf. Theory, vol.52, no.10, pp.4394-4412, 2006.
21. J. Liu, J. Chen, and J. Ye, "Large-scale sparse logistic regression," Proc. 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.547-556, New York, NY, USA, 2009.
22. M. Masaeli, G. Fung, and J.G. Dy, "From transformation-based dimensionality reduction to feature selection," Proc. 27th International Conference on Machine Learning, pp.751-758, 2010.
23. H. Peng, F. Long, and C. Ding, "Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and minredundancy," IEEE Trans. Pattern Anal. Mach. Intell., vol.27, no.8, pp.1226-1238, 2005.
24. I. Rodriguez-Lujan, R. Huerta, C. Elkan, and C.S. Cruz, "Quadratic programming feature selection," J. Machine Learning Research, vol.11, no.8, pp.1491-1516, Aug. 2010.
25. Y. Saeys, I. Inza, and P. Larranaga, "A review of feature selection techniques in bioinformatics," Bioinformatics, vol.23, no.19, pp.2507-2517, 2007.
26. M.W. Schmidt, G. Fung, and R. Rosales, "Fast optimization methods for L1 regularization: A comparative study and two new approaches," European Conference on Machine Learning, pp.286-297, 2007.
27. B. Scholkopf and A.J. Smola, Learning with Kernels, MIT Press, Cambridge, MA, USA, 2002.
28. L. Song, A. Smola, A. Gretton, K.M. Borgwardt, and J. Bedo, "Supervised feature selection via dependence estimation," Proc. 24th Annual International Conference on Machine Learning, pp.823-830, 2007.
29. I. Steinwart, "On the influence of the kernel on the consistency of support vector machines," J. Machine Learning Research, vol.2, pp.67-93, Nov. 2001.
30. T. Suzuki and M. Sugiyama, "Sufficient dimension reduction via squared-loss mutual information estimation," Neural Comput., vol.25, no.3, pp.725-758, 2013.
31. T. Suzuki, M. Sugiyama, J. Sese, and T. Kanamori, "Approximating mutual information by maximum likelihood density ratio estimation," Proc. ECML-PKDD2008 Workshop on New Challenges for Feature Selection in Data Mining and Knowledge Discovery 2008 (FSDM2008), ed. Y. Saeys, H. Liu, I. Inza, L. Wehenkel, and Y.V. de Peer, JMLRWorkshop and Conference Proceedings, vol.4, pp.5-20, Antwerp, Belgium, Sept. 2008.
32. T. Suzuki andM. Sugiyama, "Least-squares independent component analysis," Neural Comput., vol.23, no.1, pp.284-301, 2011.
33. T. Suzuki, M. Sugiyama, T. Kanamori, and J. Sese, "Mutual information estimation reveals global associations between stimuli and biological processes," BMC Bioinformatics, vol.10, no.S-1, p.S52, 2009.
34. R. Tibshirani, "Regression shrinkage and selection via the lasso," J. Royal Statistical Society (Series B), vol.58, no.1, pp.267-288, 1996.
35. K. Torkkola, "Feature extraction by non-parametric mutual information maximization," J. Machine Learning Research, vol.3, no.3, pp.1415-1438, March 2003.
36. J. Weston, A. Elisseeff, B. Scholkopf, and M. Tipping, "Use of the zero norm with linear models and kernel methods," J. Machine Learning Research, vol.3, no.3, pp.1439-1461, March 2003.
37. Z. Zhao and H. Liu, "Spectral feature selection for supervised and unsupervised learning," Proc. 24th International Conference on Machine Learning, pp.1151-1157, New York, NY, USA, 2007.
38. Z. Zhao, L. Wang, and H. Liu, "Efficient spectral feature selection with minimum redundancy," Proc. Twenty-Fourth AAAI Conference on Artificial Intelligence, pp.673-678, 2010.
39. J. Zhu, S. Rosset, T. Hastie, and R. Tibshirani, "1-norm support vector machines," in Advances in Neural Information Processing Systems 16, ed. S. Thrun, L. Saul, and B. Scholkopf, MIT Press, Cambridge, MA, USA, 2004.