Feature selection with ensembles, artificial variables, and redundancy elimination

Journal of Machine Learning Research

Volumn 10, Issue , 2009, Pages 1341-1366

  Tuv, Eugene ^a   Borisov, Alexander ^b   Runger, George ^c   Torkkola, Kari ^d  

^a INTEL CORPORATION   (United States)

^b Intel Corporation   (Russian Federation)

^c Arizona State University   (United States)

^d AMAZON   (United States)

Author keywords

Importance; Masking; Resampling; Residuals; Trees

Indexed keywords

IMPORTANCE; MASKING; RESAMPLING; RESIDUALS; TREES;

PHOTOMASKS;

ALGORITHMS; MASKING; MODELS; SAMPLING; SIMULATION; TREES; WRAPPERS;

EID: 68949154557     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (43)

1. H. Almuallin and T. G. Dietterich. Learning boolean concepts in the presence of many irrelevant features. Artificial Intelligence, 69(1-2):279-305, 1994.
2. Y. Amit and D. Geman. Shape quantization and recognition with randomized trees. Neural Computation, 9(7):1545-88, 1997.
3. E. Bauer and R. Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting and variants. Machine Learning, 36(1/2):525-536, 1999.
4. A. Berrado and G.C. Runger. Using metarules to organize and group discovered association rules. Data Mining and Knowledge Discovery, 14(3):409-431, 2007.
5. A. Borisov, V. Eruhimov, and E. Tuv. Tree-based ensembles with dynamic soft feature selection. In I. Guyon, S. Gunn, M. Nikravesh, and L. Zadeh, editors, Feature Extraction Foundations and Applications: Studies in Fuzziness and Soft Computing. Springer, 2006.
6. B. Boser, I. Guyon, and V. Vapnik. A training algorithm for optimal margin classifiers. In D. Haussler, editor, 5th Annual ACM Workshop on COLT, Pittsburgh, PA, pages 144-152. ACM Press, 1992.
7. O. Bousquet and A. Elisseeff. Algorithmic stability and generalization performance. In Advances in Neural Information Processing Systems, volume 13, pages 196-202. MIT Press, 2001.
8. L. Breiman. Bagging predictors. Machine Learning, 24(2):123-140, 1996.
9. L. Breiman. Arcing classifiers. The Annals of Statistics, 26(3):801-849, 1998.
10. L. Breiman. Random forests. Machine Learning, 45(1):5-32, 2001.
11. L. Breiman, J. Friedman, R. Olshen, and C. Stone. Classification and Regression Trees. Wadsworth, Belmont, MA, 1984.
12. S. Cost and S. Salzberg. A weighted nearest neighbor algorithm for learning with symbolic features. Machine Learning, 10(1):57-78, 1993.
13. P. Diaconis and B. Efron. Computer intensive methods in statistics. Scientific American, (248): 116-131, 1983.
14. T. G. Dietterich. An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization. Machine Learning, 40(2):139-157, 2000a.
15. T. G. Dietterich. Ensemble methods in machine learning. In First International Workshop on Multiple Classifier Systems 2000, Cagliari, Italy, volume 1857 of Lecture Notes in Computer Science, pages 1-15. Springer, 2000b.
16. B. Efron, T. Hastie, L. Johnstone, and R. Tibshirani. Least angle regression. Annals of Statistics, 32:407-499, 2004.
17. Y. Freund and R. E. Schapire. Experiments with a new boosting algorithm. In The 13th International Conference on Machine Learning, pages 148-156. Morgan Kaufman, 1996.
18. J. Friedman. Greedy function approximation: a gradient boosting machine. Technical report, Dept. of Statistics, Stanford University, 1999.
19. J. Friedman, T. Hastie, and R. Tibshirani. Additive logistic regression: A statistical view of boosting. Annals of Statistics, 28:832-844, 2000.
20. J. H Friedman,M. Jacobson, andW. Stuetzle. Projection pursuit regression. Journal of the American Statistical Association, 76:817-823, 1981.
21. I. Guyon and A. Elisseeff. An introduction to variable and feature selection. Journal of Machine Learning Research, 3:1157-1182, Mar 2003.
22. I. Guyon, J. Weston, S. Barnhill, and V. Vapnik. Gene selection for cancer classification using support vector machines. Machine Learning, 46(1-3):389-422, 2002.
23. M. A. Hall. Correlation-based feature selection for discrete and numeric class machine learning. In Proceedings of the 17th International Conference on Machine Learning, pages 359-366, 2000.
24. L. K. Hansen and P. Salamon. Neural network ensembles. IEEE Trans. on Pattern Analysis and Machine Intelligence, 12(10):993-1001, 1990.
25. T. K. Ho. The random subspace method for constructing decision forests. IEEE Trans. on Pattern Analysis and Machine Intelligence, 20(8):832-844, 1998.
26. K. Kira and L. A. Rendell. A practical approach to feature selection. In ML92: Proceedings of the ninth international workshop on Machine learning, pages 249-256, San Francisco, CA, USA, 1992. Morgan Kaufmann Publishers Inc. ISBN 1-5586-247-X.
27. D. Koller and M. Sahami. Toward optimal feature selection. In Proceedings of ICML-96, 13th International Conference on Machine Learning, pages 284-292, Bari, Italy, 1996. URL citeseer.nj.nec.com/ koller96toward.html.
28. S. Kullback and R.A. Liebler. On information and sufficiency. Annals of Mathematical Statistics, 22:76-86, 1951.
29. H. Liu and L. Yu. Toward integrating feature selection algorithms for classification and clustering. IEEE Trans. Knowledge and Data Eng., 17(4):491-502, 2005.
30. S. Mukherjee, P. Niyogi, T. Poggio, and R. Rifkin. Learning theory: Stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Advances in Computational Mathematics, 25:161-193, 2006.
31. B. Parmanto, P. Munro, and H. Doyle. Improving committee diagnosis with resampling techniques. In D. S. Touretzky, M. C. Mozer, and M. Hesselmo, editors, Advances in Neural Information Processing Systems 8, pages 882-888. Cambridge, MA: MIT Press, 1996.
32. T. Poggio, R. Rifkin, S. Mukherjee, and A. Rakhlin. Bagging regularizes. In CBCL Paper 214/AI Memo 2002-003. MIT, Cambridge, MA, 2002.
33. T. Poggio, R. Rifkin, S. Mukherjee, and P. Niyogi. General conditions for predictivity in learning theory. Nature, 428:419-422, 2004.
34. M. Robnik-Sikonja and I. Kononenko. Theoretical and empirical analysis of relief and relieff. Machine Learning, 53:23-69, 2003.
35. C. Stanfill and D. Waltz. Toward memory-based reasoning. Communications of the ACM, 29: 1213-1228, December 1986.
36. A. Statnikov and C.F. Aliferis. Tied: An artificially simulated dataset with multiple Markov boundaries. Journal of Machine Learning Research Workshop Conference & Proceedings, 2009. to appear.
37. H. Stoppiglia, G. Dreyfus, R. Dubois, and Y. Oussar. Ranking a random feature for variable and feature selection. Journal of Machine Learning Research, 3:1399-1414, March 2003.
38. E. Tuv. Ensemble learning and feature selection. In I. Guyon, S. Gunn,M. Nikravesh, and L. Zadeh, editors, Feature Extraction, Foundations and Applications. Springer, 2006.
39. E. Tuv, A. Borisov, and K. Torkkola. Feature selection using ensemble based ranking against artificial contrasts. In Proceedings of the International Joint Conference on Neural Networks (IJCNN), 2006.
40. G. Valentini and T. Dietterich. Low bias bagged support vector machines. In ICML 2003, pages 752-759, 2003.
41. G. Valentini and F. Masulli. Ensembles of learning machines. In M. Marinaro and R. Tagliaferri, editors, Neural Nets WIRN Vietri-02, Lecture Notes in Computer Science. Springer-Verlag, 2002.
42. J.W. Wisnowski, J.R. Simpson, D.C. Montgomery, and G.C. Runger. Resampling methods for variable selection in robust regression. Computational Statistics and Data Analysis, 43(3):341-355, 2003.
43. L. Yu and H. Liu. Efficient feature selection via analysis of relevance and redundancy. J. of Machine Learning Research, 5:1205-1224, 2004.