A bias correction algorithm for the gini variable importance measure in classification trees

Journal of Computational and Graphical Statistics

Volumn 17, Issue 3, 2008, Pages 611-628

  Sandri, Marco ^a   Zuccolotto, Paola ^b  

^a University of Brescia ^*  (Italy)

^b UNIVERSITY OF BRESCIA   (Italy)

Author keywords

Bias; Learning ensemble; Variable importance; Variable selection

Indexed keywords

EID: 53549131556     PISSN: 10618600     EISSN: None     Source Type: Journal    
DOI: 10.1198/106186008X344522     Document Type: Article

References (34)

1. Bell, D. and Wang, H. (2000): "A Formalism for Relevance and its Application in Feature Subset Selection," Machine Learning, 4, 2, 175-195.
2. Breiman, L. (2001a), "Random Forests," Machine Learning, 45, 5-32.
3. (2001b), "Statistical Modeling: The Two Cultures," Statistical Science, 16, 3, 199-231.
4. (2002), "Manual on Setting Up, Using, and Understanding Random Forests v3.1." Technical Report, ftp://ftp.stat.berkeley.edu/pub/users/ breiman/Using-random-forests-v3.1.pdf.
5. Breiman, L., Friedman, J.H., Olshen, R.A., and Stone, C.J. (1984), Classification and Regression Trees, London: Chapman & Hall.
6. Breiman, L., Cutler, A., Liaw, A., and Wiener, M. (2006), Breiman and Cutler's Random Forests for Classification and Regression. R package version 4.5-18 http://cran.r-project.org/doc/packages/randomForest.pdf.
7. Bureau, A., Dupuis, J., Hayward, B., Falls, K., and Van Eerdewegh, P. (2003), "Mapping Complex Traits using Random Forests," BMC Genetics, 4(Suppl. 1): S64, http://www.biomedcentral.eom/1471-2156/4/s1/ S64.
8. De'ath, G. (2007), "Boosted Trees for Ecological Modeling and Prediction," Ecology, 88, 1, 243-251.
9. Diaz-Uriarte, R., and Alvarez de Andrés, S. (2006), "Gene Selection and Classification of Microarray Data using Random Forest," BMC Genetics, 7:3, http://www.biomedcentral.com/1471-2105/7/3.
10. Dobra, A., and Gehrke, J. (2001), "Bias Correction in Classification Tree Construction," in Proceedings of the Seventeenth International Conference on Machine Learning, eds. C.E. Brodley and A. P. Danyluk, Williams College, Williamstown, MA, 90-97.
11. Friedman, J.H. (2001), "Greedy Function Approximation: A Gradient Boosting Machine," The Annals of Statistics, 29, 1189-1232.
12. (2002), "Tutorial: Getting Started with MART in R," Technical Report, Standford University. Available online at http://www-stat.stanford. edu/∼jhf/r-mart/tutorial/tutorial.pdf.
13. Friedman, J.H., and Meulman, J.J. (2003), "Multiple Additive Regression Trees with Application in Epidemiology," Statistics in Medicine, 22, 1365-1381.
14. Gong, G. (1986), "Cross-Validation, the Jackknife, and the Bootstrap: Excess Error Estimation in Forward Logistic Regression," Journal of the American Statistical Association, 81, 108-113.
15. Guglielmi, A., Ruzzenente, A., Sandri, M., Kind, R., Lombardo, F., Rodella, L., Catalano, F., De Manzoni, G. and Cordiano, C. (2002), "Risk Assessment and Prediction of Rebleeding in Bleeding Gastroduodenal Ulcer," Endoscopy, 34, 771-779.
16. Guha, R., and Jurs, P.C. (2004), "Development of Linear, Ensemble, and Nonlinear Models for the Prediction and Interpretation of the Biological Activity of a Set of PDGFR Inhibitors," Journal of Chemical Inference and Computer Science, 44, 2179-2189.
17. Hothorn, T., Hornik, K., and Zeileis, A. (2006a), "Unbiased Recursive Partitioning: A Conditional Inference Framework," Journal of Computational and Graphical Statistics, 15, 651-674.
18. Hothorn, T., Hornik, K., and Zeileis, A. (2006b), "party: A Laboratory for Recursive Partitioning," R package version 0.9-11. Available online at http://cran.r-project.org/doc/vignettes/party/party.pdf.
19. John, G.H., Kohavi, R., and Pfleger, K. (1994), "Irrelevant Features and the Subset Selection Problem," in Proceedings of the llth International Conference on Machine Learning, eds. W. W. Cohen and H. Hirsch, New Brunswick, NJ: Morgan Kaufmann, 121-129.
20. Kim, H., and Loh, W. (2001), "Classification Trees with Unbiased Multiway Splits," Journal of the American Statistical Association, 96, 589-604.
21. Kononenko, I. (1995), "On Biases in Estimating Multi-Valued Attributes," in Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, ed. C. Mellish, Montréal, Canada, 1034-1040.
22. Loh, W.-Y., Shih, Y.-S. (1997), "Split Selection Methods for Classification Trees," Statististica Sinica, 7, 815-840.
23. Lunetta, K.L., Hayward, B.L., Segal, J., and Van Eerdewegh, P. (2004), "Screening Large-Scale Association Study Data: Exploiting Interactions using Random Forests," BMC Genetics, 5, 32. Available online at http://www.biomedcentral.com/1471-2156/5/32.
24. Menze, B.H., Petrich, W., and Hamprecht F.A. (2007), "Multivariate Feature Selection and Hierarchical Classification for Infrared Spectroscopy: Serum-Based Detection of Bovine Spongiform Encepbalopathy," Analytical and Bioanalytical Chemistry, doi:10.1007/s00216-006-1070-5.
25. Pearl, J. (1988), Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, San Francisco, CA: Morgan Kaufmann Publishers, Inc.
26. Ridgeway, G. (2007), "Generalized Boosted Models: A Guide to the gbm Package," Available online at http://i-pensieri.com/gregr/papers/gbm- vignette.pdf.
27. Schonlau, M. (2005), "Boosted Regression (Boosting): A Tutorial and a Stata Plugin," The Stata Journal, 5, 3, 330-354.
28. Strobl, C. (2005), "Statistical Sources of Variable Selection Bias in Classification Trees Based on the Gini Index," Technical Report, SFB 386, http://epub.ub.uni-muenchen.de/archive/00001789/01/paper.420.pdf.
29. Strobl, C., Boulesteix, A.-L., Zeileis, A., and Hothorn, T. (2007a), "Bias in Random Forest Variable Importance Measures: Illustrations, Sources and a Solution," BMC Bioinformatics, 8, 25, doi:10.1186/1471-2105- 8-25.
30. Strobl, C., Boulesteix, A.-L., and Augustin, T. (2007b), "Unbiased Split Selection for Classification Trees Based on the Gini Index," Computational Statistics & Data Analysis, doi:10.1016/j.csda.2006.12. 030.
31. Svetnik, V., Wang, T., Tong, C., Liaw, A., Sheridan, R.P., and Song Q. (2005), "Boosting: An Ensemble Learning Tool for Compound Classification and QSAR Modeling," Journal of Chemical Information and Modeling, 45, 786-799
32. White, A.P., and Liu, W.Z. (1994), "Bias in Information-Based Measures in Decision Tree Induction," Machine Learning, 15, 321-329.
33. Wu, Y., Boos, D.D., and Stefanski, L.A. (2007), "Controlling Variable Selection by the Addition of Pseudovariables," Journal of the American Statistical Association, 102, 235-243.
34. Yu, L., and Liu, H. (2004), "Efficient Feature Selection via Analysis of Relevance and Redundancy," Journal of Machine Learning Research, 5, 1205-1224.