Empirical comparison of tree ensemble variable importance measures

Chemometrics and Intelligent Laboratory Systems

Volumn 105, Issue 2, 2011, Pages 157-170

  Auret, Lidia ^a   Aldrich, Chris ^a  

^a STELLENBOSCH UNIVERSITY   (South Africa)

Author keywords

Boosted trees; Conditional inference forests; Decision trees; Ensemble learning; Fault identification; Random forests; Variable importance

Indexed keywords

ACCURACY; ARTICLE; CONTROLLED STUDY; DECISION TREE; INTERMETHOD COMPARISON; MATHEMATICAL COMPUTING; MATHEMATICAL VARIABLE; PRIORITY JOURNAL; PROCESS MODEL; PROCESS OPTIMIZATION; QUALITATIVE ANALYSIS; SENSITIVITY AND SPECIFICITY; STATISTICAL ANALYSIS; STATISTICAL MODEL; VALIDATION PROCESS;

EID: 79951953906     PISSN: 01697439     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.chemolab.2010.12.004     Document Type: Article

References (36)

1. Garzón M.B., Blazek R., Neteler M., Dios R.S.D., Ollero H.S., Furlanello C. Predicting habitat suitability with machine learning models: the potential area of Pinus sylvestris L. in the Iberian Peninsula. Ecological Modelling 2006, 197(3):383-393.
2. Cutler A., Stevens J.R. Random forests for microarrays. Methods in Enzymology; DNA Microarrays, Part B: Databases and Statistics 2006, vol. 411:422-432. Academic Press.
3. Peters J., Baets B.D., Verhoest N.E.C., Samson R., Degroeve S., Becker P.D., Huybrechts W. Random forests as a tool for ecohydrological distribution modelling. Ecological Modelling 2007, 207(2):304-318.
4. Svetnik V., Liaw A., Tong C., Culberson J.C., Sheridan R.P., Feuston B.P. Random forest: a classification and regression tool for compound classification and QSAR modeling. Journal of Chemical Information and Computer Sciences 2003, 43(6):1947-1958.
5. Svetnik V., Wang T., Tong C., Liaw A., Sheridan R.P., Song Q. Boosting: an ensemble learning tool for compound classification and QSAR modeling. Journal of Chemical Information and Modeling 2005, 45(3):786-799.
6. Gupta S., Matthew S., Abreu P.M., Aires-de-Sousa J. QSAR analysis of phenolic antioxidants using MOLMAP descriptors of local properties. Bioorganic & medicinal chemistry 2006, 14(4):1199-1206.
7. Sakiyama Y., Yuki H., Moriya T., Hattori K., Suzuki M., Shimada K., Honma T. Predicting human liver microsomal stability with machine learning techniques. Journal of Molecular Graphics and Modelling 2008, 26(6):907-915.
8. Zhang M.H., Xu Q.S., Daeyaert F., Lewi P.J., Massart D.L. Application of boosting to classification problems in chemometrics. Analytica Chimica Acta 2005, 544(1):167-176.
9. Cummings M., Segal M. Few amino acid positions in rpoB are associated with most of the rifampin resistance in Mycobacterium tuberculosis. BMC Bioinformatics 2004, 5(1):137-143.
10. Diaz-Uriarte R., Alvarez de Andres S. Gene selection and classification of microarray data using random forest. BMC Bioinformatics 2006, 7(1):3-15.
11. Pang H., Lin A., Holford M., Enerson B.E., Lu B., Lawton M.P., Floyd E., Zhao H. Pathway analysis using random forests classification and regression. Bioinformatics 2006, 22(16):2028-2036.
12. Archer K.J., Kimes R.V. Empirical characterization of random forest variable importance measures. Computational Statistics & Data Analysis 2008, 52(4):2249-2260.
13. Nicodemus K.K., Malley J.D. Predictor correlation impacts machine learning algorithms: implications for genomic studies. Bioinformatics 2009, 25(15):1884-1890.
14. Strobl C., Zeileis A. Danger: High Power! Exploring the Statistical Properties of a Test for Random Forest Variable Importance 2008, Department of Statistics, University of Munich.
15. Strobl C., Boulesteix A.L., Zeileis A., Hothorn T. Bias in random forest variable importance measures: illustrations, sources and a solution. BMC Bioinformatics 2007, 8:25-45.
16. Strobl C., Boulesteix A., Kneib T., Augustin T., Zeileis A. Conditional variable importance for random forests. BMC Bioinformatics 2008, 9(1):307-317.
17. Grömping U. Variable importance assessment in regression: linear regression versus random forest. The American Statistician 2009, 63(4):308-319.
18. Genuer R., Poggi J., Tuleau-Malot C. Variable selection using random forests. Pattern Recognition Letters 2010, 10.1016/j.patrec.2010.03.014.
19. Sandri M., Zuccolotto P. A bias correction algorithm for the Gini variable importance measure in classification trees. Journal of Computational and Graphical Statistics 2008, 17(3):611-628.
20. Breiman L., Friedman J.H., Olshen R.A., Stone C.J. Classification and Regression Trees 1993, Chapman & Hall.
21. Dietterich T.G. Ensemble methods in machine learning. Proceedings of the First International Workshop on Multiple Classifier Systems 2000, 1857:1-15.
22. Breiman L. Bagging predictors. Machine Learning 1996, 24(2):123-140.
23. Breiman L. Random forests. Machine Learning 2001, 45(1):5-32.
24. Hothorn T., Hornik K., Zeileis A. Unbiased recursive partitioning: a conditional inference framework. Journal of Computational and Graphical Statistics 2006, 15(3):651-674.
25. Friedman J.H. Greedy function approximation: a gradient boosting machine. The Annals of Statistics 2001, 29(5):1189-1232.
26. Ridgeway W. Generalized boosted models: a guide to the gbm package [Online]. Available:. http://i-pensieri.com/gregr/papers/gbm-vignette.pdf.
27. Breiman L., Cutler A. Manual on setting up, using, and understanding random forests v4.0 [Online]. Available:, [Accessed: 30-May-2008]. ftp://ftp.stat.berkeley.edu/pub/users/breiman/Using_random_forests_v4.0.pdf.
28. Rodenburg W., Heidema A.G., Boer J.M.A., Bovee-Oudenhoven I.M.J., Feskens E.J.M., Mariman E.C.M., Keijer J. A framework to identify physiological responses in microarray-based gene expression studies: selection and interpretation of biologically relevant genes. Physiological Genomics Oct. 2008, 33(1):78-90.
29. R Development Core Team R: A Language and Environment for Statistical Computing 2010, R Foundation for Statistical Computing, Vienna, Austria.
30. Liaw A., Wiener M. Classification and regression by randomForest. R News 2002, 2(3):18-22.
31. Ridgeway G. gbm: generalized boosted regression models [Online]. Available:. http://www.i-pensieri.com/gregr/gbm.shtml.
32. Nicodemus K.K., Malley J.D., Strobl C., Ziegler A. The behaviour of random forest permutation-based variable importance measures under predictor correlation. BMC Bioinformatics 2010, 11(1):110-122.
33. Chong I., Jun C. Performance of some variable selection methods when multicollinearity is present. Chemometrics and Intelligent Laboratory Systems Jul. 2005, 78(1):103-112.
34. Russell E.L., Chiang L.H., Braatz R.D. Fault detection in industrial processes using canonical variate analysis and dynamic principal component analysis. Chemometrics and Intelligent Laboratory Systems 2000, 51(1):81-93.
35. Downs J.J., Vogel E.F. A plant-wide industrial process control problem. Computers & Chemical Engineering 1993, 17(3):245-255.
36. Russell E.L., Chiang L.H., Braatz R.D. Data-driven techniques for fault detection and diagnosis in chemical processes. Springer 2000.