Mining data with random forests: A survey and results of new tests

Pattern Recognition

Volumn 44, Issue 2, 2011, Pages 330-349

  Verikas, A ^a,b   Gelzinis, A ^b   Bacauskiene, M ^b  

^a HALMSTAD UNIVERSITY   (Sweden)

^b KAUNAS UNIVERSITY OF TECHNOLOGY   (Lithuania)

Author keywords

Classifier; Data proximity; Random forests; Variable importance; Variable selection

Indexed keywords

APPLICATION EXAMPLES; DATA EXPLORATION; DATA PROXIMITY; IMPORTANCE MEASURE; LITERATURE SURVEY; OUTLIER DETECTION; RANDOM FORESTS; SMALL DATA SET; VARIABLE IMPORTANCE; VARIABLE RANKING; VARIABLE SELECTION;

CLASSIFIERS; SURVEYS;

DECISION TREES;

EID: 77958064179     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2010.08.011     Document Type: Article

References (112)

1. V.N. Vapnik Statistical Learning Theory 1998 Wiley New York
2. J. Shawe-Taylor, and N. Cristianini Kernel Methods for Pattern Analysis 2004 Cambridge University Press Cambridge, UK
3. C.M. Bishop Pattern Recognition and Machine Learning 2006 Springer Singapore
4. M.E. Tipping Sparse bayesian learning and the relevance vector machine Journal of Machine Learning Research 1 2001 211 244
5. A. Verikas, A. Lipnickas, K. Malmqvist, M. Bacauskiene, and A. Gelzinis Soft combination of neural classifiers: a comparative study Pattern Recognition Letters 20 1999 429 444
6. L. Breiman, J.H. Friedman, R.A. Olshen, and C.J. Stone Classification and Regression Trees 1993 Chapman & Hall
7. I.H. Witten, and E. Frank Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations 1999 Morgan Kaufmann Los Altos, CA 〈 http://www.cs.waikato.ac.nz/ml/weka 〉
8. L. Breiman Bagging predictors Machine Learning 24 2 1996 123 140
9. T.G. Dietterich An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization Machine Learning 40 2 2000 139 157
10. L. Breiman Random forests Machine Learning 45 2001 5 32
11. G. Biau, L. Devroye, and G. Lugosi Consistency of random forests and other averaging classifiers Journal of Machine Learning Research 9 2008 2015 2033
12. T.K. Ho The random subspace method for constructing decision forests IEEE Transactions on Pattern Analysis and Machine Intelligence 20 8 1998 832 844
13. G. Nimrod, A. Szilagyi, C. Leslie, and N. Ben-Tal Identification of DNA-binding proteins using structural, electrostatic and evolutionary features Journal of Molecular Biology 387 4 2009 1040 1053
14. H.T. Chen, T.L. Liu, and C.S. Fuh Segmenting highly articulated video objects with weak-prior random forests A. Leonardis, H. Bischof, A. Pinz, ECCV 2006, Part IV, Lecture Notes in Computer Science vol. 3954 2006 Springer-Verlag Berlin, Heidelberg 373 385
15. J. Ham, Y. Chen, M.M. Crawford, and J. Ghosh Investigation of the random forest framework for classification of hyperspectral data IEEE Transactions on Geoscience and Remote Sensing 43 3 2005 492 501
16. M.M. Crawford, J. Ham, Y. Chen, and J. Ghosh Random forests of binary hierarchical classifiers for analysis of hyperspectral data 2003 IEEE Workshop on Advances in Techniques for Analysis of Remotely Sensed Data 2004 IEEE Greenbelt, MD, USA 337 345
17. J. Peters, B. De Baets, N.E.C. Verhoest, R. Samson, S. Degroeve, P. De Becker, and W. Huybrechts Random forests as a tool for ecohydrological distribution modelling Ecological Modelling 207 2007 304 318
18. J. Peters, N.E.C. Verhoest, R. Samson, M. Van Meirvenne, L. Cockx, and B. De Baets Uncertainty propagation in vegetation distribution models based on ensemble classifiers Ecological Modelling 220 2009 791 804
19. P.A. Hernandez, I. Franke, S.K. Herzog, V. Pacheco, L. Paniagua, H.L. Quintana, A. Soto, J.J. Swenson, C. Tovar, T.H. Valqui, J. Vargas, and B.E. Young Predicting species distributions in poorly-studied landscapes Biodiversity and Conservation 17 2008 1353 1366
20. I. Oparin, O. Glembek, L. Burget, and J. Cernocky Morphological random forests for language modeling of inflectional languages 2008 IEEE Workshop on Spoken Language Technology, SLT 2008 2008 IEEE Goa, India 189 192
21. J. Ramirez, J.M. Gorriz, R. Chaves, M. Lopez, D. Salas-Gonzalez, I. Alvarez, and F. Segovia SPECT image classification using random forests Electronics Letters 45 12 2009 604 605
22. A.G. Heidema, J.M.A. Boer, N. Nagelkerke, E.C.M. Mariman, D.L. van der A, and E.J.M. Feskens The challenge for genetic epidemiologists: how to analyze large numbers of SNPs in relation to complex diseases Accident Analysis and Prevention 7 23 2006 1 15
23. P. Han, X. Zhang, R.S. Norton, and Z.P. Feng Large-scale prediction of long disordered regions in proteins using random forests BMC Bioinformatics 10 8 2009 1 9
24. J.D. Watts, R.L. Lawrence, P.R. Miller, and C. Montagne Monitoring of cropland practices for carbon sequestration purposes in north central montana by landsat remote sensing Remote Sensing of Environment 113 2009 1843 1852
25. S.R. Joelsson, J.A. Benediktsson, and J.R. Sveinsson Feature selection for morphological feature extraction using random forests Seventh Nordic Signal Processing Symposium 2006 IEEE New York, Reykjavik, Iceland 138 141
26. S. Gupta, S. Matthew, P.M. Abreu, and J.A. de Sousa QSAR analysis of phenolic antioxidants using MOLMAP descriptors of local properties Bioorganic & Medicinal Chemistry 14 2006 1199 1206
27. S. Bernard, L. Heutte, and S. Adam Using random forests for handwritten digit recognition ICDAR 2007: Ninth International Conference on Document Analysis and Recognition vol. 12 2007 Curitiba Brazil 1043 1047
28. M. Thums, C.J.A. Bradshaw, and M.A. Hindell A validated approach for supervised dive classification in diving vertebrates Journal of Experimental Marine Biology and Ecology 363 2008 75 83
29. A. Liaw, and M. Wiener Classification and regression by random forest R News 2 3 2002 18 22
30. L. Breiman, RFtoolsfor predicting and understanding data, Technical Report, Berkeley University, Berkeley, USA 〈 http://oz.berkeley.edu/users/ breiman/RandomForests/cc.papers.htm 〉, 2004.
31. V. Svetnik, A. Liaw, C. Tong, J.C. Culberson, R.P. Sheridan, and B.P. Feuston Random forest: a classification and regression tool for compound classification and QSAR modeling Journal of Chemical Information and Computer Sciences 43 6 2003 1947 1958
32. V. Svetnik, T. Wang, C. Tong, A. Liaw, R.P. Sheridan, and Q.H. Song Boosting: an ensemble learning tool for compound classification and QSAR modeling Journal of Chemical Information and Modeling 45 3 2005 786 799
33. A. Folleco, T.M. Khoshgoftaar, J. Van Hulse, and L. Bullard Software quality modeling: the impact of class noise on the random forest classifier 2008 IEEE Congress on Evolutionary Computation vol. 18 2008 IEEE Hong Kong, PR China 3853 3859
34. A. Statnikov, L. Wang, and C.F. Aliferis A comprehensive comparison of random forests and support vector machines for microarray-based cancer classification BMC Bioinformatics 9 319 2008 1 10
35. R.O. Duda, P.E. Hart, and D.G. Stork Pattern Classification second ed. 2001 John Wiley & Sons New York
36. T.K. Ho A data complexity analysis of comparative advantages of decision forest constructors Pattern Analysis and Applications 5 2002 102 112
37. T.K. Ho, and M. Basu Complexity measures of supervised classification problems IEEE Transactions on Pattern Analysis and Machine Intelligence 24 3 2002 289 300
38. D. Meyer, F. Leisch, and K. Hornik The support vector machine under test Neurocomputing 55 12 2003 169 186
39. R.E. Banfield, L.O. Hall, K.W. Bowyer, and W.P. Kegelmeyer A comparison of decision tree ensemble creation techniques IEEE Transactions on Pattern Analysis and Machine Intelligence 29 1 2007 173 180
40. Y. Freund, and R.E. Schapire Experiments with a new boosting algorithm Proceedings of the 13th International Conference on Machine Learning 1996 148 156
41. E. Alpaydin Combined 5×2 cv F test for comparing supervised classification learning algorithms Neural Computation 11 8 1999 1885 1892
42. J. Demsar Statistical comparisons of classifiers over multiple data sets Journal of Machine Learning Research 7 2006 1 30 (Pubitemid 43022939)
43. R. Diaz-Uriarte, and S. Alvarez de Andres Gene selection and classification of microarray data using random forest BMC Bioinformatics 7 3 2006 1 13
44. V. Sindhwani, P. Bhattacharyya, and S. Rakshit Information theoretic feature crediting in multiclass support vector machines R. Grossman, V. Kumar, Proceedings of the First SIAM International Conference on Data Mining 2001 SIAM Chicago, USA 1 18
45. Y. Xie, X. Li, E.W.T. Ngai, and W. Ying Customer churn prediction using improved balanced random forests Expert Systems with Applications 36 2009 5445 5449
46. W. Ying, X. Li, Y. Xie, and E. Johnson Preventing customer churn by using random forests modeling IEEE International Conference on Information Reuse and Integration (IRI-2008) 2008 IEEE Las Vegas, USA 429 434
47. K. Coussement, and D. Van den Poel Churn prediction in subscription services: an application of support vector machines while comparing two parameter-selection techniques Expert Systems with Applications 34 2008 313 327
48. K. Coussement, and D. Van den Poel Improving customer attrition prediction by integrating emotions from client/company interaction emails and evaluating multiple classifiers Expert Systems with Applications 36 2009 6127 6134
49. C. Whitrow, D.J. Hand, P. Juszczak, D. Weston, and N.M. Adams Transaction aggregation as a strategy for credit card fraud detection Data Mining and Knowledge Discovery 18 1 2009 30 55
50. D. Perdiguero-Alonso, F.E. Montero, A. Kostadinova, J.A. Raga, and J. Barrett Random forests, a novel approach for discrimination of fish populations using parasites as biological tags International Journal for Parasitology 38 2008 1425 1434
51. I. Koprinska, J. Poon, J. Clark, and J. Chan Learning to classify e-mail Information Sciences 177 2007 2167 2187
52. C. Manning, and H. Schtze Foundations of Statistical Natural Language Processing 1999 MIT Press Cambridge, MA, USA
53. W. Yan Application of random forest to aircraft engine fault diagnosis IMACS Multiconference on Computational Engineering in Systems Applications (CESA) 2006 IEEE Beijing, PR China 468 475
54. B. Slabbinck, B. De Baets, P. Dawyndt, and P. De Vos Towards large-scale FAME-based bacterial species identification using machine learning techniques Systematic and Applied Microbiology 32 2009 163 176
55. A.Z. Kouzani Faceparts for recognition TENCON 20062006 IEEE Region 10 Conference 2006 IEEE Hong Kong, PR China 1232 1235
56. B.S. Yang, X. Di, and T. Han Random forests classifier for machine fault diagnosis Journal of Mechanical Science and Technology 22 2008 1716 1725
57. A.Z. Kouzani Subcellular localisation of proteins in fluorescent microscope images using a random forest 2008 IEEE International Joint Conference on Neural Networks vol. 18 2008 IEEE Hong Kong, PR China 3926 3932
58. G. Zhang, H. Li, and B. Fang Discriminating acidic and alkaline enzymes using a random forest model with secondary structure amino acid composition Process Biochemistry 44 2009 654 660
59. M.B. Garzon, R. Blazek, M. Neteler, R.S. de Dios, H.S. Ollero, and C. Furlanello Predicting habitat suitability with machine learning models: the potential area of Pinus sylvestris L. in the Iberian peninsula Ecological Modelling 197 2006 383 393
60. A.M. Prasad, L.R. Iverson, and A. Liaw Newer classification and regression tree techniques: bagging and random forests for ecological prediction Ecosystems 9 2006 181 199
61. S. Gupta, and J. Aires-deSousa Comparing the chemical spaces of metabolites and available chemicals: models of metabolite-likeness Molecular Diversity 11 1 2007 23 36
62. S. Tognazzo, B. Emanuela, F.A. Rita, G. Stefano, M. Daniele, S.C. Fiorella, and Z. Paola Probabilistic classifiers and automated cancer registration: an exploratory application Journal of Biomedical Informatics 42 2009 1 10
63. D. Kocev, S. Dzeroski, M.D. White, G.R. Newell, and P. Griffioen Using single- and multi-target regression trees and ensembles to model a compound index of vegetation condition Ecological Modelling 220 2009 1159 1168
64. K. Marsolo, M. Twa, M.A. Bullimore, and S. Parthasarathy The support vector machine under test IEEE Transactions on Information Technology in Biomedicine 11 2 2007 203 212
65. W. Buckinx, and D. Van den Poel Customer base analysis: partial defection of behaviourally loyal clients in a non-contractual FMCG retail setting European Journal of Operational Research 164 2005 252 268
66. R. Tibshirani, T. Hastie, B. Narasimhan, and G. Chu Class prediction by nearest shrunken centroids, with applications to DNA microarrays Statistical Science 18 1 2003 104 117
67. D.J. MacKay Bayesian interpolation Neural Computation 4 1992 415 447
68. D.J.C. MacKay The evidence framework applied to classification networks Neural Computation 4 5 1992 720 736
69. T. Luo, K. Kramer, D.B. Goldgof, L.O. Hall, S. Samson, A. Remsen, and T. Hopkins Recognizing plankton images from the shadow image particle profiling evaluation recorder IEEE Transactions on Systems, Man, and CyberneticsPart B: Cybernetics 34 4 2004 1753 1762
70. V. Belle, T. Deselaers, S. Schiffer, Randomized trees for real-time one-step face detection and recognition, in: 19th International Conference on Pattern Recognition, vol. 16, Tampa, FL, USA, 2008, pp. 35473550.
71. J.A. Benediktsson, J. Chanussot, and M. Fauvel Multiple classifier systems in remote sensing: from basics to recent developments M. Haindl, J. Kittler, F. Roli, Multiple Classifier Systems, Lecture Notes in Computer Science vol. 4471 2007 Springer-Verlag Berlin, Heidelberg 501 512
72. S. Abu-Nimeh, D. Nappa, X. Wang, and S. Nair Bayesian additive regression trees-based spam detection for enhanced email privacy The Third International Conference on Availability, Reliability and Security 2008 IEEE Computer Society Barcelona, Spain 1044 1051
73. P.M. Granitto, F. Biasioli, E. Aprea, D. Mott, C. Furlanello, T.D. Mark, and F. Gasperi Rapid and non-destructive identification of strawberry cultivars by direct PTR-MS headspace analysis and data mining techniques Sensors and Actuators B 121 2007 379 385
74. D. Donald, D. Coomans, Y. Everingham, D. Cozzolino, M. Gishen, and T. Hancock Adaptive wavelet modelling of a nested 3 factor experimental design in NIR chemometrics Chemometrics and Intelligent Laboratory Systems 82 2006 122 129
75. J.R. Jang ANFIS: adaptive-network-based fuzzy inference system IEEE Transactions on Systems, Man, and Cybernetics 23 1993 665 685
76. A. Zainal, M.A. Maarof, S.M. Shamsuddin, and A. Abraham Ensemble of one-class classifiers for network intrusion detection system Fourth International Symposium on Information Assurance and Security 2008 IEEE Computer Society Napoli, Italy 180 185
77. L.I. Kuncheva, V.J. del Rio Vilas, and J.J. Rodriguez Diagnosing scrapie in sheep: a classification experiment Computers in Biology and Medicine 37 2007 1194 1202
78. A. Brenning Benchmarking classifiers to optimally integrate terrain analysis and multispectral remote sensing in automatic rock glacier detection Remote Sensing of Environment 113 1 2009 239 247
79. W. Buckinx, G. Verstraeten, and D. Van den Poel Predicting customer loyalty using the internal transactional database Expert Systems with Applications 32 2007 125 134
80. T. Hancock, R. Put, D. Coomans, Y.V. Heyden, and Y. Everingham A performance comparison of modern statistical techniques for molecular descriptor selection and retention prediction in chromatographic QSRR studies Chemometrics and Intelligent Laboratory Systems 76 2005 185 196
81. C. Strobl, A.L. Boulesteix, A. Zeileis, and T. Hothorn Bias in random forest variable importance measures: illustrations, sources and a solution BMC Bioinformatics 8 25 2007 1 21
82. D.M. Reif, A.A. Motsinger, B.A. McKinney, J.E. Crowe, and J.H. Moore Feature selection using a random forests classifier for the integrated analysis of multiple data types Proceedings of the 2006 IEEE Symposium on Computational Intelligence in Bioinformatics and Computational Biology 2006 IEEE Toronto, Canada 171 178
83. K.J. Archer, and R.V. Kimes Empirical characterization of random forest variable importance measures Computational Statistics & Data Analysis 52 4 2008 2249 2260
84. O. Okun, and H. Priisalu Random forest for gene expression based cancer classification: overlooked issues J. Marti, IbPRIA 2007, Part II, Lecture Notes in Computer Science vol. 4478 2007 Springer-Verlag Berlin, Heidelberg 483 490
85. R. Harb, X. Yan, E. Radwan, and X. Su Exploring precrash maneuvers using classification trees and random forests Accident Analysis and Prevention 41 2009 98 107
86. J. Fan, M.E. Nunn, and X. Su Multivariate exponential survival trees and their application to tooth prognosis Computational Statistics and Data Analysis 53 2009 1110 1121
87. J. Zhang, M. Zulkernine, and A. Haque Random-forests-based network intrusion detection systems IEEE Transactions on Systems, Man, and CyberneticsPart C: Applications and Reviews 38 5 2008 649 659
88. J. Zhang, and M. Zulkernine A hybrid network intrusion detection technique using random forests Proceedings of the First International Conference on Availability, Reliability and Security (ARES'06) 2006 IEEE Computer Society Vienna, Austria 262 269
89. B.H. Menze, W. Petrich, and F.A. Hamprecht Multivariate feature selection and hierarchical classification for infrared spectroscopy: serum-based detection of bovine spongiform encephalopathy Analytical and Bioanalytical Chemistry 387 5 2007 1801 1807
90. D. Donald, T. Hancock, D. Coomans, and Y. Everingham Bagged super wavelets reduction for boosted prostate cancer classification of seldi-tof mass spectral serum profiles Chemometrics and Intelligent Laboratory Systems 82 2006 2 7
91. B. Lariviere, and D. Van den Poel Predicting customer retention and profitability by using random forests and regression forests techniques Expert Systems with Applications 29 2005 472 484
92. L.C. Keely, and C.M. Tan Understanding preferences for income redistribution Journal of Public Economics 92 2008 944 961
93. K.L. Lunetta, L.B. Hayward, J. Segal, and P. Van Eerdewegh Screening large-scale association study data: exploiting interactions using random forests BMC Genetics 5 32 2004 1 13
94. K.Q. Shen, C.J. Ong, X.P. Li, Z. Hui, and E.P.V. Wilder-Smith A feature selection method for multilevel mental fatigue EEG classification IEEE Transactions on Biomedical Engineering 54 7 2007 1231 1237
95. H. Wang, C. Lin, F. Yang, and X. Hu Hedged predictions for traditional Chinese chronic gastritis diagnosis with confidence machine Computers in Biology and Medicine 39 2009 425 432
96. F. Yang, H.Z. Wang, and H. Mi A novel classification method of microarray with reliability and confidence Proceedings of the Seventh International Conference on Machine Learning and Cybernetics vol. 17 2008 IEEE Kunming, PR China 1726 1733
97. S.M. Wiseman, A. Melck, H. Masoudi, F. Ghaidi, L. Goldstein, A. Gown, S.J.M. Jones, and O.L. Griffith Molecular phenotyping of thyroid tumors identifies a marker panel for differentiated thyroid cancer diagnosis Annals of Surgical Oncology 15 10 2008 2811 2826
98. G.W. Corder, and D.I. Foreman Nonparametric Statistics for Non-Statisticians: A Step-by-Step Approach 2009 Wiley New Jersey
99. S.H. Wu, B.N. Lee, L.C. Wang, and M.S. Abadir Statistical analysis and optimization of parametric delay test IEEE International Test Conference vol. 12 2007 IEEE Santa Clara, CA, USA 613 622
100. P.O. Gislason, J.A. Benediktsson, and J.R. Sveinsson Random forests for land cover classification Pattern Recognition Letters 27 2006 294 300
101. J.B. Gray, and G. Fan Classification tree analysis using TARGET Computational Statistics & Data Analysis 52 2008 1362 1372
102. J.J. Rodrguez, and C.J. Alonso Rotation-based ensembles Current Topics in Artificial Intelligence, Lecture Notes in Computer Science vol. 3040 2003 Springer-Verlag 498 506
103. J.L. Rodrguez, L.I. Kuncheva, and C.J. Alonso Rotation forest: a new classifier ensemble method IEEE Transactions on Pattern Analysis and Machine Intelligence 28 10 2006 1619 1630
104. A. Tsymbal, M. Pechenizkiy, and P. Cunningham Dynamic integration with random forests J. Frnkranz, T. Scheffer, M. Spiliopoulou, ECML 2006, Lecture Notes in Artificial Intelligence vol. 4212 2006 Springer-Verlag Berlin, Heidelberg 801 808
105. E.E. Tripoliti, D.I. Fotiadis, and M. Argyropoulou An automated supervised method for the diagnosis of Alzheimer's disease based on fMRI data using weighted voting schemes IEEE International Workshop on Imaging Systems and TechniquesIST 2008 2008 IEEE Chania, Greece 338 343
106. G. Leshem, Y. Ritov, Traffic flow prediction using adaboost algorithm with random forests as a weak learner, in: Proceedings of World Academy of Science, Engineering and Technology, vol. 19, Bangkok, Thailand, 2007, pp. 193198.
107. H.E. Osman Online random forests based on CorrFS and CorrBE 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2008 IEEE Anchorage, AK, USA 863 869
108. A. Verikas, and M. Bacauskiene Feature selection with neural networks Pattern Recognition Letters 23 11 2002 1323 1335
109. A. Verikas, M. Bacauskiene, D. Valincius, and A. Gelzinis Predictor output sensitivity and feature similarity-based feature selection Fuzzy Sets & Systems 159 2008 422 434
110. J.C. Platt Fast training of support vector machines using sequential minimal optimization B. Scholkopf, C.J.C. Burges, A.J. Smola, Advances in Kernel Methods: Support Vector Learning 1998 MIT Press Cambridge, MA 185 208
111. T. Luo, K. Kramer, D.B. Goldgof, L.O. Hall, A.R.S. Samson, and T. Hopkins Active learning to recognize multiple types of plankton Journal of Machine Learning Research 6 2005 589 613
112. S.J. Wang, A. Mathew, Y. Chen, L.F. Xi, L. Mab, and J. Lee Empirical analysis of support vector machine ensemble classifiers Expert Systems with Applications 36 2009 6466 6476