Decision forest: Twenty years of research

Information Fusion

Volumn 27, Issue , 2016, Pages 111-125

  Rokach, Lior ^a  

^a BEN GURION UNIVERSITY OF THE NEGEV   (Israel)

Author keywords

Classification tree; Decision forest; Decision tree; Random forest

Indexed keywords

LEARNING ALGORITHMS;

CLASSIFICATION TREES; DECISION FOREST; PREDICTION FUNCTION; PREDICTIVE MODELING; PREDICTIVE PERFORMANCE; RANDOM FORESTS; SINGLE DECISION; SURVIVAL ANALYSIS;

DECISION TREES;

EID: 84933574325     PISSN: 15662535     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.inffus.2015.06.005     Document Type: Article

References (119)

1. M. Fernández-Delgado, E. Cernadas, S. Barro, and D. Amorim Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res. 15 1 2014 3133 3181
2. P. Geurts, D. Ernst, and L. Wehenkel Extremely randomized trees Mach. Learn. 63 1 2006 3 42
3. L.I. Kuncheva Combining Pattern Classifiers Methods and Algorithms second ed. 2014 Wiley Hoboken, NJ
4. M. Woźniak, M. Graña, and E. Corchado A survey of multiple classifier systems as hybrid systems Inform. Fusion 16 2014 3 17
5. A. Verikas, A. Gelzinis, and M. Bacauskiene Mining data with random forests: a survey and results of new tests Pattern Recogn. 44 2011 330 349
6. V.Y. Kulkarni, and P.K. Sinha Random forest classifiers: a survey and future research directions Int. J. Adv. Comput. 36 1 2013 1144 1153
7. A. Criminisi, and J. Shotton Decision Forests for Computer Vision and Medical Image Analysis 2013 Springer Science & Business Media
8. R.A. Fisher The use of multiple measurements in taxonomic problems Ann. Eugenic. 7 2 1936 179 188 10.1111/j.1469-1809.1936.tb02137.x
9. J.R. Quinlan C4.5: Programs for Machine Learning 1993 Morgan Kaufmann Publishers
10. L. Breiman, J.H. Friedman, R.A. Olshen, and C.J. Stone Classification and regression trees 1984 Wadsworth & Brooks/Cole Advanced Books & Software Monterey, CA 978-0-412-04841-8
11. I.H. Witten, E. Frank, and M.A. Hall Data Mining: Practical Machine Learning Tools and Techniques 3rd ed. 2011 Morgan Kaufmann San Francisco
12. T.M. Therneau, E.J. Atkinson, An Introduction to Recursive Partitioning Using the rpartRoutine, Technical Report 61, Section of Biostatistics, Mayo Clinic, Rochester, 1997. < http://www.mayo.edu/hsr/techrpt/61.pdf >.
13. S.A. Moore, D.M. Daddario, J. Kurinskas, G.M. Weiss, Are decision trees always greener on the open (source) side of the fence?, in: Proceedings of DMIN, 2009, pp. 185-188.
14. R.C. Barros, André C.P.L.F. de Carvalho, and Alex A. Freitas Automatic Design of Decision-Tree Induction Algorithms 2015 Springer
15. G. Pagallo, and D. Huassler Boolean feature discovery in empirical learning Mach. Learn. 5 1 1990 71 99
16. S. Murthy, S. Salzberg, Lookahead and pathology in decision tree induction, in: IJCAI, 1995, pp. 1025-1033.
17. S. Esmeir, and S. Markovitch Lookahead-based algorithms for anytime induction of decision trees Proceedings of the Twenty-First International Conference on Machine Learning (ICML '04) 2004 ACM New York, NY, USA 33 41
18. I. Bou-Hamad, D. Larocque, and H. Ben-Ameur A review of survival trees Stat. Surv. 5 2011 44 71
19. A. Eyal, L. Rokach, M. Kalech, O. Amir, R. Chougule, R. Vaidyanathan, and K. Pattada Survival analysis of automobile components using mutually exclusive forests IEEE Trans. Syst. Man Cybernet.: Syst. 44 2 2014 246 253
20. H. Blockeel, L. De Raedt, and J. Ramon Top-down induction of clustering trees Proceedings of the 15th International Conference on Machine Learning 1998 Morgan Kaufmann 55 63
21. F. Moosmann, E. Nowak, and F. Jurie Randomized clustering forests for image classification IEEE Trans. Pattern Anal. Mach. Intell. 30 9 2008 1632 1646
22. M. Dror, A. Shabtai, L. Rokach, and Y. Elovici OCCT: a one-class clustering tree for implementing one-to-many data linkage IEEE Trans. Knowl. Data Eng. 26 3 2014 682 697
23. A. Gershman, A. Meisels, K.H. Lüke, L. Rokach, A. Schclar, A. Sturm, A Decision Tree Based Recommender System, in: IICS, 2010, pp. 170-179.
24. L. Rokach, and S. Kisilevich Initial profile generation in recommender systems using pairwise comparison IEEE Trans. Syst. Man Cybernet. Part C: Appl. Rev. 42 6 2012 1854 1859
25. F. Xia, W. Zhang, F. Li, and Y. Yang Ranking with decision tree Knowl. Inform. Syst. 17 3 2008 381 395
26. K.M. Svore, and C.J. Burges Large-scale learning to rank using boosted decision trees Scaling Up Machine Learning 2011 Cambridge U. Press
27. Y. Freund, R. Iyer, R.E. Schapire, and Y. Singer An efficient boosting algorithm for combining preferences J. Mach. Learn. Res. 4 2003 933 969
28. L. Antwarg, L. Rokach, and B. Shapira Attribute-driven hidden markov model trees for intention prediction IEEE Trans. Syst. Man Cybernet. Part C 42 6 2012 1103 1119
29. R. Polikar Ensemble based systems in decision making IEEE Circ. Syst. Mag. 6 3 2006 21 45
30. J.W. Tukey Exploratory Data Analysis 1977 Addison-Wesley Reading, Mass
31. B.V. Dasarathy, and B.V. Sheela Composite classifier system design: concepts and methodology Proc. IEEE 67 5 1979 708 713
32. L.A. Clark, and D. Pergibon Tree-based models J.M. Chambers, T.J. Hastie, Statistical Models in S 1992 Wadsworth & Brooks Pacific Grove, CA
33. Y. Freund, R.E. Schapire, Experiments with a new boosting algorithm, in: Machine Learning: Proceedings of the Thirteenth International Conference, 1996, pp. 325-332.
34. N.C. Oza, and K. Tumer Classifier ensembles: select real-world applications Inform. Fusion 9 1 2008 4 20
35. T.G. Dietterich Ensemble learning M.A. Arbib, The Handbook of Brain Theory and Neural Networks second ed. 2002 The MIT Press Cambridge
36. T.G. Dietterich, E.B. Kong, Machine Learning Bias, Statistical Bias, and Statistical Variance of Decision Tree Algorithms, Tech. Rep., Oregon State University, 1995.
37. D.A. Cieslak, and N.V. Chawla Learning decision trees for unbalanced data Machine Learning and Knowledge Discovery in Databases 2008 Springer Berlin, Heidelberg 241 256
38. V. Nikulin, G. McLachlan, and S. Ng Ensemble approach for classification of imbalanced data Proceedings of the 22nd Australian Joint Conference on Advances in Artificial Intelligence 2009 Springer-Verlag
39. M. Galar, A. Fernandez, E. Barrenechea, H. Bustince, and F. Herrera A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches IEEE Trans. Syst. Man Cybernet. Part C: Appl. Rev. 42 4 2012 463 484
40. M. Galar, A. Fernández, E. Barrenechea, and F. Herrera Eusboost: enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling Pattern Recogn. 46 12 2013 3460 3471
41. J. Gama, R. Fernandes, and R. Rocha Decision trees for mining data streams Intell. Data Anal. 10 1 2006 23 45
42. J.Z. Kolter, M. Maloof, Dynamic weighted majority: a new ensemble method for tracking concept drift, in: Third IEEE International Conference on Data Mining, 2003, ICDM 2003, 2003, pp. 123-130.
43. L.L. Minku, A.P. White, and X. Yao The impact of diversity on online ensemble learning in the presence of concept drift IEEE Trans. Knowl. Data Eng. 22 5 2010 730 742
44. L.L. Minku, and X. Yao DDD: a new ensemble approach for dealing with concept drift IEEE Trans. Knowl. Data Eng. 24 4 2012 619 633
45. R. Bryll, R. Gutierrez-Osuna, and F. Quek Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets Pattern Recogn. 36 6 2003 1291 1302
46. J. Huang, H. Fang, and X. Fan Decision forest for classification of gene expression data Comput. Biol. Med. 40 8 2010 698 704
47. L. Rokach Genetic algorithm-based feature set partitioning for classification problems Pattern Recogn. 41 5 2008 1676 1700
48. O. Maimon, and L. Rokach Data mining by attribute decomposition with semiconductor manufacturing case study Data Mining for Design and Manufacturing 2001 Springer US 311 336
49. C. Vens, J. Struyf, L. Schietgat, S. Džeroski, and H. Blockeel Decision trees for hierarchical multi-label classification Mach. Learn. 73 2 2008 185 214
50. H. Deng, G. Runger, E. Tuv, and M. Vladimir A time series forest for classification and feature extraction Inform. Sci. 239 2013 142 153
51. T. Mitchell, The Need for Biases in Learning Generalizations, Technical Report CBM-TR-117, Rutgers University, Department of Computer Science, New Brunswick, NJ, 1980.
52. Y. Bi The impact of diversity on the accuracy of evidential classifier ensembles Int. J. Approx. Reason. 53 4 2012 584 607
53. K.M. Ali, M.J. Pazzani, On the link between error correlation and error reduction in decision tree ensembles, Information and Computer Science, University of California, Irvine, 1995, pp. 95-38.
54. P.K. Chan, S.J. Stolfo, A comparative evaluation of voting and meta-learning on partitioned data, in: Proceedings of the 12th International Conference on Machine Learning ICML-95, 1995.
55. G. Brown, J.L. Wyatt, and P. Tivno Managing diversity in regression ensembles J. Mach. Learn. Res. 6 2005 1621 1650
56. S.W. Lin, and S.C. Chen Parameter determination and feature selection for C4.5 algorithm using scatter search approach Soft Comput. 16 1 2012 63 75
57. N.V. Chawla, L.O. Hall, K.W. Bowyer, and W.P. Kegelmeyer Learning ensembles from bites: a scalable and accurate approach J. Mach. Learn. Res. Arch. 5 2004 421 451
58. K.M. Ting, J.R. Wells, S.C. Tan, S.W. Teng, and G.I. Webb Feature-subspace aggregating: ensembles for stable and unstable learners Mach. Learn. 82 3 2011 375 397
59. M. Gashler, C. Giraud-Carrier, T. Martinez, Decision tree ensemble: small heterogeneous is better than large homogeneous, in: IEEE Seventh International Conference on Machine Learning and Applications, 2008, ICMLA'08, 2008, pp. 900-905.
60. C.X. Zhang, and J.S. Zhang RotBoost: a technique for combining Rotation Forest and AdaBoost Pattern Recogn. Lett. 29 10 2008 1524 1536
61. S. Bernard, L. Heutte, and S. Adam Dynamic random forests Pattern Recogn. Lett. 33 12 2012 1580 1586
62. M.F.A. Hady, F. Schwenker, and G. Palm Semi-supervised learning for tree-structured ensembles of RBF networks with co-training Neural Networks 23 4 2010 497 509
63. P. Clark, R. Boswell, Rule induction with CN2: some recent improvements, in: Proceedings of the European Working Session on Learning, Pitman, 1991, pp. 151-163.
64. D.W. Opitz, and J.W. Shavlik Actively searching for an effective neural network ensemble Connect. Sci. 8 3-4 1996 337 354
65. W. Buntine, A Theory of Learning Classification Rules, Doctoral Dissertation, School of Computing Science, University of Technology, Sydney. Australia, 1990.
66. R.E. Shapire, and Y. Singer Improved boosting algorithms including confidence-rated predictions Mach. Learn. 37 1999 297 336
67. P. Derbeko, R. El-Yaniv, R. Meir, Variance optimized bagging, in: European Conference on Machine Learning, 2002.
68. A.K. Seewald, Towards Understanding Stacking, PhD Thesis, Vienna University of Technology, 2003.
69. E. Menahem, L. Rokach, and Y. Elovici Troika-an improved stacking schema for classification tasks Inform. Sci. 179 24 2009 4097 4122
70. A.K. Seewald, J. Faurnkranz, Grading Classifiers, Austrian Research Institute for Artificial intelligence, 2001.
71. A. Omari, A.R. Figueiras-Vidal, Post-aggregation of classifier ensembles, Inform. Fusion (2015) (in press). http://dx.doi.org/10.1016/j.inffus.2015.01.003.
72. L. Breiman Bagging predictors Mach. Learn. 24 2 1996 123 140
73. L. Breiman Random forests Mach. Learn. 45 1 2001 5 32
74. T.K. Ho, Random decision forests, in: Proceedings of the Third International Conference on Document Analysis and Recognition, vol. 1, 1995, pp. 278-282.
75. Y. Amit, D. Geman, Randomized Inquiries About Shape: An Application to Handwritten Digit Recognition (No. TR-401). CHICAGO UNIV IL DEPT OF STATISTICS, 1994.
76. T.G. Dietterich An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization Mach. Learn. 40 2 2000 139 157
77. A. Saffari, C. Leistner, J. Santner, M. Godec, H. Bischof, On-line random forests, in: 2009 IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops), 2009, pp. 1393-1400.
78. T.K. Ho The random subspace method for constructing decision forests IEEE Trans. Pattern Anal. Mach. Intell. 20 8 1998 832 844
79. C. Kamath, E. Cantu-Paz, Creating ensembles of decision trees through sampling, in: Proceedings, 33-rd Symposium on the Interface of Computing Science and Statistics, Costa Mesa, CA, June 2001.
80. C. Kamath, E. Cantu-Paz, D. Littau, Approximate splitting for ensembles of trees using histograms, in: Second SIAM International Conference on Data Mining (SDM-2002), 2002.
81. C. Désir, S. Bernard, C. Petitjean, and L. Heutte One class random forests Pattern Recogn. 46 12 2013 3490 3506
82. J.J. Rodriguez, L.I. Kuncheva, and C.J. Alonso Rotation forest: a new classifier ensemble method IEEE Trans. Pattern Anal. Mach. Intell. 28 10 2006 1619 1630
83. A. Schclar, and L. Rokach Random projection ensemble classifiers Enterprise Information Systems 2009 Springer Berlin, Heidelberg 309 316
84. J.H. Friedman Greedy function approximation: a gradient boosting machine Ann. Stat. 2001 1189 1232
85. P. Li, Abc-boost: adaptive base class boost for multi-class classification, in: ACM Proceedings of the 26th Annual International Conference on Machine Learning, 2009, pp. 625-632.
86. G. Martínez-Muñoz, and A. Suárez Using all data to generate decision tree ensembles IEEE Trans. Syst. Man Cyber. C 34 4 2004 393 397
87. L. Breiman Randomizing outputs to increase prediction accuracy Mach. Learn. 40 3 2000 229 242
88. G. Martínez-Muñoz, and A. Suárez Switching class labels to generate classification ensembles Pattern Recogn. 38 10 2005 1483 1494
89. E. Bauer, and R. Kohavi An empirical comparison of voting classification algorithms: bagging, boosting, and variants Mach. Learn. 35 1999 1 38
90. D. Opitz, and R. Maclin Popular ensemble methods: an empirical study J. Artif. Res. 11 1999 169 198
91. D. Villalba Santiago, J. RodrHebrew letter MEMguez Juan, J. Alonso Carlos, An empirical comparison of boosting methods via OAIDTB, an extensible java class library, in: II International Workshop on Practical Applications of Agents and Multiagent Systems - IWPAAMS'2003.
92. R.E. Banfield, L.O. Hall, K.W. Bowyer, and W.P. Kegelmeyer A comparison of decision tree ensemble creation techniques IEEE Trans. Pattern Anal. Mach. Intell. 29 1 2007 173 180
93. J. Shafer, R. Agrawal, M. Mehta, SPRINT: a scalable parallel classifier for data mining, in: Proceedings of 1996 International Conference Very Large Data Bases, 1996, pp. 544-555.
94. M. Mehta, R. Agrawal, and J. Rissanen SLIQ: a fast scalable classifier for data mining Advances in Database Technology - EDBT'96 1996 Springer Berlin, Heidelberg 18 32
95. P. He, L. Chen, X.H. Xu, Fast C4.5, in: 2007 International Conference on IEEE Machine Learning and Cybernetics, vol. 5, 2007, pp. 2841-2846.
96. B. Panda, J.S. Herbach, S. Basu, and R.J. Bayardo Planet: massively parallel learning of tree ensembles with mapreduce Proc. VLDB Endowment 2 2 2009 1426 1437
97. J. Ye, J.H. Chow, J. Chen, Z. Zheng, Stochastic gradient boosted distributed decision trees, in: Proceedings of the 18th ACM Conference on Information and Knowledge Management, 2009, pp. 2061-2064.
98. I. Palit, and C.K. Reddy Scalable and parallel boosting with mapreduce IEEE Trans. Knowl. Data Eng. 24 10 2012 1904 1916
99. S. del Río, V. López, J.M. Benítez, and F. Herrera On the use of MapReduce for imbalanced big data using Random Forest Inform. Sci. 285 2014 112 137
100. G. Tsoumakas, I. Partalas, I. Vlahavas, A taxonomy and short review of ensemble selection, in: Workshop on Supervised and Unsupervised Ensemble Methods and Their Applications, 2008.
101. D. Margineantu, T. Dietterich, Pruning adaptive boosting, in: Proceedings of the Fourteenth International Conference Machine Learning, 1997, pp. 211-218.
102. C. Tamon, J. Xiang, On the boosting pruning problem, in: Proceedings of the 11th European Conference on Machine Learning, 2000, pp. 404-412.
103. Z.H. Zhou, J. Wu, and W. Tang Ensembling neural networks: many could be better than all Artif. Intell. 137 2002 239 263
104. D. Hernández-Lobato, G. Martinez-Muoz, and A. Suárez Statistical instance-based pruning in ensembles of independent classifiers IEEE Trans. Pattern Anal. Mach. Intell. 31 2 2009 364 369
105. S.H. Park, and J. Furnkranz Efficient prediction algorithms for binary decomposition techniques Data Min. Knowl. Disc. 2012 1 38
106. G. Martinez-Muoz, D. Hernandez-Lobato, and A. Suarez An analysis of ensemble pruning techniques based on ordered aggregation IEEE Trans. Pattern Anal. Mach. Intell. 31 2 2009 245 259
107. A.L. Prodromidis, S.J. Stolfo, P.K. Chan, Effective and Efficient Pruning of Metaclassifiers in a Distributed Data Mining System, Technical Report CUCS-017-99, Columbia University, 1999.
108. R. Caruana, A. Niculescu-Mizil, G. Crew, A. Ksikes, Ensemble selection from libraries of models, in: Twenty-First International Conference on Machine Learning, July 04-08, 2004, Banff, Alberta, Canada, 2004.
109. Q. Hu, D. Yu, Z. Xie, and X. Li EROS: ensemble rough subspaces Pattern Recogn. 40 2007 3728 3739
110. R.E. Banfield, L.O. Hall, K.W. Bowyer, and W.P. Kegelmeyer Ensemble diversity measures and their application to thinning Inform. Fusion 6 1 2005 49 62
111. I. Partalas, G. Tsoumakas, and I. Vlahavas An ensemble uncertainty aware measure for directed hill climbing ensemble pruning Mach. Learn. 81 3 2010 257 282
112. L. Rokach Collective-agreement-based pruning of ensembles Comput. Stat. Data Anal. 53 4 2009 1015 1026
113. H. Zhang, and M. Wang Search for the smallest random forest Stat. Interface 2 2009 381 388
114. Z.H. Zhou, W. Tang, Selective ensemble of decision trees, in: Guoyin Wang, Qing Liu, Yiyu Yao, Andrzej Skowron (Eds.), Rough Sets, Fuzzy Sets, Data Mining, and Granular Computing, 9th International Conference, RSFDGrC, Chongqing, China, Proceedings, Lecture Notes in Computer Science, vol. 2639, 2003, pp. 476-483.
115. Guoyin Wang, Qing Liu, Yiyu Yao, Andrzej Skowron (Eds.), Rough Sets, Fuzzy Sets, Data Mining, and Granular Computing, 9th International Conference, RSFDGrC, Chongqing, China, Proceedings, Lecture Notes in Computer Science, vol. 2639, 2003, pp. 476-483.
116. L. Rokach, O. Maimon, and R. Arbel Selective voting - getting more for less in sensor fusion Int. J. Pattern Recogn. Artif. Intell. 20 03 2006 329 350
117. A.L. Prodromidis, and S.J. Stolfo Cost complexity-based pruning of ensemble classifiers Knowl. Inform. Syst. 3 4 2001 449 469
118. T. Windeatt, G. Ardeshir, An Empirical Comparison of Pruning Methods for Ensemble Classifiers, IDA2001, LNCS, vol. 2189, 2001, pp. 208-217.
119. Y. Zhang, S. Burer, and W.N. Street Ensemble pruning via semi-definite programming J. Mach. Learn. Res. 7 2006 1315 1338