Fifty years of classification and regression trees

International Statistical Review

Volumn 82, Issue 3, 2014, Pages 329-348

  Loh, Wei Yin ^a  

^a University of Wisconsin   (United States)

Author keywords

Classification trees; Machine learning; Prediction; Regression trees

Indexed keywords

EID: 84912521725     PISSN: 03067734     EISSN: 17515823     Source Type: Journal    
DOI: 10.1111/insr.12016     Document Type: Article

References (144)

1. Aalen O.O. (1978). Nonparametric inference for a family of counting processes. Ann. Stat., 6, 701-726.
2. Abdolell M., LeBlanc M., Stephens D. & Harrison R.V. (2002). Binary partitioning for continuous longitudinal data: categorizing a prognostic variable. Stat. Med., 21, 3395-3409.
3. Ahn H. (1994a). Tree-structured exponential regression modeling. Biometrical J., 36, 43-61.
4. Ahn H. (1994b). Tree-structured extreme value model regression. Commun. Stat.-Theor. M., 23, 153-174.
5. Ahn H. (1996a). Log-gamma regression modeling through regression trees. Commun. Stat.-Theor. M., 25, 295-311.
6. Ahn H. (1996b). Log-normal regression modeling through recursive partitioning. Comput. Stat. Data Anal., 21, 381-398.
7. Ahn H. & Chen J. (1997). Tree-structured logistic model for over-dispersed binomial data with application to modeling developmental effects. Biometrics, 53, 435-455.
8. Ahn H. & Loh W.-Y. (1994). Tree-structured proportional hazards regression modeling. Biometrics, 50, 471-485.
9. Alberg D., Last M. & Kandel A. (2012). Knowledge discovery in data streams with regression tree methods. Wil. Interdiscip. Rev.: Data Mining and Knowledge Disc., 2, 69-78.
10. Alexander W.P. & Grimshaw S.D. (1996). Treed regression. J. Comput. Graph. Stat., 5, 156-175.
11. Bacchetti P. & Segal M.R. (1995). Survival trees with time-dependent covariates: application to estimating changes in the incubation period of AIDS. Lifetime Data Anal., 1, 35-47.
12. Bishop Y.M.M., Fienberg S.E. & Holland P.W. (1975). Discrete Multivariate Analysis. Cambridge, MA: MIT Press.
13. Breiman L., Friedman J.H., Olshen R.A. & Stone C.J. (1984). Classification and Regression Trees. Belmont: Wadsworth.
14. Breiman L. (1996). Bagging predictors. Mach. Learn., 24, 123-140.
15. Breiman L. (2001). Random forests. Mach. Learn., 45, 5-32.
16. Brown D.E., Pittard C.L. & Park H (1996). Classification trees with optimal multivariate decision nodes. Pattern. Recogn. Lett., 17, 699-703.
17. Buckley J.J. & James I.R. (1979). Linear regression with censored data. Biometrika, 66, 429-436.
18. Buttrey S.E. & Karo C. (2002). Using k-nearest-neighbor classification in the leaves of a tree. Comput. Stat. Data Anal., 40, 27-37.
19. Chan K.-Y. & Loh W.-Y. (2004). LOTUS: an algorithm for building accurate and comprehensible logistic regression trees. J. Comput. Graph. Stat., 13, 826-852.
20. Chaudhuri P., Huang M.-C., Loh W.-Y. & Yao R. (1994). Piecewise-polynomial regression trees. Stat. Sinica, 4, 143-167.
21. Chaudhuri P., Lo W.-D., Loh W.-Y. & Yang C.-C. (1995). Generalized regression trees. Stat. Sinica, 5, 641-666.
22. Chaudhuri P. & Loh W.-Y. (2002). Nonparametric estimation of conditional quantiles using quantile regression trees. Bernoulli, 8, 561-576.
23. Chipman H.A., George E.I. & McCulloch R.E. (1998). Bayesian CART model search. J. Amer. Statist. Assoc., 93, 935-948.
24. Chipman H.A., George E.I. & McCulloch R.E. (2002). Bayesian treed models. Mach. Learn., 48, 299-320.
25. Cho H.J. & Hong S.-M. (2008). Median regression tree for analysis of censored survival data. IEEE T. Syst. Man Cy. A., 38, 715-726.
26. Choi Y., Ahn H. & Chen J. (2005). Regression trees for analysis of count data with extra Poisson variation. Comput. Stat. Data Anal., 49, 893-915.
27. Chou P.A. (1991). Optimal partitioning for classification and regression trees. IEEE T. Pattern. Anal., 13, 340-354.
28. Ciampi A. (1991). Generalized regression trees. Comput. Stat. Data. Anal., 12, 57-78.
29. Ciampi A., Couturier A. & Li S.L. (2002). Prediction trees with soft nodes for binary outcomes. Stat. Med., 21, 1145-1165.
30. Ciampi A., Hogg S.A., McKinney S. & Thiffault J. (1988). RECPAM: A computer program for recursive partition and amalgamation for censored survival data and other situations frequently occurring in biostatistics. Comput. Meth. Prog. Bio., 26, 239-256.
31. Ciampi A., Thiffault J., Nakache J-P. & Asselain B. (1986). Stratification by stepwise regression, correspondence analysis and recursive partition: A comparison of three methods of analysis for survival data with covariates. Comput. Stat. Data Anal., 4, 185-204.
32. Clarke J. & West M. (2008). Bayesian Weibull tree models for survival analysis of clinico-genomic data. Stat. Meth., 5, 238-262.
33. Crowley J., LeBlanc M., Gentleman R. & Salmon S. (1995). Exploratory methods in survival analysis. In Analysis of Censored Data, vol.27, Eds. H.L. Koul & J.V. Deshpande, pp. 55-77. Institute of Mathematical Statistics, IMS Lecture Notes-Monograph Series: Hayward, CA.
34. Davis R.B. & Anderson J.R. (1989). Exponential survival trees. Stat. Med., 8, 947-961.
35. De'ath G. (2002). Multivariate regression trees: a new technique for modeling species-environment relationships. Ecology, 83, 1105-1117.
36. De'ath G. & Fabricius K.E. (2000). Classification and regression trees: A powerful yet simple technique for ecological data analysis. Ecology, 81, 3178-3192.
37. Denison D.G.T., Mallick B.K. & Smith A.F.M. (1998). A Bayesian CART algorithm. Biometrika, 85, 363-377.
38. Dietterich T.G. (2000). Ensemble methods in machine learning. In Multiple Classifier Systems, LBCS-1857, pp. 1-15. New York: Springer.
39. Ding Y. & Simonoff J.S. (2010). An investigation of missing data methods for classification trees applied to binary response data. J. Mach. Learn. Res., 11, 131-170.
40. Dobra A. & Gehrke J.E. (2002). SECRET: A scalable linear regression tree algorithm. In Proceedings of the eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 481-487. Edmonton, Canada: ACM Press.
41. Doyle P. (1973). The use of automatic interaction detector and similar search procedures. Oper. Res. Quart., 24, 465-467.
42. Dusseldorp E., Conversano C. & VanOs B.J. (2010). Combining an additive and tree-based regression model simultaneously: STIMA. J. Comput. Graph. Stat., 19, 514-530.
43. Dusseldorp E. & Meulman J.J. (2004). The regression trunk approach to discover treatment covariate interaction. Psychometrika, 69, 355-374.
44. Einhorn H.J. (1972). Alchemy in the behavioural sciences. Public Opin. Quart., 36, 367-378.
45. Fan G. & Gray J.B. (2005). Regression tree analysis using TARGET. J. Comput. Graph. Stat., 14, 1-13.
46. Fan J., Su X., Levine R.A., Nunn M.E. & LeBlanc M. (2006). Trees for correlated survival data by goodness of split, with applications to tooth prognosis. J. Amer. Statist. Assoc., 101, 959-967.
47. Fisher R.A. (1936). The use of multiple measurements in taxonomic problems. Ann. Eugenic., 7(2), 179-188.
48. Freund Y. & Schapire R.E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci., 55(1), 119-139.
49. Friedman J., Hastie T. & Tibshirani R. (2000). Additive logistic regression: A statistical view of boosting. Ann. Stat., 38(2), 337-374.
50. Gama J. (2004). Functional trees. Mach. Learn., 55, 219-250.
51. Gao F., Manatunga A.K. & Chen S. (2004). Identification of prognostic factors with multivariate survival data. Comput. Stat. Data Anal., 45, 813-824.
52. Garg L., McClean S., Meenan B.J. & Millard P. (2011). Phase-type survival trees and mixed distribution survival trees for clustering patients' hospital length of stay. Informatica, 22, 57-72.
53. Gashler M., Giraud-Carrier C.G. & Martinez T.R. (2008). Decision tree ensemble: small heterogeneous is better than large homogeneous. In Seventh International Conference on Machine Learning and Applications, pp. 900-905. Washington, DC: IEEE Computer Society.
54. Gehrke J. (2009). Scalable decision tree construction. In Encyclopedia of Database Systems, Eds. L. Liu & T. Ozsu pp. 2469-2474. New York: Springer.
55. Gordon L. & Olshen R.A. (1985). Tree-structured survival analysis. Cancer Treat. Rep., 69, 1065-1069.
56. Gray J.B. & Fan G. (2008). Classification tree analysis using TARGET. Comput. Stat. Data Anal., 52, 1362-1372.
57. Guerts P., Ernst D. & Wehenkel L. (2006). Extremely randomized trees. Mach. Learn., 63, 3-42.
58. Hall M., Frank E., Holmes G., Pfahringer B., Reutemann P. & Witten I.H. (2009). The WEKA data mining software: an update. SIGKDD Explorations, 11, 10-18.
59. Hamza M. & Larocque D. (2005). An empirical comparison of ensemble methods based on classification trees. J. Stat. Comput. Sim., 75, 629-643.
60. Harper P.R. (2005). A review and comparison of classification algorithms for medical decision making. Health Policy, 71, 315-331.
61. Henrichon Jr. E.G. & Fu K.-S. (1969). A nonparametric partitioning procedure for pattern classification. IEEE T. Comput., C-18, 614-624.
62. Hoaglin D.C. & Velleman P.F. (1995). A critical look at some analyses of major league baseball salaries. Am. Stat., 49, 277-285.
63. Hothorn T., Bühlmann P., Dudoit S., Molinaro A. & vander Laan M.J. (2006a). Survival ensembles. Biostatistics, 7, 355-373.
64. Hothorn T., Hornik K. & Zeileis A. (2006b). Unbiased recursive partitioning: a conditional inference framework. J. Comput. Graph. Stat., 15, 651-674.
65. Hothorn T. & Lausen B. (2005). Bundling classifiers by bagging trees. Comput. Stat. Data Anal., 49, 1068-1078.
66. Hothorn T., Lausen B., Benner A. & Radespiel-Tröger M. (2004). Bagging survival trees. Stat. Med., 23, 77-91.
67. Hsiao W.-C. & Shih Y.-S. (2007). Splitting variable selection for multivariate regression trees. Stat. Probab. Lett., 77, 265-271.
68. Ishwaran H., Blackstone E.H., Pothier C.E. & Lauer M. (2004). Relative risk forests for exercise heart rate recovery as a predictor of mortality. J. Amer. Statist. Assoc., 99, 591-600.
69. Ishwaran H., Kogalur U.B., Blackstone E.H. & Lauer M. (2006). Random survival forests. Ann. Appl. Stat., 2, 841-860.
70. Jin H., Lu Y., Stone K. & Black D.M. (2004). Survival analysis based on variance of survival time. Med. Decis. Making, 24, 670-680.
71. Kass G.V. (1980). An exploratory technique for investigating large quantities of categorical data. Ann. Appl. Stat., 29, 119-127.
72. Kim H. & Loh W.-Y. (2001). Classification trees with unbiased multiway splits. J. Amer. Statist. Assoc., 96, 589-604.
73. Kim H. & Loh W.-Y. (2003). Classification trees with bivariate linear discriminant node models. J. Comput. Graph. Stat., 12, 512-530.
74. Kim H., Loh W.-Y., Shih Y.-S. & Chaudhuri P. (2007). Visualizable and interpretable regression models with good prediction power. IIE Transactions, 39, 565-579.
75. Kononenko I. (1995). On biases in estimating multi-valued attributes. In 14th International Joint Conference on Artificial Intelligence, pp. 1034-1040. Burlington, MA: Morgan Kaufmann.
76. Kooperberg C., Bose S. & Stone C.J. (1997). Polychotomous regression. J. Amer. Statist. Assoc., 92, 117-127.
77. Laird N.M. & Ware J.H. (1982). Random-effects models for longitudinal data. Biometrics, 38, 963-974.
78. Landwehr N., Hall M. & Frank E. (2005). Logistic model trees. Mach. Learn., 59, 161-205.
79. Larsen D.R. & Speckman P.L. (2004). Multivariate regression trees for analysis of abundance data. Biometrics, 60, 543-549.
80. LeBlanc M. & Crowley J. (1992). Relative risk trees for censored survival data. Biometrics, 48, 411-425.
81. LeBlanc M. & Crowley J. (1993). Survival trees by goodness of split. J. Amer. Statist. Assoc., 88, 457-467.
82. Lee P.H. & Yu P.L.H. (2010). Distance-based tree models for ranking data. Comput. Stat. Data Anal., 54, 1672-1682.
83. Lee S.K. (2005). On generalized multivariate decision tree by using GEE. Comput. Stat. Data Anal., 49, 1105-1119.
84. Lemon S.C., Roy J., Clark M.A., Friedman P.D. & Rakowski W. (2003). Classification and regression tree analysis in public health: Methodological review and comparison with logistic regression. Ann. Behav. Med., 26, 172-181.
85. Light R.J. & Margolin B.H. (1971). An analysis of variance for categorical data. J. Amer. Statist. Assoc., 66, 534-544.
86. Lim T.-S., Loh W.-Y. & Shih Y.-S. (2000). A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Mach. Learn., 40, 203-228.
87. Loh W.-Y. (1991). Survival modeling through recursive stratification. Comput. Stat. Data Anal., 12, 295-313.
88. Loh W.-Y. (2002). Regression trees with unbiased variable selection and interaction detection. Stat. Sinica, 12, 361-386.
89. Loh W.-Y. (2006a). Logistic regression tree analysis. In Handbook of engineering statistics, Ed. H. Pham, pp. 537-549. New York: Springer
90. Loh W.-Y. (2006b). Regression tree models for designed experiments. In Second E. L. Lehmann Symposium, Vol.49, Ed. J. Rojo. IMS Lecture Notes-Monograph Series, pp. 210-228. Bethesda, MD: Institute of Mathematical Statistics.
91. Loh W.-Y. (2008a). Classification and regression tree methods. In Encyclopedia of Statistics in Quality and Reliability, Eds. F. Ruggeri, R. Kenett & F.W. Faltin, pp. 315-323. Chichester, UK: Wiley.
92. Loh W.-Y. (2008b). Regression by parts: fitting visually interpretable models with GUIDE. In Handbook of Data Visualization, Eds. C. Chen, W. Härdle & A. Unwin. pp. 447-469. New York: Springer.
93. Loh W.-Y. (2009). Improving the precision of classification trees. Ann. Appl. Stat., 3, 1710-1737.
94. Loh W.-Y. (2011). Classification and regression trees. Wil. Interdiscip. Rev.: Data Mining and Knowledge Disc., 1, 14-23.
95. Loh W.-Y. (2012). Variable selection for classification and regression in large p, small n problems. In Probability approximations and beyond, Vol.205, Eds. A. Barbour, H.P. Chan & D. Siegmund. Lecture Notes in Statistics-Proceedings. pp. 133-157. New York: Springer.
96. Loh W.-Y., Chen C.-W. & Zheng W. (2007). Extrapolation errors in linear model trees. ACM Trans. On Knowledge Disc. From Data, 1. DOI: 10.1145/1267066.1267067.
97. Loh W.-Y. & Shih Y.-S. (1997). Split selection methods for classification trees. Stat. Sinica, 7, 815-840.
98. Loh W.-Y. & Vanichsetakul N. (1988). Tree-structured classification via generalized discriminant analysis (with discussion). J. Amer. Statist. Assoc., 83, 715-728.
99. Loh W.-Y. & Zheng W. (2013). Regression trees for longitudinal and multiresponse data. Ann. Appl. Stat., 7(1), 495-522.
100. Meisel W.S. & Michalopoulos D.A. (1973). A partitioning algorithm with application in pattern classification and the optimization of decision trees. IEEE T. Comput., C-22, 93-103.
101. Merkle E.C. & Shaffer V.A. (2011). Binary recursive partitioning: Background, methods, and application to psychology. Brit. J. Math. Stat. Psy., 64, 161-181.
102. Messenger R. & Mandell L. (1972). A modal search technique for predictive nominal scale multivariate analysis. J. Amer. Statist. Assoc., 67, 768-772.
103. Molinaro A.M., Dudoit S. & vander Laan M.J. (2004). Tree-based multivariate regression and density estimation with right-censored data. J. Multivariate Anal., 90, 154-177.
104. Morgan J.N. & Sonquist J.A. (1963). Problems in the analysis of survey data, and a proposal. J. Amer. Statist. Assoc., 58, 415-434.
105. Murthy S.K. (1998). Automatic construction of decision trees from data: a multi-disciplinary survey. Data Min. Knowl. Disc., 2, 345-389.
106. Payne H.J. & Meisel W.S. (1977). An algorithm for constructing optimal binary decision trees. IEEE T. Comput., C-26, 905-916.
107. Perlich C., Provost F. & Simonoff J.S. (2004). Tree induction vs. logistic regression: A learning-curve analysis. J. Mach. Learn. Res., 4, 211-255.
108. Peto R. & Peto J. (1972). Asymptotically efficient rank invariant test procedures. J. R. Stat. Soc. Ser.-A, 135, 185-207.
109. Potts D. & Sammut C. (2005). Incremental learning of linear model trees. Mach. Learn., 61, 5-48.
110. Quinlan J.R. (1986). Induction of decision trees. Mach. Learn., 1, 81-106.
111. Quinlan J.R. (1992). Learning with continuous classes. In Proceedings of the 5th Australian Joint Conference on Artificial Intelligence, pp. 343-348. Singapore: World Scientific.
112. Quinlan J.R. (1993). C4.5: Programs for Machine Learning. San Mateo, CA: Morgan Kaufmann.
113. R Core Team. (2014). R: A language and environment for statistical computing, R Foundation for Statistical Computing. Vienna, Austria. ISBN 3-900051-07-0.
114. Rusch T. & Zeileis A. (2013). Gaining insight with recursive partitioning of generalized linear models. J. Stat. Comput. Sim., 83(7), 1301-1315.
115. Sandri M. & Zuccolotto P. (2008). A bias correction algorithm for the Gini variable importance measure in classification trees. J. Comput. Graph. Stat., 17, 611-628.
116. Segal M.R. (1988). Regression trees for censored data. Biometrics, 44, 35-47.
117. Segal M.R. (1992). Tree structured methods for longitudinal data. J. Amer. Statist. Assoc., 87, 407-418.
118. Sela R.J. & Simonoff J.S. (2011). Reemtree: Regression trees with random effects. R package version 0.90.3.
119. Sela R.J. & Simonoff J.S. (2012). RE-EM trees: A data mining approach for longitudinal and clustered data. Mach. Learn., 86, 169-207.
120. Sethi I.K. & Chatterjee B. (1977). Efficient decision tree design for discrete variable pattern recognition problems. Pattern Recogn., 9, 197-206.
121. Shih Y.S. (2004). A note on split selection bias in classification trees. Comput. Stat. Data Anal., 45, 457-466.
122. Shih Y.S. & Tsai H.W. (2004). Variable selection bias in regression trees with constant fits. Comput. Stat. Data Anal., 45, 595-607.
123. Smyth P., Gray A. & Fayyad U.M. (1995). Retrofitting decision tree classifiers using kernel density estimation. In Proceedings of the 12th International Conference on Machine Learning, pp. 506-514. Burlington, MA: Morgan Kaufmann.
124. Strobl C., Boulesteix A. & Augustin T. (2007a). Unbiased split selection for classification trees based on the Gini index. Comput. Stat. Data Anal., 52, 483-501.
125. Strobl C., Boulesteix A., Kneib T., Augustin T. & Zeileis A. (2008). Conditional variable importance for random forests. BMC Bioinformatics, 9, 307.
126. Strobl C., Boulesteix A., Zeileis A. & Hothorn T. (2007b). Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformatics, 8, 25.
127. Strobl C., Kopf J. & Zeileis A. (2010). A new method for detecting differential item functioning in the Rasch model. Tech. Rep. 92, Department of Statistics, Ludwig-Maximilians-Universitaet Muenchen.
128. Strobl C., Malley J. & Tutz G. (2009). An introduction to recursive partitioning: Rationale, application and characteristics of classification and regression trees, bagging and random forests. Psychol. Methods, 14, 323-348.
129. Strobl C., Wickelmaier F. & Zeileis A. (2011). Accounting for individual differences in Bradley-Terry models by means of recursive partitioning. J. Educ. Behav. Stat., 36(2), 135-153.
130. Su X. & Fan J. (2004). Multivariate survival trees: A maximum likelihood approach based on frailty models. Biometrics, 60, 93-99.
131. Su X.G., Wang M. & Fan J.J. (2004). Maximum likelihood regression trees. J. Comput. Graph. Stat., 13, 586-598.
132. Taddy M.A., Gramacy R.B. & Polson N.G. (2011). Dynamic trees for learning and design. J. Amer. Statist. Assoc., 106, 109-123.
133. Therneau T.M. & Atkinson B. (2011). rpart: Recursive partitioning, R port by Brian Ripley. R package version 3.1-50.
134. Torgo L. (1997). Functional models for regression tree leaves. In Proceedings of the Fourteenth International Conference on Machine Learning, Ed. D.H. Fisher. pp. 385-393. Burlington, MA: Morgan Kaufmann.
135. Wang Y. & Witten I.H. (1996). Induction of model trees for predicting continuous classes, Working paper series, Department of Computer Science, University of Waikato.
136. White A.P. & Liu W.Z. (1994). Technical note: bias in information-based measures in decision tree induction. Mach. Learn., 15, 321-329.
137. Yildiz O.T. & Alpaydin E. (2001). Omnivariate decision trees. IEEE T. Neural. Networ., 12, 1539-1546.
138. Yildiz O.T. & Alpaydin E. (2005a). Linear discriminant trees. Int. J. Pattern Recogn., 19, 323-353.
139. Yildiz O.T. & Alpaydin E. (2005b). Model selection in omnivariate decision trees. In Machine learning: ECML 2005, Proceedings, Vol.3720, Eds. J. Gama, R. Camacho, P. Brazdil, A. Jorge & L. Torgo. Lecture Notes in Artificial Intelligence, pp. 473-484. Berlin: Springer-Verlag.
140. Yu Y. & Lambert D. (1999). Fitting trees to functional data, with an application to time-of-day patterns. J. Comput. Graph. Stat., 8, 749-762.
141. Zeileis A., Hothorn T. & Hornik K. (2008). Model-based recursive partitioning. J. Comput. Graph. Stat., 17, 492-514.
142. Zhang H. (1998). Classification trees for multiple binary responses. J. Amer. Statist. Assoc., 93, 180-193.
143. Zhang H. & Singer B.H. (2010). Recursive Partitioning and Applications, 2nd ed. New York: Springer.
144. Zhang H. & Ye Y. (2008). A tree-based method for modeling a multivariate ordinal response. Stat. and Its Interface, 1, 169-178.