Finding structure in data using multivariate tree boosting

Psychological Methods

Volumn 21, Issue 4, 2016, Pages 583-602

  Miller, Patrick J ^a   Lubke, Gitta H ^a,b   McArtor, Daniel B ^a   Bergeman, C S ^a  

^a UNIVERSITY OF NOTRE DAME   (United States)

^b VU UNIVERSITY AMSTERDAM   (Netherlands)

Author keywords

Boosting; Decision trees; Model selection; Multivariate; Nonparametric regression

Indexed keywords

DECISION TREE; HUMAN; MODEL; MULTIPLE REGRESSION; OUTCOME VARIABLE; PREDICTION; PSYCHOLOGICAL WELL-BEING; SAMPLE SIZE; ALGORITHM; INFORMATION PROCESSING; MULTIVARIATE ANALYSIS;

ALGORITHMS; DATASETS AS TOPIC; DECISION TREES; HUMANS; MULTIVARIATE ANALYSIS;

EID: 85004115628     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/met0000087     Document Type: Article

References (96)

1. Bartone, P. T., Ursano, R. J., Wright, K. M., & Ingraham, L. H. (1989). The impact of a military air disaster on the health of assistance workers. A prospective study. Journal of Nervous and Mental Disease, 177, 317-328. http://dx.doi.org/10.1097/00005053-198906000-00001
2. Belloc, N. B., Breslow, L., & Hochstim, J. R. (1971). Measurement of physical health in a general population survey. American Journal of Epidemiology, 93, 328-336.
3. Benbouzid, D., Busa-Fekete, R., Casagrande, N., Collin, F.-D., & Kégl, B. (2012). MultiBoost: A multi-purpose boosting package. Journal of Machine Learning Research, 13, 549-553.
4. Bergeman, C. S., & Deboeck, P. R. (2014). Trait stress resistance and dynamic stress dissipation on health and well-being: The reservoir model. Research in Human Development, 11, 108-125. http://dx.doi.org/10.1080/15427609.2014.906736
5. Block, J., & Kremen, A. M. (1996). IQ and ego-resiliency: Conceptual and empirical connections and separateness. Journal of Personality and Social Psychology, 70, 349-361. http://dx.doi.org/10.1037/0022-3514.70.2.349
6. Bradley, A. P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30, 1145-1159. http://dx.doi.org/10.1016/S0031-3203(96)00142-2
7. Brandmaier, A. M., von Oertzen, T., McArdle, J. J., & Lindenberger, U. (2013). Structural equation model trees. Psychological Methods, 18, 71-86. http://dx.doi.org/10.1037/a0030001
8. Bray, J. H., & Maxwell, S. E. (1985). Multivariate analysis of variance (Vol. 54). Newbury Park, CA: Sage.
9. Breiman, L. (1996). Bagging predictors. Machine Learning, 24, 123-140. http://dx.doi.org/10.1007/BF00058655
10. Breiman, L. (2001). Random forests. Machine Learning, 45, 5-32. http://dx.doi.org/10.1023/A:1010933404324
11. Breiman, L., Friedman, J., Stone, C. J., & Olshen, R. A. (1984). Classification and regression trees. London, UK: CRC press.
12. Brodley, C. E., & Utgoff, P. E. (1995). Multivariate decision trees. Machine Learning, 19, 45-77. http://dx.doi.org/10.1007/BF00994660
13. Brown, D. E., Pittard, C. L., & Park, H. (1996). Classification trees with optimal multivariate decision nodes. Pattern Recognition Letters, 17, 699-703. http://dx.doi.org/10.1016/0167-8655(96)00033-5
14. Bůhlmann, P., & Hothorn, T. (2007). Boosting algorithms: Regularization, prediction and model fitting. Statistical Science, 22, 477-505. http://dx.doi.org/10.1214/07-STS242
15. Bůhlmann, P., & Yu, B. (2003). Boosting with the L2 loss: Regression and classification. Journal ofthe American Statistical Association, 98, 324-339. http://dx.doi.org/10.1198/016214503000125
16. Bůhlmann, P., & Yu, B. (2006). Sparse boosting. Journal of Machine Learning Research, 7, 1001-1024.
17. Burnham, K. P., & Anderson, D. R. (2002). Model selection and multi-model inference: A practical information-theoretic approach. New York, NY: Springer Science & Business Media.
18. Chapelle, O., Shivaswamy, P., Vadrevu, S., Weinberger, K., Zhang, Y., & Tseng, B. (2010). Multi-task learning for boosting with application to web search ranking. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1189-1198). New York, NY: ACM.
19. Chen, H., Chiang, R. H., & Storey, V. C. (2012). Business Intelligence and Analytics: From Big Data to Big Impact. Management Information Systems Quarterly, 36, 1165-1188.
20. Cohen, S., Kamarck, T., & Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24, 385-396. http://dx.doi.org/10.2307/2136404
21. De'Ath, G. (2002). Multivariate regression trees: A new technique for modeling species-environment relationships. Ecology, 83, 1105-1117.
22. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the Expectation-Maximization (EM) algorithm. Journal ofthe Royal Statistical Society Series B. Methodological, 39, 1-38.
23. Devins, G. M., & Orme, C. M. (1985). Center for epidemiologic studies depression scale. Test Critiques, 20, 144-160.
24. Dine, A., Larocque, D., & Bellavance, F. (2009). Multivariate trees for mixed outcomes. Computational Statistics & Data Analysis, 53, 3795-3804. http://dx.doi.org/10.1016/J.csda.2009.04.003
25. Elith, J., Leathwick, J. R., & Hastie, T. (2008). A working guide to boosted regression trees. Journal ofAnimal Ecology, 77, 802-813. http://dx.doi.org/10.1111/j.1365-2656.2008.01390.x
26. Faddoul, J. B., Chidlovskii, B., Torre, F., & Gilleron, R. (2010, December). Boosting multi-task weak learners with applications to textual and social data. In Machine Learning and Applications (ICMLA), 2010 Ninth International Conference on (pp. 367-372). Washington, DC: IEEE. http://dx.doi.org/10.1109/ICMLA.2010.61
27. Ferreira, M. A. R., & Purcell, S. M. (2009). A multivariate test of association. Bioinformatics, 25, 132-133. http://dx.doi.org/10.1093/bioinformatics/btn563
28. Franco-Arcega, A., Carrasco-Ochoa, J. A., Sánchez-Diaaz, G., & Martianez-Trinidad, J. F. (2010). Multivariate decision trees using different splitting attribute subsets for large datasets. In A. Farzindar & V. Kešelj (Eds.), Advances in artificial intelligence (Vol. 6085, pp. 370-373). Berlin, Germany: Springer. http://dx.doi.org/10.1007/978-3-642-13059-5_49
29. Freund, Y., & Schapire, R. E. (1996). Experiments with a new boosting algorithm (Vol. 96, pp. 148-156). Presented at the International Conference on Machine Learning Bari, Italy.
30. Freund, Y., & Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. Journal ofComputer and System Sciences, 55, 119-139. http://dx.doi.org/10.1006/jcss.1997.1504
31. Friedman, J. (2001). Greedy function approximation: A gradient boosting machine. Annals ofStatistics, 29, 1189-1232. http://dx.doi.org/10.1214/aos/1013203451
32. Friedman, J. (2002). Stochastic gradient boosting. Computational Statistics & Data Analysis, 38, 367-378. http://dx.doi.org/10.1016/S0167-9473(01)00065-2
33. Friedman, J., Hastie, T., & Tibshirani, R. (2000). Additive logistic regression: A statistical view of boosting (With discussion and a rejoinder by the authors). Annals of Statistics, 28, 337-407. http://dx.doi.org/10.1214/aos/1016218223
34. Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal ofStatistical Software, 33, 1-22. http://dx.doi.org/10.18637/jss.v033.i01
35. Friedman, J. H., & Meulman, J. J. (2003). Multiple additive regression trees with application in epidemiology. Statistics in Medicine, 22, 1365-1381. http://dx.doi.org/10.1002/sim.1501
36. Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking"and the research hypothesis was posited ahead oftime. Unpublished manuscript.
37. Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015). Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation. Journal ofComputational and Graphical Statistics, 24, 44-65. http://dx.doi.org/10.1080/10618600.2014.907095
38. Groll, A., & Tutz, G. (2011). Variable selection for generalized additive mixed models by likelihood-based boosting. Munich, Germany: Ludwig-Maximilians-University.
39. Groll, A., & Tutz, G. (2012). Regularization for generalized additive mixed models by likelihood-based boosting. Methods ofInformation in Medicine, 51, 168-177. http://dx.doi.org/10.3414/ME11-02-0021
40. Grömping, U. (2015). Variable importance in regression models. Wiley Interdisciplinary Reviews: Computational Statistics, 7, 137-152. http://dx.doi.org/10.1002/wics.1346
41. Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29-36. http://dx.doi.org/10.1148/radiology.143.1.7063747
42. Hastie, T., & Tibshirani, R. (1990). Generalized additive models (Vol. 43). London, UK: CRC Press.
43. Hastie, T., Tibshirani, R., Friedman, J., Hastie, T., Friedman, J., & Tib-shirani, R. (2009). The elements ofstatistical learning (Vol. 2). New York, NY: Springer. http://dx.doi.org/10.1007/978-0-387-84858-7
44. Hocking, R. R., & Leslie, R. N. (1967). Selection of the best subset in regression analysis. Technometrics, 9, 531-540.
45. Hofner, B., Mayr, A., Robinzonov, N., & Schmid, M. (2014). Model-based boosting in R: A hands-on tutorial using the R package mboost. Computational Statistics, 29, 3-35. http://dx.doi.org/10.1007/s00180-012-0382-5
46. Holland, P. W., & Wainer, H. (Eds.). (1993). Differential item functioning. New York, NY: Routledge.
47. Hothorn, T., Bůhlmann, P., Kneib, T., Schmid, M., & Hofner, B. (2015). mboost: Model-Based Boosting, R package version R package version 2.5-0. Retrieved from http://CRAN.R-project.org/package-mboost
48. Hothorn, T., Bůhlmann, P., Kneib, T., Schmid, M., & Hofner, B. (2010). Model-based boosting 2.0. Journal ofMachine Learning Research, 99, 2109-2113.
49. Hothorn, T., Hornik, K., & Zeileis, A. (2006). Unbiased recursive partitioning: A conditional inference framework. Journal of Computational and Graphical Statistics, 5, 651-674.
50. Howe, D., Costanzo, M., Fey, P., Gojobori, T., Hannick, L., Hide, W., Rhee, S. Y. (2008). Big data: The future of biocuration. Nature, 455, 47-50. http://dx.doi.org/10.1038/455047a
51. Hsiao, W. C., & Shih, Y. S. (2007). Splitting variable selection for multivariate regression trees. Statistics & Probability Letters, 77, 265-271. http://dx.doi.org/10.1016/j.spl.2006.08.014
52. Huo, Y., & Budescu, D. V. (2009). Erratum to "An extension of dominance analysis to canonical correlation analysis." Multivariate Behavioral Research 44, 859. http://dx.doi.org/10.1080/00273170903467679
53. Hyafil, L., & Rivest, R. L. (1976). Constructing optimal binary decision trees is NP-complete. Information Processing Letters, 5, 15-17. http://dx.doi.org/10.1016/0020-0190(76)90095-8
54. Johnson, S. C. (1967). Hierarchical clustering schemes. Psychometrika, 32, 241-254. http://dx.doi.org/10.1007/BF02289588
55. Lambert, Z. V., Wildt, A. R., & Durand, R. M. (1988). Redundancy analysis: An alternative to canonical correlation and multivariate multiple regression in exploring interset associations. Psychological Bulletin, 104, 282-289. http://dx.doi.org/10.1037/0033-2909.104.2.282
56. Loh, W. Y., & Zheng, W. (2013). Regression trees for longitudinal and multiresponse data. The Annals of Applied Statistics, 7, 495-522. http://dx.doi.org/10.1214/12-AOAS596
57. Lutz, R. W., & Bůhlmann, P. (2006). Boosting for high-multivariate responses in high-dimensional linear regression. Statistica Sinica, 16, 471-494.
58. Mann, J. J., Ellis, S. P., Waternaux, C. M., Liu, X., Oquendo, M. A., Malone, K. M., Currier, D. (2008). Classification trees distinguish suicide attempters in major psychiatric disorders: A model of clinical decision making. The Journal of Clinical Psychiatry, 69, 23-31. http://dx.doi.org/10.4088/JCP.v69n0104
59. Manovich, L. (2012). Trending: The promises and the challenges of big social data. In M. K. Gold (Ed.), Debates in the digital humanities (pp. 460-475). Minneapolis, MN: University of Minnesota Press. http://dx.doi.org/10.5749/minnesota/9780816677948.003.0047
60. Mentch, L., & Hooker, G. (2014). Ensemble trees and CLTs: Statistical inference for supervised learning. arXiv:1404.6473.
61. Nathans, L. L., Oswald, F. L., & Nimon, K. (2012). Interpreting multiple linear regression: A guidebook of variable importance. Practical Assessment, Research & Evaluation, 17, 1-19.
62. Nimon, K., Henson, R. K., & Gates, M. S. (2010). Revisiting interpretation of canonical correlation analysis: A tutorial and demonstration of canonical commonality analysis. Multivariate Behavioral Research, 45, 702-724. http://dx.doi.org/10.1080/00273171.2010.498293
63. Obozinski, G., Taskar, B., & Jordan, M. I. (2006). Multi-task feature selection. Statistics Department, UC Berkeley, Tech. Rep. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi = 10.1.1.94. 951&rep=rep1&type=pdf
64. Pallant, J. F. (2000). Development and validation of a scale to measure perceived control of internal states. Journal ofPersonality Assessment, 75, 308-337. http://dx.doi.org/10.1207/S15327752JPA7502_10
65. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal ofMachine Learning Research, 12, 2825-2830.
66. Politis, D. N., & Romano, J. P. (1994). Large sample confidence regions based on subsamples under minimal assumptions. Annals of Statistics, 22, 2031-2050. http://dx.doi.org/10.1214/aos/1176325770
67. Procidano, M. E., & Heller, K. (1983). Measures of perceived social support from friends and from family: Three validation studies. American Journal ofCommunity Psychology, 11, 1-24. http://dx.doi.org/10.1007/BF00898416
68. Proctor, L. J., Van Dusen Randazzo, K., Litrownik, A. J., Newton, R. R., Davis, I. P., & Villodas, M. (2011). Factors associated with caregiver stability in permanent placements: A classification tree approach. Child Abuse & Neglect: The International Journal, 35, 425-436. http://dx.doi.org/10.1016/j.chiabu.2011.02.002
69. R Core Team. (2015). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.
70. Reid, D. W., & Ziegler, M. (1981). The desired control measure and adjustment among the elderly. In H. M. Lefcourt (Ed.), Research with the locus ofcontrol construct: Vol. 1, Assessment methods (pp. 127-159). New York, NY: Academic Press. http://dx.doi.org/10.1016/B978-0-12-443201-7.50008-7
71. Ridgeway, G. (1999). The state of boosting. Computing Science and Statistics, 31, 172-181.
72. Ridgeway, G. (2015). gbm: Generalized boosted regression models. [Computer software]. Retrieved from http://CRAN.R-project.org/package=gbm
73. Robinzonov, N., Tutz, G., & Hothorn, T. (2012). Boosting techniques for nonlinear time series models. Advances in Statistical Analysis, 96, 99-122. http://dx.doi.org/10.1007/s10182-011-0163-4
74. Russell, D., Peplau, L. A., & Cutrona, C. E. (1980). The revised UCLA Loneliness Scale: Concurrent and discriminant validity evidence. Journal of Personality and Social Psychology, 39, 472-480. http://dx.doi.org/10.1037/0022-3514.39.3.472
75. Ryff, C. D., & Keyes, C. L. M. (1995). The structure of psychological well-being revisited. Journal ofPersonality and Social Psychology, 69, 719-727. http://dx.doi.org/10.1037/0022-3514.69.4.719
76. Scott, S. B., Jackson, B. R., & Bergeman, C. S. (2011). What contributes to perceived stress in later life? A recursive partitioning approach. Psychology and Aging, 26, 830-843. http://dx.doi.org/10.1037/a0023180
77. Scott, S. B., Whitehead, B. R., Bergeman, C. S., & Pitzer, L. (2013). Combinations of stressors in midlife: Examining role and domain stressors using regression trees and random forests. The Journals of Geron-tology Series B, Psychological Sciences and Social Sciences, 68, 464-475. http://dx.doi.org/10.1093/geronb/gbs166
78. Segal, M. R. (1992). Tree-structured methods for longitudinal data. Journal ofthe American Statistical Association, 87, 407-418. http://dx.doi.org/10.1080/01621459.1992.10475220
79. Segal, M., & Xiao, Y. (2011). Multivariate random forests. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1, 80-87.
80. Sela, R. J., & Simonoff, J. S. (2012). RE-EM trees: A data mining approach for longitudinal and clustered data. Machine Learning, 86, 169-207. http://dx.doi.org/10.1007/s10994-011-5258-3
81. Seroczynski, A. D., Cole, D. A., & Maxwell, S. E. (1997). Cumulative and compensatory effects of competence and incompetence on depressive symptoms in children. Journal of Abnormal Psychology, 106, 586-597. http://dx.doi.org/10.1037/0021-843X.106.4.586
82. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Belmont, CA: Wadsworth Cengage Learning.
83. Shafik, N., & Tutz, G. (2009). Boosting nonlinear additive autoregressive time series. Computational Statistics & Data Analysis, 53, 2453-2464. http://dx.doi.org/10.1016/j.csda.2008.12.006
84. Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007). Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformatics, 8, 25. http://dx.doi.org/10.1186/1471-2105-8-25
85. Strobl, C., Malley, J., & Tutz, G. (2009). An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological Methods, 14, 323-348. http://dx.doi.org/10.1037/a0016973
86. Struyf, J., & Dzeroski, S. (2006). Constraint based induction of multi-objective regression trees. In F. Bonchi & J. F. Boulicaut (Eds.), Knowledge discovery in inductive databases (Vol. 3933, pp. 222-233). Berlin, Germany: Springer. http://dx.doi.org/10.1007/11733492_13
87. Tanner, M. A., & Wong, W. H. (1987). The calculation of posterior distributions by data augmentation. Journal ofthe American Statistical Association, 82, 528-540. http://dx.doi.org/10.1080/01621459.1987.10478458
88. Thompson, B. (1995). Stepwise regression and stepwise discriminant analysis need not apply. Educational and Psychological Measurement, 55, 524-534.
89. Thompson, B. (2005). Canonical correlation analysis. In B. S. Everitt & D. Howell (Eds.), Encyclopedia ofstatistics in behavioral science. Hoboken, NJ: Wiley, Ltd. http://dx.doi.org/10.1002/0470013192.bsa068
90. Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal ofthe Royal Statistical Society. Series B. Methodological, 58, 267-288.
91. Troyanskaya, O., Cantor, M., Sherlock, G., Brown, P., Hastie, T., Tibshi-rani, R., Altman, R. B. (2001). Missing value estimation methods for DNA microarrays. Bioinformatics, 17, 520-525. http://dx.doi.org/10.1093/bioinformatics/17.6.520
92. Van der Sluis, S., Posthuma, D., & Dolan, C. V. (2013). TATES: Efficient multivariate genotype-phenotype analysis for genome-wide association studies. PLOS Genetics, 9, e1003235. http://dx.doi.org/10.1371/journal.pgen.1003235
93. Wallace, K. A., Bergeman, C. S., & Maxwell, S. E. (2002). Predicting well-being outcomes in later life: An application of classification and regression tree (CART) analysis. In S. P. Shohov (Ed.), Advances in psychology research (Vol. 17, pp. 71-92). Hauppauge, NY: Nova Science Publishers.
94. Wang, X., Zhang, C., & Zhang, Z. (2009). Boosted multi-task learning for face verification with applications to web image and video search. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (pp. 142-149). Washington, DC: IEEE.
95. Wood, S. (2006). Generalized additive models: An introduction with R. London, UK: CRC press.
96. Zeileis, A., Hothorn, T., & Hornik, K. (2008). Model-based recursive partitioning. Journal of Computational and Graphical Statistics, 17, 492-514. http://dx.doi.org/10.1198/106186008X319331