Boosting algorithms: Regularization, prediction and model fitting

Statistical Science

Volumn 22, Issue 4, 2007, Pages 477-505

  Bühlmann, Peter ^a   Hothorn, Torsten ^b  

^a ETH ZURICH   (Switzerland)

^b UNIVERSITY OF MUNICH   (Germany)

Author keywords

Generalized additive models; Generalized linear models; Gradient boosting; Software; Survival analysis; Variable selection

Indexed keywords

EID: 41549141939     PISSN: 08834237     EISSN: None     Source Type: Journal    
DOI: 10.1214/07-STS242     Document Type: Article

References (96)

1. AMIT, Y. and GEMAN, D. (1997). Shape quantization and recognition with randomized trees. Neural Computation 9 1545-1588.
2. AUDRINO, F. and BARONE-ADESI, G. (2005). Functional gradient descent for financial time series with an application to the measurement of market risk. J. Banking and Finance 29 959-977.
3. AUDRINO, F. and BARONE-ADESI, G. (2005). A multivariate FGD technique to improve VaR computation in equity markets. Comput. Management Sci. 2 87-106.
4. AUDRINO, F. and BÜHLMANN, P. (2003). Volatility estimation with functional gradient descent for very highdimensional financial time series. J. Comput. Finance 6 65-89.
5. BARTLETT, P. (2003). Prediction algorithms: Complexity, concentration and convexity. In Proceedings of the 13th IFAC Symp. on System Identification.
6. BARTLETT, P. L., JORDAN, M. and MCAULIFFE, J. (2006). Convexity, classification, and risk bounds. J. Amer. Statist. Assoc. 101 138-156. MR2268032
7. BARTLETT, P. and TRASKIN, M. (2007). AdaBoost is consistent. J. Mach. Learn. Res. 8 2347-2368.
8. BENNER, A. (2002). Application of "aggregated classifiers" in survival time studies. In Proceedings in Computational Statistics (COMPSTAT) (W. Härdle and B. Rönz, eds.) 171-176. Physica-Verlag, Heidelberg. MR1973489
9. BINDER, H. (2006). GAMBoost: Generalized additive models by likelihood based boosting. R package version 0.9-3. Available at http://CRAN.R-project.org.
10. BISSANTZ, N., HOHAGE, T., MUNK, A. and RUYMGAART, F. (2007). Convergence rates of general regularization methods for statistical inverse problems and applications. SIAM J. Numer. Anal. 45 2610-2636.
11. BLAKE, C. L. and MERZ, C. J. (1998). UCI repository of machine learning databases. Available at http://www.ics.uci.ed:u/~mlearn/ MLRepository.html.
12. BLANCHARD, G., LUGOSI, G. and VAYATIS, N. (2003). On the rate of convergence of regularized boosting classifiers. J. Machine Learning Research 4 861-894. MR2076000
13. BREIMAN, L. (1995). Better subset regression using the nonnegative garrote. Technometrics 37 373-384. MR1365720
14. BREIMAN, L. (1996). Bagging predictors. Machine Learning 24 123-140.
15. BREIMAN, L. (1998). Arcing classifiers (with discussion). Ann. Statist. 26 801-849. MR 1635406
16. BREIMAN, L. (1999). Prediction games and arcing algorithms. Neural Computation 11 1493-1517.
17. BREIMAN, L. (2001). Random forests. Machine Learning 45 5-32.
18. BÜHLMANN, P. (2006). Boosting for high-dimensional linear models. Ann. Statist. 34 559-583. MR2281878
19. BÜHLMANN, P. (2007). Twin boosting: Improved feature selection and prediction. Technical report, ETH Zürich. Available at ftp://ftp.stat.math.ethz.ch/Research-Reports/Other-Manuscripts/buhlmann/ TwinBoosting1.pdf.
20. BÜHLMANN, P. and LUTZ, R. (2006). Boosting algorithms: With an application to bootstrapping multivariate time series. In The Frontiers in Statistics (J. Fan and H. Koul, eds.) 209-230. Imperial College Press, London. MR2326003
21. BÜHLMANN, P. and YU, B. (2000). Discussion on "Additive logistic regression: A statistical view," by J. Friedman, T. Hastie and R. Tibshirani. Ann. Statist. 28 377-386.
22. 2 loss: Regression and classification. J. Amer. Statist. Assoc. 98 324-339. MR1995709
23. BÜHLMANN, P. and YU, B. (2006). Sparse boosting. J. Machine Learning Research 7 1001-1024. MR2274395
24. BUJA, A., STUETZLE, W. and SHEN, Y. (2005). Loss functions for binary class probability estimation: Structure and applications. Technical report, Univ. Washington. Available at http://www.stat.washington.edu/wxs/Learning-papers/paper-proper-scoring.pdf.
25. DETTLING, M. (2004). BagBoosting for tumor classification with gene expression data. Bioinformatics 20 3583-3593.
26. DETTLING, M. and BÜHLMANN, P. (2003). Boosting for tumor classification with gene expression data. Bioinformatics 19 1061-1069.
27. DIMARZIO, M. and TAYLOR, C. (2008). On boosting kernel regression. J. Statist. Plann. Inference. To appear.
28. EFRON, B., HASTIE, T., JOHNSTONE, I. and TIBSHIRANI, R. (2004). Least angle regression (with discussion). Ann. Statist. 32 407-499. MR2060166
29. FREUND, Y. and SCHAPIRE, R. (1995). A decision-theoretic generalization of on-line learning and an application to boosting. In Proceedings of the Second European Conference on Computational Learning Theory. Springer, Berlin.
30. FREUND, Y and SCHAPIRE, R. (1996). Experiments with a new boosting algorithm. In Proceedings of the Thirteenth International Conference on Machine Learning. Morgan Kaufmann, San Francisco, CA.
31. FREUND, Y and SCHAPIRE, R. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. System Sci. 55 119-139. MR1473055
32. FRIEDMAN, J. (2001). Greedy function approximation: A gradient boosting machine. Ann. Statist. 29 1189-1232. MR1873328
33. FRIEDMAN, J., HASTIE, T. and TIBSHIRANI, R. (2000). Additive logistic regression: A statistical view of boosting (with discussion). Ann. Statist. 28 337-407. MR1790002
34. GARCIA, A. L., WAGNER, K., HOTHORN, T., KOEBNICK, C., ZUNFT, H. J. and TRIPPO, U. (2005). Improved prediction of body fat by measuring skinfold thickness, circumferences, and bone breadths. Obesity Research 13 626-634.
35. GENTLEMAN, R. C., CAREY, V. J., BATES, D. M., BOLSTAD, B., DETTLING, M., DUDOIT, S., ELLIS, B., GAUTIER, L., GE, Y., GENTRY, J., HORNIK, K., HOTHORN, T., HUBER, M., IACUS, S., IRIZARRY, R., LEISCH, F., LI, C., MÄCHLER, M., ROSSINI, A. J., SAWITZKI, G., SMITH, C., SMYTH, G., TIERNEY, L., YANG, J. Y. and ZHANG, J. (2004). Bioconductor: Open software development for computational biology and bioinformatics. Genome Biology 5 R80.
36. GREEN, P. and SILVERMAN, B. (1994). Nonparametric Regression and Generalized Linear Models: A Roughness Penalty Approach. Chapman and Hall, New York. MR1270012
37. GREENSHTEIN, E. and RITOV, Y. (2004). Persistence in high-dimensional predictor selection and the virtue of overparametrization. Bernoulli 10 971-988. MR2108039
38. HANSEN, M. and YU, B. (2001). Model selection and minimum description length principle. J. Amer. Statist. Assoc. 96 746-774. MR 1939352
39. HASTIE, T. and EFRON, B. (2004). Lars: Least angle regression, lasso and forward stagewise. R package version 0.9-7. Available at http://CRAN.R-project.org.
40. HASTIE, T. and TIBSHIRANI, R. (1986). Generalized additive models (with discussion). Statist. Sci. 1 297-318. MR0858512
41. HASTIE, T. and TIBSHIRANI, R. (1990). Generalized Additive Models. Chapman and Hall, London. MR1082147
42. HASTIE, T., TIBSHIRANI, R. and FRIEDMAN, J. (2001). The Elements of Statistical Learning; Data Mining, Inference and Prediction. Springer, New York. MR1851606
43. HOTHORN, T. and BÜHLMANN, P. (2007). Mboost: Model-based boosting. R package version 0.5-8. Available at http://CRAN.R-project.org/.
44. HOTHORN, T. and BÜHLMANN, P. (2006). Model-based boosting in high dimensions. Bioinformatics 22 2828-2829.
45. HOTHORN, T., BÜHLMANN, P., DUDOIT, S., MOLINARO, A. and VAN DER LAAN, M. (2006). Survival ensembles. Biostatistics 7 355-373.
46. HOTHORN, T., HORNIK, K. and ZEILEIS, A. (2006). Party: A laboratory for recursive part(y)itioning. R package version 0.9-11. Available at http://CRAN.R-project.org/.
47. HOTHORN, T., HORNIK, K. and ZEILEIS, A. (2006). Unbiased recursive partitioning: A conditional inference framework. J. Comput. Graph. Statist. 15 651-674. MR2291267
48. HUANG, J., MA, S. and ZHANG, C-H. (2008). Adaptive Lasso for sparse high-dimensional regression. Statist. Sinica. To appear.
49. HURVICH, C., SIMONOFF, J. and TSAI, C.-L. (1998). Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. J. Roy. Statist. Soc. Ser. B 60 271-293. MR1616041
50. IYER, R., LEWIS, D., SCHAPIRE, R., SINGER, Y and SINGHAL, A. (2000). Boosting for document routing. In Proceedings of CIKM-00, 9th ACM Int. Conf on Information and Knowledge Management (A. Agah, J. Callan and E. Rundensteiner, eds.). ACM Press, New York.
51. JIANG, W. (2004). Process consistency for AdaBoost (with discussion). Ann. Statist. 32 13-29, 85-134. MR2050999
52. KEARNS, M. and VALIANT, L. (1994). Cryptographic limitations on learning Boolean formulae and finite automata. J. Assoc. Comput. Machinery 41 67-95. MR1369194
53. KOLTCHINSKII, V. and PANCHENKO, D. (2002). Empirical margin distributions and bounding the generalization error of combined classifiers. Ann. Statist. 30 1-50. MR1892654
54. LEITENSTORFER, F. and TUTZ, G. (2006). Smoothing with curvature constraints based on boosting techniques. In Proceedings in Computational Statistics (COMPSTAT) (A. Rizzi and M. Vichi, eds.). Physica-Verlag, Heidelberg.
55. LEITENSTORFER, F. and TUTZ, G. (2007). Generalized monotonie regression based on B-splines with an application to air pollution data. Biostatistics 8 654-673.
56. LEITENSTORFER, F. and TUTZ, G. (2007). Knot selection by boosting techniques. Comput. Statist. Data Anal. 514605-4621.
57. LOZANO, A., KULKARNI, S. and SCHAPIRE, R. (2006). Convergence and consistency of regularized boosting algorithms with stationary ß-mixing observations. In Advances in Neural Information Processing Systems (Y. Weiss, B. Schölkopf and J. Platt, eds.) 18. MIT Press.
58. LUGOSI, G. and VAYATIS, N. (2004). On the Bayes-risk consistency of regularized boosting methods (with discussion). Ann. Statist. 32 30-55, 85-134. MR2051000
59. LUTZ, R. and BÜHLMANN, P. (2006). Boosting for highmultivariate responses in high-dimensional linear regression. Statist. Sinica 16 471-494. MR2267246
60. MALLAT, S. and ZHANG, Z. (1993). Matching pursuits with time-frequency dictionaries. IEEE Transactions on Signal Processing 41 3397-3415.
61. MANNOR, S., MEIR, R. and ZHANG, T. (2003). Greedy algorithms for classification-consistency, convergence rates, and adaptivity. J. Machine Learning Research 4 713-741. MR2072266
62. MASON, L., BAXTER, J., BARTLETT, P. and FREAN, M. (2000). Functional gradient techniques for combining hypotheses. In Advances in Large Margin Classifiers (A. Smola, P. Bartlett, B. Schölkopf and D. Schuurmans, eds.) 221-246. MIT Press, Cambridge.
63. MCCAFFREY, D. F., RIDGEWAY, G. and MORRAL, A. R. G. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods 9 403-425.
64. MEASE, D., WYNER, A. and BUJA, A. (2007). Cost-weighted boosting with jittering and over/under-sampling: JOUS-boost. J. Machine Learning Research 8 409-439.
65. MEINSHAUSEN, N. and BÜHLMANN, P. (2006). High-dimensional graphs and variable selection with the Lasso. Ann. Statist. 34 1436-1462. MR2278363
66. MEIR, R. and RATSCH, G. (2003). An introduction to boosting and leveraging. In Advanced Lectures on Machine Learning (S. Mendelson and A. Smola, eds.). Springer, Berlin.
67. OSBORNE, M., PRESNELL, B. and TURLACH, B. (2000). A new approach to variable selection in least squares problems. IMA J. Numer. Anal. 20 389-403. MR1773265
68. PARK, M.-Y and HASTIE, T. (2007). An L1 regularizationpath algorithm, for generalized linear models. J. Roy. Statist. Soe. Ser. B 69 659-677.
69. R DEVELOPMENT CORE TEAM (2006). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Available at http://www.R-project.org.
70. RATSCH, G., ONODA, T. and MÜLLER, K. (2001). Soft margins for AdaBoost. Machine Learning 42 287-320.
71. RIDGEWAY, G. (1999). The state of boosting. Comput. Sci. Statistics 31 172-181.
72. RIDGEWAY, G. (2000). Discussion on "Additive logistic regression: A statistical view of boosting," by J. Friedman, T. Hastie, R. Tibshirani. Ann. Statist. 28 393-400.
73. RIDGEWAY, G. (2002). Looking for lumps: Boosting and bagging for density estimation. Comput. Statist. Data Anal. 38 379-392. MR1884870
74. RIDGEWAY, G. (2006). Gbm: Generalized boosted regression models. R package version 1.5-7. Available at http://www.i-pensieri.com/gregr/gbm. shtml.
75. SCHAPIRE, R. (1990). The strength of weak learnability. Machine Learning S 197-227.
76. SCHAPIRE, R. (2002). The boosting approach to machine learning: An overview. Nonlinear Estimation and Classification. Lecture Notes in Statist. 171 149-171. Springer, New York. MR2005788
77. SCHAPIRE, R., FREUND, Y., BARTLETT, P. and LEE, W. (1998). Boosting the margin: A new explanation for the effectiveness of voting methods. Ann. Statist. 26 1651-1686. MR1673273
78. SCHAPIRE, R. and SINGER, Y. (2000). Boostexter: A boosting-based system for text categorization. Machine Learning 39 135-168.
79. SOUTHWELL, R. (1946). Relaxation Methods in Theoretical Physics. Oxford, at the Clarendon Press. MR0018983
80. STREET, W. N., MANGASARIAN, O. L., and WOLBERG, W. H. (1995). An inductive learning approach to prognostic prediction. In Proceedings of the Twelfth International Conference on Machine Learning. Morgan Kaufmann, San Francisco, CA.
81. TEMLYAKOV, V. (2000). Weak greedy algorithms. Adv. Comput. Math. 12 213-227. MR1745113
82. TIBSHIRANI, R. (1996). Regression shrinkage and selection via the Lasso. J. Roy. Statist. Soe. Ser. B 58 267-288. MR1379242
83. TUKEY, J. (1977). Exploratory Data Analysis. Addison-Wesley, Reading, MA.
84. TUTZ, G. and BINDER, H. (2006). Generalized additive modelling with implicit variable selection by likelihood based boosting. Biometrics 62 961-971. MR2297666
85. TUTZ, G. and BINDER, H. (2007). Boosting Ridge regression. Comput. Statist. Data Anal. 51 6044-6059.
86. TUTZ, G. and HECHENBICHLER, K. (2005). Aggregating classifiers with ordinal response structure. J. Statist. Comput. Simul. 75 391-408. MR2136546
87. TUTZ, G. and LEITENSTORFER, F. (2007). Generalized smooth monotonie regression in additive modelling. J. Comput. Graph. Statist. 16 165-188.
88. TUTZ, G. and REITHINGER, F. (2007). Flexible semiparametric mixed models. Statistics in Medicine 26 2872-2900.
89. VAN DER LAAN, M. and ROBINS, J. (2003). Unified Methods for Censored Longitudinal Data and Causality. Springer, New York. MR1958123
90. WEST, M., BLANCHETTE, C., DRESSMAN, H., HUANG, E., ISHIDA, S., SPANG, R., ZUZAN, H., OLSON, J., MARKS, J. and NEVINS, J. (2001). Predicting the clinical status of human breast cancer by using gene expression profiles. Proc. Natl. Acad. Sci. USA 98 11462-11467.
91. YAO, Y., ROSASCO, L. and CAPONNETTO, A. (2007). On early stopping in gradient descent learning. Constr. Approx. 26 289-315. MR2327601
92. ZHANG, T. and YU, B. (2005). Boosting with early stopping: Convergence and consistency. Ann. Statist. 33 1538-1579. MR2166555
93. ZHAO, P. and YU, B. (2007). Stagewise Lasso. J. Mach. Learn. Res. 8 2701-2726.
94. ZHAO, P. and YU, B. (2006). On model selection consistency of Lasso. J. Machine Learning Research 7 2541-2563. MR2274449
95. ZHU, J., ROSSET, S., ZOU, H. and HASTIE, T. (2005). Multiclass AdaBoost. Technical report, Stanford Univ. Available athttp://www-stat.stanford.ed.u/~hastie/Papers/samm.e.pdf.
96. ZOU, H. (2006). The adaptive Lasso and its oracle properties. J. Amer. Statist. Assoc. 1011418-1429. MR2279469