Extending statistical boosting: An overview of recent methodological developments

Methods of Information in Medicine

Volumn 53, Issue 6, 2014, Pages 428-435

  Mayr, Andreas ^a   Binder, H ^b   Gefeller, O ^a   Schmid, M ^a,c  

^a UNIVERSITY OF ERLANGEN NUREMBERG   (Germany)

^b JOHANNES GUTENBERG UNIVERSITY   (Germany)

^c UNIVERSITY OF BONN   (Germany)

Author keywords

Algorithms; Classification; Machine learning; Statistical computing; Statistical models

Indexed keywords

ALGORITHM; HUMAN; MATHEMATICAL COMPUTING; MEDICAL RESEARCH; REGRESSION ANALYSIS; STATISTICAL MODEL; STATISTICS AND NUMERICAL DATA;

ALGORITHMS; BIOMEDICAL RESEARCH; HUMANS; LIKELIHOOD FUNCTIONS; MATHEMATICAL COMPUTING; MODELS, STATISTICAL; REGRESSION ANALYSIS;

EID: 84914104651     PISSN: 00261270     EISSN: 2511705X     Source Type: Journal    
DOI: 10.3414/ME13-01-0123     Document Type: Article

References (86)

1. Schapire RE. The Strength of Weak Learnability. Machine Learning 1990; 5 (2): 197-227
2. Freund Y. Boosting a Weak Learning Algorithm by Majority. In: Fulk MA, Case J, editors. Proceedings of the Third Annual Workshop on Computational Learning Theory, COLT 1990, University of Rochester, Rochester, NY, USA, August 6–8, 1990; 1990. pp 202-216
3. Freund Y, Schapire R. Experiments With a New Boosting Algorithm. In: Proceedings of the Thirteenth International Conference on Machine Learning Theory. San Francisco, CA: San Francisco: Morgan Kaufmann Publishers Inc.; 1996. pp 148-156
4. Mayr A, Binder H, Gefeller O, Schmid M. The Evolution of Boosting Algorithms-From Machine Learning to Statistical Modelling. Methods Inf Med 2014; 53: 419-427
5. Hastie T, Tibshirani R. Generalized Additive Models. London: Chapman & Hall; 1990
6. Eilers PHC, Marx BD. Flexible Smoothing with B-splines and Penalties (with discussion). Statistical Science 1996; 11: 89-121
7. Kruppa J, Liu Y, Biau G, Kohler M, König IR, Malley JD, et al. Probability Estimation with Machine Learning Methods for Dichotomous and Multi-Category Outcome: Theory. Biometrical Journal 2014. Available from: http://dx.doi.org/10.1002/ bimj.201300068
8. Boulesteix AL, Schmid M. Discussion: Machine Learning Versus Statistical Modeling. Biometrical Journal 2014. Available from: http://dx.doi.org/ 10.1002/bimj.201300226
9. Friedman JH. Greedy Function Approximation: A Gradient Boosting Machine. The Annals of Statistics 2001; 29: 1189-1232
10. Bühlmann P, Hothorn T. Boosting Algorithms: Regularization, Prediction and Model Fitting (with Discussion). Statistical Science 2007; 22: 477–522
11. Tutz G, Binder H. Generalized Additive Modeling with Implicit Variable Selection by Likelihoodbased Boosting. Biometrics 2006; 62: 961-971
12. Tutz G, Binder H. Boosting Ridge Regression. Computational Statistics & Data Analysis. 2007; 51 (12): 6044-6059
13. Kneib T, Hothorn T, Tutz G. Variable Selection and Model Choice in Geoadditive Regression Models. Biometrics 2009; 65 (2): 626-634
14. Fan J, Lv J. A Selective Overview of Variable Selection in High Dimensional Feature Space. Statistica Sinica 2010; 20: 101–148
15. Bühlmann P, Yu B. Sparse Boosting. Journal of Machine Learning Research 2007; 7: 1001–1024
16. Bühlmann P. Bagging, Boosting and Ensemble Methods. In: Gentle YJE, Härdle W, Mori Y, editors. Handbook of Computational Statistics. Springer Handbooks; 2012. pp 985-1022
17. Hansen MH, Yu B. Model Selection and the Principle of Minimum Description Length. Journal of the American Statistical Association 2001; 96 (454): 746-774
18. Hurvich CM, Tsai CL.Regression and Time Series Model Selection in Small Samples. Biometrika 1989; 76 (2): 237-397
19. Greven S, Kneib T. On the Behaviour of Marginal and Conditional Akaike Information Criteria in Linear Mixed Models. Biometrika 2010; 97 (4): 773-789
20. Hastie T. Comment: Boosting Algorithms: Regularization, Prediction and Model Fitting. Statistical Science 2007; 22 (4): 513-515
21. Mayr A, Hofner B, Schmid M. The Importance of Knowing When to Stop-A Sequential Stopping Rule for Component-Wise Gradient Boosting. Methods Inf Med 2012; 51 (2): 178-186
22. Chang YCI, Huang Y, Huang YP. Early Stopping in L2 Boosting. Comput Stat Data Anal 2010; 54 (10): 2203-2213
23. Bühlmann P, Hothorn T. Twin Boosting: Improved Feature Selection and Prediction. Statistics and Computing 2010; 20 (2): 119-138
24. Meinshausen N, Bühlmann P. Stability Selection (with Discussion). Journal of the Royal Statistical Society Series B 2010; 72: 417-473
25. Shah RD, Samworth RJ. Variable Selection with Error Control: Another Look at Stability Selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2013; 75 (1): 55-80
26. Hothorn T. Discussion: Stability Selection. Journal of the Royal Statistical Society Series B 2010; 72: 463-464
27. Sauerbrei W, Schumacher W. A Bootstrap Resampling Procedure for Model-Building-Application to the Cox Regression-Model. Statistics in Medicine 1992; 11: 2093-2109
28. Bühlmann P. Boosting for High-Dimensional Linear Models. The Annals of Statistics 2006; 34: 559-583
29. Schmid M, Hothorn T, Krause F, Rabe C. A PAUCbased Estimation Technique for Disease Classification and Biomarker Selection. Statistical Applications in Genetics and Molecular Biology 2012; 11 (5), doi: 10.1515/1544-6115.1792
30. Schmid M, Hothorn T. Boosting Additive Models Using Component-Wise P-splines. Computational Statistics & Data Analysis 2008; 53: 298-311
31. Bühlmann P, Yu B. Boosting with the L2 Loss: Regression and Classification. Journal of the American Statistical Association 2003; 98: 324-338
32. Kneib T, Müller J, Hothorn T. Spatial Smoothing Techniques for the Assessment of Habitat Suitability. Environmental and Ecological Statistics 2008; 15: 343-364
33. Robinzonov N, Hothorn T. Boosting for Estimating Spatially Structured Additive Models. In: Kneib T, Tutz G, editors. Statistical Modelling and Regression Structures. Springer; 2010. pp 181–196
34. Sobotka F, Kneib T. Geoadditive Expectile Regression. Computational Statistics and Data Analysis 2012; 56: 755-767
35. Groll A, Tutz G. Regularization for Generalized Additive Mixed Models by Likelihood-based Boosting. Methods Inf Med 2012; 51 (2): 168-177
36. Hofner B, Kneib T, Hothorn T. A Unified Framework of Constrained Regression. arXiv preprint 2014. Available from: http://arxiv.org/abs/14037118.
37. Leitenstorfer F, Tutz G. Generalized Monotonic Regression based on B-splines with an Application to Air Pollution Data. Biostatistics 2007; 8 (3): 654-673
38. Hofner B, Hothorn T, Kneib T, Schmid M. A Framework for Unbiased Model Selection Based on Boosting. Journal of Computational and Graphical Statistics 2011; 20: 956-971
39. Buja A, Hastie T, Tibshirani R. Linear Smoothers and Additive Models. The Annals of Statistics 1989; 17 (2): 453-510
40. Gertheiss J, Tutz G. Penalized Regression with Ordinal Predictors. International Statistical Review 2009; 77: 345-365
41. Tutz G, Gertheiss J. Feature Extraction in Signal Regression: A Boosting Technique for Functional Data Regression. Journal of Computational and Graphical Statistics 2010; 19: 154-174
42. Ferraty F, Vieu P. Additive Prediction and Boosting for Functional Data. Computational Statistics & Data Analysis 2009; 53 (4): 1400-1413
43. Gertheiss J, Hogger S, Oberhauser C, Tutz G. Selection of Ordinally Scaled Independent Variables with Applications to International Classification of Functioning Core Sets. Applied Statistics 2010; 60 (3): 377–395
44. Tutz G, Ulbricht J. Penalized Regression with Correlation-Based Penalty. Statistical Computing 2008; 19: 239-253
45. Zou H, Hastie T. Regularization and Variable Selection via the Elastic Net. Journal of the Royal Statistical Society, Series B 2005; 67: 301–320
46. Hothorn T, Bühlmann P, Dudoit S, Molinaro A, Van Der Laan MJ. Survival Ensembles. Biostatistics 2006; 7 (3): 355-373
47. Ridgeway G. The State of Boosting. Computing Science and Statistics 1999; 31: 172-181
48. Binder H, Schumacher M. Allowing for Mandatory Covariates in Boosting Estimation of Sparse-High-Dimensional Survival Models. BMC Bioinformatics 2008; 9 (14), doi: 10.1186/1471-2105-9-14
49. Hofner B, Hothorn T, Kneib T. Variable Selection and Model Choice in Structured Survival Models. Computational Statistics 2013; 28 (3): 1079-1101
50. Binder H, Allignol A, Schumacher M, Beyersmann J. Boosting for High-Dimensional Time-to-Event Data with Competing Risks. Bioinformatics 2009; 25 (7): 890-896
51. Schmid M, Hothorn T. Flexible Boosting of Accelerated Failure Time Models. BMC Bioinformatics 2008; 9 (269)
52. Schmid M, Potapov S, Pfahlberg A, Hothorn T. Estimation and Regularization Techniques for Regression Models with Multidimensional Pre-diction Functions. Statistics and Computing 2010; 20: 139-150
53. Ma S, Huang J, Xie Y, Yi N. Identification of Breast Cancer Prognosis Markers Using Integrative Sparse Boosting. Methods Inf Med 2012; 51 (2): 152-161
54. Johnson BA, Long Q. Survival Ensembles by the Sum of Pairwise Differences with Application to Lung Cancer Microarray Studies. The Annals of Applied Statistics 2011; 5 (2A): 1081–1101
55. Wang Z, Wang C. Buckley-James Boosting for Survival Analysis with High-Dimensional Biomarker Data. Statistical Applications in Genetics and Molecular Biology 2010; 9 (1)
56. Schmid M, Hothorn T, Maloney KO, Weller DE, Potapov S. Geoadditive Regression Modeling of Stream Biological Condition. Environmental and Ecological Statistics 2011; 18: 709-733
57. Mayr A, Fenske N, Hofner B, Kneib T, Schmid M. Generalized Additive Models for Location, Scale and Shape for High-Dimensional Data-A Flexible Aproach Based on Boosting. Journal of the Royal Statistical Society: Series C (Applied Statistics) 2012; 61 (3): 403-427
58. Rigby RA, Stasinopoulos DM. Generalized Additive Models for Location, Scale and Shape (with discussion). Applied Statistics 2005; 54: 507-554
59. Schmid M, Wickler F, Maloney KO, Mitchell R, Fenske N, Mayr A. Boosted Beta Regression. PloS ONE 2013; 8 (4): e61623
60. Kneib T. Beyond Mean Regression. Statistical Modelling 2013; 13 (4): 275-303
61. Fenske N, Kneib T, Hothorn T. Identifying Risk Factors for Severe Childhood Malnutrition by Boosting Additive Quantile Regression. Journal of the American Statistical Association 2011; 106 (494): 494-510
62. Mayr A, Hothorn T, Fenske N. Prediction Intervals for Future BMI Values of Individual Children-A Non-Parametric Approach by Quantile Boosting. BMC Medical Research Methodology 2012; 12 (6), doi: 10.1186/1471-2288-12-6
63. Hothorn T, Kneib T, Bühlmann P. Conditional Transformation Models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2014; 76 (1): 3-27
64. Dettling M, Bühlmann P. Boosting for Tumor Classification with Gene Expression Data. Bioinformatics 2003; 19 (9): 1061–1069
65. Boulesteix AL, Hothorn T. Testing the Additional Predictive Value of High-Dimensional Molecular Data. BMC Bioinformatics 2010; 11 (78)
66. Binder H, Schumacher M. Incorporating Pathway Information into Boosting Estimation of High-Dimensional Risk Prediction Models. BMC Bio-informatics 2009; 10 (18)
67. Gade S, Porzelius C, Fälth M, Brase JC, Wuttig D, Kuner R, et al. Graph based Fusion of miRNA and mRNA Expression Data Improves Clinical Outcome Prediction in Prostate Cancer. BMC Bio-informatics 2011; 12 (488)
68. Binder H, Benner A, Bullinger L, Schumacher M. Tailoring Sparse Multivariable Regression Techniques for Prognostic Single-Nucleotide Poly-morphism Signatures. Statistics in Medicine 2013; 32 (10): 1778-1791
69. Zwiener I, Frisch B, Binder H. Transforming RNASeq Data to Improve the Performance of Prognostic Gene Signatures. PLoS ONE 2014; 9 (1): e85150
70. Binder H, Müller T, Schwender H, Golka K, Steffens M, Hengstler JG, et al. Cluster-localized sparse logistic regression for SNP data. Statistical Applications ind Genetics and Molecular Biology 2012; 11 (4)
71. Ma S, Huang J. Regularized ROC Method for Disease Classification and Biomarker Selection with Microarray Data. Bioinformatics 2005; 21 (24): 4356-4362
72. Wang Z. HingeBoost: ROC-Based Boost for Classification and Variable Selection. The International Journal of Biostatistics 2011; 7 (1): 1–30
73. Steck H. Hinge Rank Loss and the Area Under the ROC Curve. In: Machine Learning: ECML 2007. Springer; 2007. pp 347–358
74. Wang Z. Multi-class HingeBoost. Method and Application to the Classification of Cancer Types Using Gene Expression Data. Methods Inf Med 2012; 51 (2): 162–167
75. Komori O, Eguchi S. A Boosting Method for Maximizing the Partial Area Under the ROC Curve. BMC Bioinformatics 2010; 11 (314)
76. Wang Z, Chang YCI. Marker Selection via Maximizing the Partial Area Under the ROC Curve of Linear Risk Scores. Biostatistics 2011; 12 (2): 369-385
77. Harrell FE, Califf RM, Pryor DB, Lee KL, Rosati RA. Evaluating the Yield of Medical Tests. Journal of the American Medical Association 1982; 247 (18): 2543-2546
78. Heagerty PJ, Zheng Y. Survival Model Predictive Accuracy and ROC Curves. Biometrics 2005; 61 (1): 92–105
79. Chen Y, Jia Z, Mercola D, Xie X. A Gradient Boosting Algorithm for Survival Analysis via Direct Optimization of Concordance Index. Computational and Mathematical Methods in Medicine 2013. Available from: http://dx.doi.org/10.1155/2013/ 873595
80. Mayr A, Schmid M. Boosting the Concordance Index for Survival Data-A Unified Framework to Derive and Evaluate Biomarker Combinations. PloS ONE 2014; 9 (1): e84483
81. Uno H, Cai T, Pencina MJ, D’Agostino RB, Wei LJ. On the C-statistics for Evaluating Overall Adequacy of Risk Prediction Procedures with Censored Survival Data. Statistics in Medicine 2011; 30 (10): 1105-1117
82. Schmid M, Potapov S. A Comparison of Estimators to Evaluate the Discriminatory Power of Time-to-Event Models. Statistics in Medicine 2012; 31 (23): 2588-2609
83. Reiser V, Porzelius C, Stampf S, Schumacher M, Binder H. Can Matching Improve the Performance of Boosting for Identifying Important Genes in Observational Studies? Computational Statistics 2013; 28 (1): 37-49
84. Rücker G, Reiser V, Motschall E, Binder H, Meerpohl JJ, Antes G, et al. Boosting Qualifies Capture-Recapture Methods for Estimating the Comprehensiveness of Literature Searches for Systematic Reviews. Journal of Clinical Epidemiology 2011; 64 (12): 1364-1372
85. Fenske N, Burns J, Hothorn T, Rehfuess EA. Understanding Child Stunting in India: A Comprehensive Analysis of Socio-Economic, Nutritional and Environmental Determinants Using Additive Quantile Regression. PloS ONE 2013; 8 (11): e78692
86. Faschingbauer F, Beckmann M, Goecke T, Yazdi B, Siemer J, Schmid M, et al. A New Formula for Optimized Weight Estimation in Extreme Fetal Macrosomia (_ 4500 g). European Journal of Ultrasound 2012; 33 (05): 480-488.