A calibration hierarchy for risk models was defined: From utopia to empirical data

Journal of Clinical Epidemiology

Volumn 74, Issue , 2016, Pages 167-176

  Van Calster, Ben ^a,b   Nieboer, Daan ^b   Vergouwe, Yvonne ^b   De Cock, Bavo ^a   Pencina, Michael J ^c,d   Steyerberg, Ewout W ^b  

^a UNIVERSITY OF LEUVEN   (Belgium)

^b ERASMUS MEDICAL CENTER   (Netherlands)

^c DUKE CLINICAL RESEARCH INSTITUTE   (United States)

^d DUKE UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

Calibration; Decision curve analysis; External validation; Loess; Overfitting; Risk prediction models

Indexed keywords

ARTICLE; CALIBRATION; DECISION MAKING; DECISION SUPPORT SYSTEM; HUMAN; NONLINEAR SYSTEM; PREDICTION; PREDICTOR VARIABLE; PREVALENCE; PRIORITY JOURNAL; SAMPLE SIZE; COMPUTER SIMULATION; REPRODUCIBILITY; RISK; RISK ASSESSMENT; STATISTICAL BIAS; STATISTICAL MODEL;

BIAS (EPIDEMIOLOGY); CALIBRATION; COMPUTER SIMULATION; DECISION SUPPORT TECHNIQUES; HUMANS; MODELS, STATISTICAL; REPRODUCIBILITY OF RESULTS; RISK; RISK ASSESSMENT;

EID: 84957076656     PISSN: 08954356     EISSN: 18785921     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2015.12.005     Document Type: Article

References (37)

1. G.S. Collins, J.A. de Groot, S. Dutton, O. Omar, M. Shanyinde, A. Tajar, and et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting BMC Med Res Methodol 14 2014 40
2. E.W. Steyerberg Clinical prediction models. A practical approach to development, validation, and updating 2009 Springer-Verlag New York
3. K.I. Kim, and R. Simon Probabilistic classifiers with high-dimensional data Biostatistics 12 2011 399 412
4. M.S. Pepe, and H. Janes Methods for evaluating prediction performance of biomarkers and tests M.L.T. Lee, M. Gail, R. Pfeiffer, G. Satten, T. Cai, A. Gandy, Risk Assessment and Evaluation of Predictions 2013 Springer-Verlag New York 107 142
5. W. Vach Calibration of clinical prediction rules does not just assess bias J Clin Epidemiol 66 2013 1296 1301
6. F.E. Harrell Jr. Regression modeling strategies. With applications to linear models, logistic regression, and survival analysis 2001 Springer-Verlag New York
7. D.W. Hosmer, T. Hosmer, S. Le Cessie, and S. Lemeshow A comparison of goodness-of-fit tests for the logistic regression model Stat Med 16 1997 965 980
8. D.P. Ankerst, A. Boeck, S.J. Freedland, I.M. Thompson, A.M. Cronin, M.J. Roobol, and et al. Evaluating the PCPT risk calculator in ten international biopsy cohorts: results from the Prostate Biopsy Collaborative Group World J Urol 30 2012 181 187
9. D.R. Cox Two further applications of a model for binary regression Biometrika 45 1958 562 565
10. M.E. Miller, C.D. Langefeld, W.M. Tierney, S.L. Hui, and C.J. McDonald Validation of probabilistic predictions Med Decis Making 13 1993 49 58
11. P.C. Austin, and E.W. Steyerberg Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers Stat Med 33 2014 517 535
12. G.S. Collins, E.O. Ogundimu, and D.G. Altman Sample size considerations for the external validation of a multivariable prognostic model: a resampling study Stat Med 35 2016 214 226
13. E.W. Steyerberg, A.J. Vickers, N.R. Cook, T. Gerds, M. Gonen, N. Obuchowski, and et al. Assessing the performance of prediction models: a framework for traditional and novel measures Epidemiology 21 2010 128 138
14. A.J. Vickers, and A.M. Cronin Everything you always wanted to know about evaluating prediction models (but were too afraid to ask) Urology 76 2010 1298 1301
15. P.C. Austin, and E.W. Steyerberg Bootstrap confidence intervals for loess-based calibration curves Stat Med 33 2014 2699 2700
16. K. Van Hoorde, S. Van Huffel, D. Timmerman, T. Bourne, and B. Van Calster A spline-based tool to assess and visualize the calibration of multiclass risk predictions J Biomed Inform 54 2015 283 293
17. A. Decarli, S. Calza, G. Masala, C. Specchia, D. Palli, and M.H. Gail Gail model for prediction of absolute risk of invasive breast cancer: independent evaluation in the Florence-European Prospective Investigation into Cancer and Nutrition cohort J Natl Cancer Inst 98 2006 1686 1693
18. B. Van Calster, and A.J. Vickers Calibration of risk prediction models: impact on decision-analytic performance Med Decis Making 35 2015 162 169
19. B. Van Calster, A.J. Vickers, M.J. Pencina, S.G. Baker, D. Timmerman, and E.W. Steyerberg Evaluation of markers and risk prediction models: overview of relationships between NRI and decision-analytic measures Med Decis Making 33 2013 490 501
20. A.J. Vickers, and E.B. Elkin Decision curve analysis: a novel method for evaluating prediction models Med Decis Making 26 2006 565 574
21. S.G. Pauker, and J.P. Kassirer Therapeutic decision making: a cost-benefit analysis N Engl J Med 293 1975 229 234
22. C. Chatfield Model uncertainty, data mining and statistical inference J R Stat Soc Ser A 158 1995 419 466
23. S. Nemes, J.M. Jonasson, A. Genell, and G. Steineck Bias in odds ratios by logistic regression modelling and sample size BMC Med Res Methodol 9 2009 56
24. P.C. Austin, and E.W. Steyerberg The number of subjects per variable required in linear regression analyses J Clin Epidemiol 68 2015 627 636
25. E.W. Steyerberg, M.J.C. Eijkemans, and J.D.F. Habbema Application of shrinkage techniques in logistic regression analysis: a case study Stat Neerlandica 55 2001 76 88
26. J.C. Van Houwelingen Shrinkage and penalized likelihood as methods to improve predictive accuracy Stat Neerlandica 55 2001 17 34
27. G. Ambler, S. Seaman, and R.Z. Omar An evaluation of penalised survival methods for developing prognostic models with rare events Stat Med 31 2012 1150 1161
28. W.A. Fuller Measurement error models 1987 John Wiley & Sons New York
29. R.J. Carroll, A. Delaigle, and P. Hall Nonparametric prediction in measurement error models J Am Stat Assoc 104 2009 993 1014
30. W. Vach Regression models as a tool in medical research 2013 Chapman and Hall/CRC Boca Raton
31. M. Ennis, G. Hinton, D. Naylor, M. Revow, and R. Tibshirani A comparison of statistical learning methods on the Gusto database Stat Med 17 1998 2501 2508
32. P. Royston, D.G. Altman, and W. Sauerbrei Dichotomizing continuous predictors in multiple regression: a bad idea Stat Med 25 2006 127 141
33. J.N. Marewski, and G. Gigerenzer Heuristic decision making in medicine Dialogues Clin Neurosci 14 2012 77 89
34. E.W. Steyerberg, G.J. Borsboom, H.C. van Houwelingen, M.J. Eijkemans, and J.D. Habbema Validation and updating of predictive logistic regression models: a study on sample size and shrinkage Stat Med 23 2004 2567 2586
35. K. Van Hoorde, Y. Vergouwe, D. Timmerman, S. Van Huffel, E.W. Steyerberg, and B. Van Calster Simple dichotomous updating methods improved the validity of polytomous prediction models J Clin Epidemiol 66 2013 1158 1165
36. Y. Vergouwe, E.W. Steyerberg, M.J. Eijkemans, and J.D. Habbema Substantial effective sample sizes were required for external validation studies of predictive logistic regression models J Clin Epidemiol 58 2005 475 483
37. D.W. Hosmer, and N.L. Hjort Goodness-of-fit processes for logistic regression: simulation results Stat Med 21 2002 2723 2738