A review of statistical updating methods for clinical prediction models

Statistical Methods in Medical Research

Volumn 27, Issue 1, 2018, Pages 185-197

  Su, Ting Li ^a   Jaki, Thomas ^b   Hickey, Graeme L ^c   Buchan, Iain ^d   Sperrin, Matthew ^d  

^a UNIVERSITY OF MANCHESTER   (United Kingdom)

^b LANCASTER UNIVERSITY   (United Kingdom)

^c UNIVERSITY OF LIVERPOOL   (United Kingdom)

^d UNIVERSITY OF MANCHESTER   (United Kingdom)

Author keywords

Clinical prediction model; Model recalibration; Model updating; Model validation; Risk score

Indexed keywords

ARTICLE; CALIBRATION; HEART SURGERY; HUMAN; MORTALITY; PREDICTION; STATISTICAL ANALYSIS; UNITED KINGDOM; VALIDATION PROCESS; FORECASTING; REGISTER; REGRESSION ANALYSIS; REPRODUCIBILITY; STATISTICAL MODEL;

CARDIAC SURGICAL PROCEDURES; FORECASTING; HUMANS; MODELS, STATISTICAL; REGISTRIES; REGRESSION ANALYSIS; REPRODUCIBILITY OF RESULTS; UNITED KINGDOM;

EID: 85041430479     PISSN: 09622802     EISSN: 14770334     Source Type: Journal    
DOI: 10.1177/0962280215626466     Document Type: Article

References (54)

1. Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. New York: Springer, 2008.
2. Hingorani AD, van Windt DA, Riley RD, et al. Prognosis research strategy (PROGRESS) 4: Stratified medicine research. Br Med J 2013; 346: e5793.
3. Beattie P, Nelson R, et al. Clinical prediction rules: What are they and what do they tell us? Aust J Physiother 2006; 52: 157–163.
4. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21: 128–138.
5. Lovegrove J, Valencia O, Treasure T, et al. Monitoring the results of cardiac surgery by variable life-adjusted display. Lancet 1997; 350: 1128–1130.
6. Rogers CA, Reeves BC, Caputo M, et al. Control chart methods for monitoring cardiac surgical performance and their interpretation. J Thorac Cardiovasc Surg 2004; 128: 811–819.
7. Hemingway H, Croft P, Perel P, et al. Prognosis research strategy (PROGRESS) 1: A framework for researching clinical outcomes. BMJ 2013; 346: e5595.
8. Riley RD, Hayden JA, Steyerberg EW, et al. Prognosis research strategy (PROGRESS) 2: Prognostic factor research. PLoS Med 2013; 10: e1001380.
9. Steyerberg EW, Moons KG, van der Windt DA, et al. Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med 2013; 10: e1001381.
10. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement. Ann Intern Med 2015; 162: 55–63.
11. Bridgewater B, Keogh B, Kinsman R, et al. The Society for Cardiothoracic Surgery in Great Britain & Ireland, 6th national adult cardiac surgical database report; demonstrating quality, 2008.
12. Janssen K, Moons K, Kalkman C, et al. Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 2008; 61: 76–86.
13. Hickey GL, Grant SW, Caiado C, et al. Dynamic prediction modeling approaches for cardiac surgery. Circ Cardiovasc Qual Outcomes 2013; 6: 649–658.
14. Altman DG, Vergouwe Y, Royston P, et al. Prognosis and prognostic research: validating a prognostic model. BMJ 2009; 338: 1432–1435.
15. Fawcett T. An introduction to ROC analysis. Pattern Recognit Lett 2006; 27: 861–874.
16. Hanley JA and McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143: 29–36.
17. Gengsheng Q and Hotilovac L. Comparison of non-parametric confidence intervals for the area under the ROC curve of a continuous-scale diagnostic test. Stat Methods Med Res 2008; 17: 207–221.
18. Cortes C and Mohri M. Confidence intervals for the area under the ROC curve. Adv Neural Inf Process Syst 2005; 17: 305–312.
19. DeLong ER, DeLong DM and Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach on JSTOR. Biometrics 1988; 44: 837–844.
20. Hosmer DW and Lemeshow S. Goodness of fit tests for the multiple logistic regression model. Commun Stat Methods 1980; 9: 1043–1069.
21. Cox DR. Two futher applications of a model for binary regression. Biometrika 1958; 45: 562–565.
22. Austin PC and Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 2014; 33: 517–535.
23. Steyerberg EW, Borsboom GJJM, van Houwelingen HC, et al. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med 2004; 23: 2567–2586.
24. Janssen KJM, Kalkman CJ, Grobbee DE, et al. The risk of severe postoperative pain: modification and validation of a clinical prediction rule. Anesth Analg 2008; 107: 1330–1339.
25. Van Hoorde K, Vergouwe Y, Timmerman D, et al. Simple dichotomous updating methods improved the validity of polytomous prediction models. J Clin Epidemiol 2013; 66: 1158–1165.
26. Van Houwelingen HC and Thorogood J. Construction, validation and updating of a prognostic model for kidney graft survival. Stat Med 1995; 14: 1999–2008.
27. Van Houwelingen HC. Validation, calibration, revision and combination of prognostic survival models. Stat Med 2000; 19: 3401–3415.
28. Janssen KJM, Vergouwe Y, Kalkman CJ, et al. A simple method to adjust clinical prediction models to local circumstances. Can J Anaesth 2009; 56: 194–201.
29. Harrell FE, Lee KL and Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996; 15: 361–387.
30. Kappen TH, Vergouwe Y, van Klei WA, et al. Adaptation of clinical prediction models for application in local settings. Med Decis Making 2012; 32: E1–E10.
31. Shahian DM, O’Brien SM, Filardo G, et al. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1– coronary artery bypass grafting surgery. Ann Thorac Surg 2009; 88(1 Suppl): S2–S22.
32. O’Brien SM, Shahian DM, Filardo G, et al. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 2– isolated valve surgery. Ann Thorac Surg 2009; 88(1 Suppl): S23–S42.
33. Shahian DM, O’Brien SM, Filardo G, et al. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 3– valve plus coronary artery bypass grafting surgery. Ann Thorac Surg 2009; 88(1 Suppl): S43–S62.
34. Debray TPA, Koffijberg H, Vergouwe Y, et al. Aggregating published prediction models with individual participant data: a comparison of different approaches. Stat Med 2012; 31: 2697–2712.
35. Debray TPA, Moons KGM, Ahmed I, et al. A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis. Stat Med 2013; 32: 3158–3180.
36. Debray TPA, Koffijberg H, Nieboer D, et al. Meta-analysis and aggregation of multiple published prediction models. Stat Med 2014; 33: 2341–2362.
37. Raftery AE, Ka´rný M and Ettler P. Online prediction under model uncertainty via dynamic model averaging: Application to a cold rolling mill. Technometrics 2010; 52: 52–66.
38. McCormick TH, Raftery AE, Madigan D, et al. Dynamic logistic regression and dynamic model averaging for binary classification. Biometrics 2012; 68: 23–30.
39. Raftery AE, Gneiting T, Balabdaoui F, et al. Using Bayesian model averaging to calibrate forecast ensembles. Mon Weather Rev 2005; 133: 1155–1174.
40. Strobl AN, Vickers AJ, van Calster B, et al. Improving patient prostate cancer risk assessment: Moving from static, globally-applied to dynamic, practice-specific cancer risk calculators. J Biomed Inform 2015; 56: 87–93.
41. Roques F. The logistic EuroSCORE. Eur Heart J 2003; 24: 882–883.
42. Nashef SAM, Roques F, Michel P, et al. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardio-Thoracic Surg 1999; 16: 9–13.
43. Hickey GL, Grant SW, Murphy GJ, et al. Dynamic trends in cardiac surgery: why the logistic EuroSCORE is no longer suitable for contemporary cardiac surgery and implications for future risk models. Eur J Cardio-Thoracic Surg 2013; 43: 1146–1152.
44. Hickey GL, Grant SW, Murphy GJ, et al. Dynamic trends in cardiac surgery: why the logistic EuroSCORE is no longer suitable for contemporary cardiac surgery and implications for future risk models. Eur J Cardio-Thoracic Surg 2013; 43: 1146–1152.
45. Grant SW, Hickey GL, Dimarakis I, et al. How does EuroSCORE II perform in UK cardiac surgery; an analysis of 23 740 patients from the Society for Cardiothoracic Surgery in Great Britain and Ireland National Database. Heart 2012; 98: 1568–1572.
46. Hickey GL, Grant SW, Cosgriff R, et al. Clinical registries: governance, management, analysis and applications. Eur J Cardiothorac Surg 2013; 44: 605–614.
47. van Buuren S and Groothuis-Oudshoorn K. mice: Multivariate imputation by chained equations in R. J Stat Softw 2011; 45: 1–67.
48. Little RJA and Rubin DB. Statistical analysis with missing data. New York: Wiley, 2002.
49. Justice AC. Assessing the generalizability of prognostic information. Ann Intern Med 1999; 130: 515–524.
50. R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, 2014, http://www.R-project.org/
51. Robin X, Turck N, Hainard A, et al. pROC: an open-source package for R and Sþ to analyze and compare ROC curves. BMC Bioinform 2011; 12: 77.
52. McCormick TH, Raftery AE and Madigan D. DMA: dynamic model averaging, http://www.cran.r-project.org/package=dma
53. Austin PC and Steyerberg EW. Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models. Stat Methods Med Res 2017; 26: 796–808.
54. Jin R, Furnary AP, Fine SC, et al. Using Society of Thoracic Surgeons risk models for risk-adjusting cardiac surgery results. Ann Thorac Surg 2010; 89: 677–682.