Calibration of risk prediction models: Impact on decision-analytic performance

Medical Decision Making

Volumn 35, Issue 2, 2015, Pages 162-169

  Van Calster, Ben ^a   Vickers, Andrew J ^b  

^a UNIVERSITY OF LEUVEN   (Belgium)

^b MEMORIAL SLOAN KETTERING CANCER CENTER   (United States)

Author keywords

calibration; decision curve analysis; decision analytic measures; discrimination; net benefit; prediction models; risk scores

Indexed keywords

COMPUTER SIMULATION; DECISION SUPPORT SYSTEM; HUMAN; MALE; NORWAY; PREDICTIVE VALUE; PROCEDURES; RECEIVER OPERATING CHARACTERISTIC; RISK ASSESSMENT; STATISTICAL BIAS; STATISTICAL MODEL; TESTICULAR NEOPLASMS; THEORETICAL MODEL;

BIAS (EPIDEMIOLOGY); COMPUTER SIMULATION; DECISION SUPPORT TECHNIQUES; HUMANS; LINEAR MODELS; MALE; MODELS, THEORETICAL; NORWAY; PREDICTIVE VALUE OF TESTS; RISK ASSESSMENT; ROC CURVE; TESTICULAR NEOPLASMS;

EID: 84921383600     PISSN: 0272989X     EISSN: 1552681X     Source Type: Journal    
DOI: 10.1177/0272989X14547233     Document Type: Article

References (27)

1. Collins GS, Moons KGM. Comparing risk prediction models. BMJ. 2012 ; 344: e3186
2. Tzoulaki I, Liberopoulos G, Ioannidis JPA. Assessment of claims of improved prediction beyond the Framingham risk score. JAMA. 2009 ; 302: 2345-52
3. Bouwmeester W, Zuithoff NPA, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PloS Med. 2012 ; 9: e1001221
4. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006 ; 26: 565-74
5. Greenland S. The need for reorientation toward cost-effective prediction: comments on "Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.". Stat Med. 2008 ; 27: 199-206
6. Baker SG, Cook NR, Vickers A, Kramer BS. Using relative utility curves to evaluate risk prediction. J R Stat Soc A. 2009 ; 172: 729-48
7. Vickers AJ, Cronin AM. Everything you always wanted to know about evaluating prediction models (but were too afraid to ask). Semin Oncol. 2010 ; 76: 1298-301
8. Pencina MJ, D'Agostino RB, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011 ; 30: 11-21
9. Van Calster B, Vickers AJ, Pencina MJ, Baker SG, Timmerman D, Steyerberg EW. Evaluation of markers and risk prediction models: overview of relationships between NRI and decision-analytic measures. Med Decis Making. 2013 ; 33: 490-501
10. Pauker SG, Kassirer JP. Therapeutic decision making: a cost-benefit analysis. N Engl J Med. 1975 ; 293: 229-34
11. Mallett S, Halligan S, Thompson M, Collins GS, Altman DG. Interpreting diagnostic accuracy studies for patient care. BMJ. 2012 ; 345: e3999
12. Moons KGM, de Groot JAH, Linnet K, Reitsma JB, Bossuyt PMM. Quantifying the added value of a diagnostic test or marker. Clin Chem. 2012 ; 58: 1408-17
13. Localio AR, Goodman S. Beyond the usual prediction accuracy metrics: reporting results for clinical decision making. Ann Intern Med. 2012 ; 157: 294-5
14. Steyerberg EW New York: Springer ; 2009:
15. PepeMFangJFengZGerdsTHildenJ. The net reclassification index (NRI): a misleading measure of prediction improvement with miscalibrated or overfit models. UW Biostatistics Working Paper Series, Working Paper 392. Available from: URL: http://biostats.bepress.com/uwbiostat/paper392
16. Baker SG, Van Calster B, Steyerberg EW. Evaluating a new marker for risk prediction using the test tradeoff: an update. Int J Biostat. 2012 ; 8:
17. Hlatky MA, Greenland P, Arnett DK, et al. Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation. 2009 ; 119: 2408-16
18. Cox DR. Two further applications of a model for binary regression. Biometrika. 1958 ; 45: 562-5
19. Vergouwe Y, Moons KGM, Steyerberg EW. External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol. 2010 ; 172: 971-80
20. Eckermann S, Briggs A, Willan AR. Health technology assessment in the cost-disutility plane. Med Decis Making. 2008 ; 28: 172-81
21. Vergouwe Y, Steyerberg EW, Foster RS, et al. Predicting retroperitoneal histology in postchemotherapy testicular germ cell cancer: a model update and multicentre validation with more than 1000 patients. Eur Urol. 2007 ; 51: 424-32
22. Collins GS, Altman DG. Predicting the 10 year risk of cardiovascular disease in the United Kingdom: independent and external validation of an updated version of QRISK2. BMJ. 2012 ; 344: e4181
23. Nam RK, Kattan MW, Chin JL, et al. Prospective multi-institutional study evaluating the performance of prostate cancer risk calculators. J Clin Oncol. 2011 ; 29: 2959-64
24. Steyerberg EW, Pencina MJ, Lingsma HF, Kattan MW, Vickers AJ, Van Calster B. Assessing the incremental value of diagnostic and prognostic markers: a review and illustration. Eur J Clin Invest. 2012 ; 42: 216-28
25. Van Calster B, Steyerberg EW, D'Agostino RB, Pencina MJ. Sensitivity and specificity can change in opposite directions when new predictive markers are added to risk models. Med Decis Making. 2014 ; 34: 513-22
26. Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med. 2014 ; 33: 517-35
27. Vach W. Calibration of clinical prediction rules does not just assess bias. J Clin Epidemiol. 2013 ; 66: 1296-301