Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures?

Statistical Methods in Medical Research

Volumn 27, Issue 11, 2018, Pages 3505-3522

  Snell, Kym I E ^a   Ensor, Joie ^a   Debray, Thomas P A ^b   Moons, Karel G M ^b   Riley, Richard D ^a  

^a KEELE UNIVERSITY   (United Kingdom)

^b UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

Author keywords

between study distribution; C statistic; calibration; discrimination; heterogeneity; meta analysis; performance statistics; simulation; Validation

Indexed keywords

ARTICLE; CALIBRATION; GENERAL PRACTICE; HUMAN; META ANALYSIS; NORMAL DISTRIBUTION; PREDICTION; QRISK SCORE; SIMULATION; STATISTICS; VALIDATION PROCESS; ALGORITHM; FORECASTING; MEDICAL RESEARCH; STATISTICAL MODEL; TREATMENT OUTCOME; VALIDATION STUDY;

ALGORITHMS; BIOMEDICAL RESEARCH; CALIBRATION; FORECASTING; MODELS, STATISTICAL; TREATMENT OUTCOME; VALIDATION STUDIES AS TOPIC;

EID: 85043710768     PISSN: 09622802     EISSN: 14770334     Source Type: Journal    
DOI: 10.1177/0962280217705678     Document Type: Article

References (43)

1. Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating, New York, NY: Springer, 2010
2. Hendriksen JM, Geersing GJ, Moons KG, Diagnostic and prognostic prediction models. J Thromb Haemost 2013; 11: 129–141
3. Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996; 15: 361–387
4. Van Calster B, Nieboer D, Vergouwe Y, A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol 2016; 74: 167–176
5. Siontis GC, Tzoulaki I, Castaldi PJ, External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 2015; 68: 25–34
6. Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med 1999; 130: 515–524
7. Bleeker SE, Moll HA, Steyerberg EW, External validation is necessary in prediction research: a clinical example. J Clin Epidemiol 2003; 56: 826–832
8. Altman DG, Vergouwe Y, Royston P, Prognosis and prognostic research: validating a prognostic model. Br Med J 2009; 338: b605–b605
9. Moons KG, Kengne AP, Grobbee DE, Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012; 98: 691–698
10. Debray TP, Vergouwe Y, Koffijberg H, A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015; 68: 279–289
11. Debray TP, Riley RD, Rovers MM, Individual participant data (IPD) meta-analyses of diagnostic and prognostic modeling studies: guidance on their use. PLoS Med 2015; 12: e1001886–e1001886
12. Riley RD, Ensor J, Snell KI, External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ 2016; 353: i3140–i3140
13. Steyerberg EW, Harrell FE, Jr. Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol 2016; 69: 245–247
14. van Klaveren D, Steyerberg EW, Perel P, Assessing discriminative ability of risk models in clustered data. BMC Med Res Methodol 2014; 14
15. Pennells L, Kaptoge S, White IR, Assessing risk prediction models using individual participant data from multiple studies. Am J Epidemiol 2014; 179: 621–632
16. Austin PC, van Klaveren D, Vergouwe Y, Geographic and temporal validity of prediction models: different approaches were useful to examine model performance. J Clin Epidemiol 2016; 79: 76–85
17. Debray TP, Damen JA, Snell KI, A guide to systematic review and meta-analysis of prediction model performance. BMJ 2017; 356: i6460–i6460
18. van Klaveren D, Gonen M, Steyerberg EW, A new concordance measure for risk prediction models in external validation settings. Stat Med 2016; 35: 4136–4152
19. Snell KI, Hua H, Debray TP, Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model. J Clin Epidemiol 2016; 69: 40–50
20. Vergouwe Y, Moons KGM, Steyerberg EW. External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010; 172: 971–980
21. Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. Br Med J 2011; 342: 964–967
22. Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. J Royal Stat Soc Ser A-Stat Soc 2009; 172: 137–159
23. Baker R, Jackson D. A new approach to outliers in meta-analysis. Health Care Manag Sci 2008; 11: 121–131
24. Lee KJ, Thompson SG. Flexible parametric models for random-effects distributions. Stat Med 2008; 27: 418–434
25. Hippisley-Cox J, Coupland C, Vinogradova Y, Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2. BMJ 2008; 336: 1475–1482
26. Harrell FE. Regression modeling strategies with applications to linear models, logistic regression, and survival analysis, New York, NY: Springer, 2001
27. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143: 29–36
28. Steyerberg EW, Vickers AJ, Cook NR, Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21: 128–138
29. Steyerberg EW, Borsboom GJJM, van Houwelingen HC, Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med 2004; 23: 2567–2586
30. Hartung J, Knapp G. On tests of the overall treatment effect in meta-analysis with normally distributed responses. Stat Med 2001; 20: 1771–1782
31. Sidik K, Jonkman JN. A simple confidence interval for meta-analysis. Stat Med 2002; 21: 3153–3159
32. IntHout J, Ioannidis JP, Borm GF. The Hartung–Knapp–Sidik–Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian–Laird method. BMC Med Res Methodol 2014; 14: 25–25
33. Partlett C, Riley RD. Random effects meta-analysis: Coverage performance of 95% confidence and prediction intervals following REML estimation. Stat Med 2017; 36: 301–317
34. Qin G, Hotilovac L. Comparison of non-parametric confidence intervals for the area under the ROC curve of a continuous-scale diagnostic test. Stat Meth Med Res 2008; 17: 207–221
35. Trikalinos TA, Trow P, Schmid CH. Simulation-based comparison of methods for meta-analysis of proportions and rates, Rockville, MD: Agency for Healthcare Research and Quality, 2013. Available at: www.effectivehealthcare.ahrq.gov/reports/final.cfm
36. Oudega R, Moons KG, Hoes AW. Ruling out deep venous thrombosis in primary care. A simple diagnostic algorithm including D-dimer testing. Thromb Haemost 2005; 94: 200–205
37. Meads C, Ahmed I, Riley RD. A systematic review of breast cancer incidence risk prediction models with meta-analysis of their performance. Breast Cancer Res Treat 2012; 132: 365–377
38. Terasawa T, Nishida H, Kato K, Prediction of gastric cancer development by serum pepsinogen test and Helicobacter pylori seropositivity in Eastern Asians: a systematic review and meta-analysis. PLoS One 2014; 9: e109783–e109783
39. Tangri N, Grams ME, Levey AS, Multinational assessment of accuracy of equations for predicting risk of kidney failure: a meta-analysis. JAMA 2016; 315: 164–174
40. Austin PC, Steyerberg EW. Interpreting the concordance statistic of a logistic regression model: relation to the variance and odds ratio of a continuous explanatory variable. BMC Med Res Methodol 2012; 12: 82–82
41. Reitsma JB, Glas AS, Rutjes AWS, Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol 2005; 58: 982–990
42. Chu H, Cole SR. Bivariate meta-analysis of sensitivity and specificity with sparse data: a generalized linear mixed model approach. J Clin Epidemiol 2006; 59: 1331–1332; author reply 2–3
43. Royston P, Sauerbrei W. A new measure of prognostic separation in survival data. Stat Med 2004; 23: 723–748