A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes

Statistical Methods in Medical Research

Volumn 28, Issue 9, 2019, Pages 2768-2786

  Debray, Thomas PA ^a   Damen, Johanna AAG ^a   Riley, Richard D ^b   Snell, Kym ^b   Reitsma, Johannes B ^a   Hooft, Lotty ^a   Collins, Gary S ^c   Moons, Karel GM ^a  

^a UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

^b KEELE UNIVERSITY   (United Kingdom)

^c UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

aggregate data; calibration; discrimination; evidence synthesis; Meta analysis; prediction; prognosis; systematic review; validation

Indexed keywords

ARTICLE; CALIBRATION; EUROSCORE; FRAMINGHAM RISK SCORE; HUMAN; META ANALYSIS; PREDICTION; PROGNOSIS; SYNTHESIS; SYSTEMATIC REVIEW; VALIDATION PROCESS; BAYES THEOREM; META ANALYSIS (TOPIC); METHODOLOGY; PROCEDURES; RISK ASSESSMENT; STATISTICAL MODEL;

BAYES THEOREM; CALIBRATION; HUMANS; META-ANALYSIS AS TOPIC; MODELS, STATISTICAL; PROGNOSIS; RESEARCH DESIGN; RISK ASSESSMENT; SYSTEMATIC REVIEWS AS TOPIC;

EID: 85050571269     PISSN: 09622802     EISSN: 14770334     Source Type: Journal    
DOI: 10.1177/0962280218785504     Document Type: Article

References (68)

1. Moons KGM, Royston P, Vergouwe Y, Prognosis and prognostic research: what, why, and how? BMJ 2009; 338: b375–b375.
2. Steyerberg EW, Moons KGM, van der Windt DA, Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med 2013; 10: e1001381–e1001381.
3. Nashef SAM, Roques F, Sharples LD, EuroSCORE II. Eur J Cardiothorac Surg 2012; 41: 734–744; discussion 744–745.
4. Wilson PW, D’Agostino RB, Levy D, Prediction of coronary heart disease using risk factor categories. Circulation 1998; 97: 1837–1847.
5. Damen JAAG, Hooft L, Schuit E, Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ 2016; 353: i2416–i2416.
6. Moons KGM, Kengne AP, Grobbee DE, Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012; 98: 691–698.
7. Debray TP, Koffijberg H, Nieboer D, Meta-analysis and aggregation of multiple published prediction models. Stat Med 2014; 33: 2341–2362.
8. Steyerberg EW, Borsboom GJJM, van Houwelingen HC, Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med 2004; 23: 2567–2586.
9. Debray TPA, Vergouwe Y, Koffijberg H, A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015; 68: 279–289.
10. Vergouwe Y, Moons KGM, Steyerberg EW. External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010; 172: 971–980.
11. Collins GS, de Groot JA, Dutton S, External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014; 14: 40–40.
12. Geersing GJ, Bouwmeester W, Zuithoff P, Search filters for finding prognostic and diagnostic prediction studies in Medline to enhance systematic reviews. PLoS ONE 2012; 7: e32844–e32844.
13. Moons KGM, de Groot JA, Bouwmeester W, Critical appraisal and data extraction for systematic reviews of clinical prediction modelling studies: the CHARMS checklist. PLoS Med 2014; 11: e1001744–e1001744.
14. Debray TPA, Damen JAAG, Snell KIE, A guide to systematic review and meta-analysis of prediction model performance. BMJ 2017; 356: i6460–i6460.
15. Debray T. metamisc: diagnostic and prognostic meta-analysis, https://CRAN.R-project.org/package=metamisc (accessed 1 January 2018).
16. Bouwmeester W, Zuithoff NPA, Mallett S, Reporting and methods in clinical prediction research: a systematic review. PLoS Med 2012; 9: 1–12.
17. Guida P, Mastro F, Scrascia G, Performance of the European System for Cardiac Operative Risk Evaluation II: a meta-analysis of 22 studies involving 145,592 cardiac surgery procedures. J Thorac Cardiovasc Surg 2014; 148: 3049–3057.e1.
18. Damen JAAG, Pajouheshnia R, Heus P, Performance of the Framingham risk models and pooled cohort equations: a systematic review and meta-analysis. PLOS ONE 2018. Under review (submitted).
19. Newson R. Parameters behind nonparametric statistics: Kendalls tau, Somers D and median differences. Stata J 2002; 2: 45–64.
20. Austin PC, Steyerberg EW. Interpreting the concordance statistic of a logistic regression model: relation to the variance and odds ratio of a continuous explanatory variable. BMC Med Res Methodol 2012; 12: 82–82.
21. Harrell FE, Califf RM, Pryor DB, Evaluating the yield of medical tests. JAMA 1982; 247: 2543–2546.
22. Lirette ST, Aban I. Quantifying predictive accuracy in survival models. J Nucl Cardiol 2015; 24: 1998–2003. DOI:10.1007/s12350-015-0296-z.
23. Austin PC, Pencinca MJ, Steyerberg EW. Predictive accuracy of novel risk factors and markers: a simulation study of the sensitivity of different performance measures for the Cox proportional hazards regression model. Stat Methods Med Res 2015; 26: 1053–1077. DOI:10.1177/ 0962280214567141.
24. Royston P, Sauerbrei W. A new measure of prognostic separation in survival data. Stat Med 2004; 23: 723–748.
25. Jinks RC, Royston P, Parmar MKB. Discrimination-based sample size calculations for multivariable prognostic models for time-to-event data. BMC Med Res Methodol 2015; 15: 82–82.
26. White IR, Rapsomaniki E. Emerging Risk Factors Collaboration. Covariate-adjusted measures of discrimination for survival data. Biom J 2015; 57: 592–613.
27. Kamarudin AN, Cox T, Kolamunnage-Dona R. Time-dependent ROC curve analysis in medical research: current methods and applications. BMC Med Res Methodol 2017; 17: 53–53.
28. Blanche P, Dartigues JF, Jacqmin-Gadda H. Review and comparison of ROC curve estimators for a time-dependent outcome with marker-dependent censoring. Biom J 2013; 55: 687–704.
29. Newcombe RG. Confidence intervals for an effect size measure based on the Mann-Whitney statistic. Part 2: asymptotic methods and evaluation. Stat Med 2006; 25: 559–573.
30. Snell KI, Ensor J, Debray TP, Meta-analysis of prediction model performance across multiple studies: which scale helps ensure between-study normality for the C -statistic and calibration measures? Stat Meth Med Res 2017, pp. 096228021770567–096228021770567. DOI:10.1177/0962280217705678.
31. Oehlert GW. A note on the delta method. Am Stat 1992; 46: 27–27.
32. Altman DG, Bland JM. How to obtain the P value from a confidence interval. BMJ 2011; 343: d2304–d2304.
33. Howell NJ, Head SJ, Freemantle N, The new EuroSCORE II does not improve prediction of mortality in high-risk patients undergoing cardiac surgery: a collaborative analysis of two European centres. Eur J Cardiothorac Surg 2013; 44: 1006–1011; discussion 1011.
34. Steyerberg EW, Vickers AJ, Cook NR, Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21: 128–138.
35. Cox DR. Two further applications of a model for binary regression. Biometrika 1958; 45: 562–565.
36. Dorfman R. A note on the method for finding variance formulae. Biometric Bull 1938; 1: 129–137.
37. Buitrago F, Calvo-Hueros JI, Can-Barroso L, Original and REGICOR Framingham functions in a nondiabetic population of a Spanish health care center: a validation study. Ann Fam Med 2011; 9: 431–438.
38. Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Software 2010; 36: 1–48. DOI:10.18637/jss.v036.i03.
39. Plummer M. JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In: Proceedings of the 3rd international workshop on distributed statistical computing, Vienna, Austria, March 2003.
40. Borenstein M, Hedges LV, Higgins JPT, A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Meth 2010; 1: 97–111.
41. Kundu S, Mazumdar M, Ferket B. Impact of correlation of predictors on discrimination of risk models in development and external populations. BMC Med Res Methodol 2017; 17: 63–63.
42. Nieboer D, van der Ploeg T, Steyerberg EW. Assessing discriminative performance at external validation of clinical prediction models. PLoS ONE 2016; 11: e0148820–e0148820.
43. Stijnen T, Hamza TH, Zdemir P. Random effects meta-analysis of event outcome in the framework of the generalized linear mixed model with applications in sparse data. Stat Med 2010; 29: 3046–3067.
44. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trial 1986; 7: 177–188.
45. Veroniki AA, Jackson D, Viechtbauer W, Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Meth 2016; 7: 55–79.
46. Partlett C, Riley RD. Random effects meta-analysis: coverage performance of 95% confidence and prediction intervals following REML estimation. Stat Med 2017; 36: 301–317.
47. Cornell JE, Mulrow CD, Localio R, Random-effects meta-analysis of inconsistent effects: a time for change. Ann Intern Med 2014; 160: 267–270.
48. Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ 2011; 342: d549–d549.
49. Greenland S. Bayesian perspectives for epidemiological research: I. Foundations and basic methods. Int J Epidemiol 2006; 35: 765–775.
50. Spiegelhalter DJ. Evidence synthesis. In: Senn S and Barnett V (eds.) Bayesian approaches to clinical trials and health-care evaluation. Chichester, UK: John Wiley & Sons, Ltd, 2004, pp.267–303.
51. Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc 2009; 172: 137–159.
52. Bodnar O, Link A, Arendack B, Bayesian estimation in random effects meta-analysis using a non-informative prior. Stat Med 2017; 36: 378–399.
53. Turner RM, Jackson D, Wei Y, Predictive distributions for between-study heterogeneity and simple methods for their application in Bayesian meta-analysis. Stat Med 2015; 34: 984–998.
54. Simpson D, Rue H, Riebler A, Penalising model component complexity: a principled, practical approach to constructing priors. Stat Sci 2017; 32: 1–28.
55. Greenland S. Bayesian perspectives for epidemiological research. II. Regression analysis. Int J Epidemiol 2007; 36: 195–202.
56. Burke DL, Bujkiewicz S, Riley RD. Bayesian bivariate meta-analysis of correlated effects: Impact of the prior distributions on the between-study correlation, borrowing of strength, and joint inferences. Stat Meth Med Res 2018; 27: 428–450. DOI: 10.1177/0962280216631361.
57. Lambert PC, Sutton AJ, Burton PR, How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS. Stat Med 2005; 24: 2401–2428.
58. Spiegelhalter D. Prior distributions. In: Senn S and Barnett V (eds.) Bayesian approaches to clinical trials and health-care evaluation. Chichester, UK: John Wiley & Sons, Ltd, 2004, pp.139–180.
59. Gelman A. Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper). Bayes Analys 2006; 1: 515–534.
60. Gelman A, Rubin DB. Inference from iterative simulation using multiple sequences. Stat Science 1992; 7: 457–472.
61. Pennells L, Kaptoge S, White IR, Assessing risk prediction models using individual participant data from multiple studies. Am J Epidemiol 2013; 179: 621–632.
62. Ban JW, Emparanza JI, Urreta I, Design characteristics influence performance of clinical prediction rules in validation: a meta-epidemiological study. PLoS ONE 2016; 11: e0145779–e0145779.
63. van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat Med 2002; 21: 589–624.
64. Austin PC. Estimating multilevel logistic regression models when the number of clusters is low: a comparison of different statistical software procedures. Int J Biostat 2010; 6: Article 16.
65. Sidik K, Jonkman JN. A comparison of heterogeneity variance estimators in combining results of studies. Stat Med 2007; 26: 1964–1981.
66. Langan D, Higgins JPT, Simmonds M. Comparative performance of heterogeneity variance estimators in meta-analysis: a review of simulation studies. Res Synth Meth 2017; 8: 181–198. DOI: 10.1002/jrsm.1198.
67. Debray TPA, Moons KGM, van Valkenhoef G, Get real in individual participant data (IPD) meta-analysis: a review of the methodology. Res Synth Meth 2015; 6: 239–309.
68. van Klaveren D, Gnen M, Steyerberg EW, A new concordance measure for risk prediction models in external validation settings. Stat Med 2016; 35: 4136–4152. DOI:10.1002/sim.6997.