A guide to systematic review and meta-analysis of prediction model performance

BMJ (Online)

Volumn 356, Issue , 2017, Pages

  Debray, Thomas P A ^a   Damen, Johanna A A G ^a   Snell, Kym I E ^b   Ensor, Joie ^b   Hooft, Lotty ^a   Reitsma, Johannes B ^a   Riley, Richard D ^b   Moons, Karel G M ^a  

^a UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

^b KEELE UNIVERSITY   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

CALIBRATION; CASE STUDY; CORONARY ARTERY BYPASS GRAFT; DATA EXTRACTION; EUROSCORE; EVIDENCE BASED MEDICINE; HIGH RISK PATIENT; MEDICAL RESEARCH; META ANALYSIS (TOPIC); PREDICTION; PRIORITY JOURNAL; QUANTITATIVE STUDY; REVIEW; RISK ASSESSMENT; SENSITIVITY ANALYSIS; SURGICAL MORTALITY; SURVIVAL RATE; SYSTEMATIC REVIEW (TOPIC); VALIDATION PROCESS; DECISION SUPPORT SYSTEM; HUMAN; LITERATURE; MORTALITY; PROCEDURES; STANDARDS; VALIDATION STUDY;

CORONARY ARTERY BYPASS; DECISION SUPPORT TECHNIQUES; HUMANS; META-ANALYSIS AS TOPIC; REVIEW LITERATURE AS TOPIC; RISK ASSESSMENT; VALIDATION STUDIES AS TOPIC;

EID: 85009129532     PISSN: 09598146     EISSN: 17561833     Source Type: Journal    
DOI: 10.1136/bmj.i6460     Document Type: Review

References (46)

1. Khan K, Kunz R, Kleijnen J, et al. Systematic reviews to support evidence-based medicine: how to review and apply findings of healthcare research. CRC Press, 2nd ed, 2011.
2. Steyerberg EW, Moons KGM, van der Windt DA, et al. PROGRESS Group. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med 2013;10:e1001381. doi:10.1371/journal. pmed.1001381.
3. Geersing GJ, Bouwmeester W, Zuithoff P, Spijker R, Leeflang M, Moons KG. Search filters for finding prognostic and diagnostic prediction studies in Medline to enhance systematic reviews [correction in: PLoS One 2012;7(7): doi/10.1371/annotation/96bdb520-d704-45f0-a143-43a48552952e]. PLoS One 2012;7:e32844. doi:10.1371/journal. pone.0032844.
4. Wong SS, Wilczynski NL, Haynes RB, Ramkissoonsingh R. Hedges Team. Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE. AMIA Annu Symp Proc 2003:728-32.
5. Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc 2001;8:391-7. doi:10.1136/ jamia.2001.0080391.
6. Moons KGM, de Groot JAH, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med 2014;11:e1001744. doi:10.1371/journal.pmed.1001744.
7. Wolff R, Whiting P, Mallett S, et al. PROBAST: a risk of bias tool for prediction modelling studies. Cochrane Colloquium Vienna. 2015.
8. Debray TPA, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg EW, Moons KG. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015;68:279-89. doi:10.1016/j.jclinepi.2014.06.018.
9. Snell KI, Hua H, Debray TP, et al. Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model. J Clin Epidemiol 2016;69:40-50. doi:10.1016/j.jclinepi.2015.05.009.
10. Altman DG, Royston P. What do we mean by validating a prognostic model?Stat Med 2000;19:453-73. doi:10.1002/ (SICI)1097-0258(20000229)19:4<453::AID-SIM350>3.0.CO;2-5.
11. Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med 1999;130:515-24. doi:10.7326/0003-4819-130-6-199903160-00016.
12. Collins GS, Omar O, Shanyinde M, Yu LM. A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods. J Clin Epidemiol 2013;66:268-77. doi:10.1016/j.jclinepi.2012.06.020.
13. Siregar S, Groenwold RHH, de Heer F, Bots ML, van der Graaf Y, van Herwerden LA. Performance of the original EuroSCORE. Eur J Cardiothorac Surg 2012;41:746-54. doi:10.1093/ejcts/ezr285.
14. Echouffo-Tcheugui JB, Batty GD, Kivimäki M, Kengne AP. Risk models to predict hypertension: a systematic review. PLoS One 2013;8:e67370. doi:10.1371/journal.pone.0067370.
15. Tzoulaki I, Liberopoulos G, Ioannidis JPA. Assessment of claims of improved prediction beyond the Framingham risk score. JAMA 2009;302:2345-52. doi:10.1001/jama.2009.1757.
16. Eichler K, Puhan MA, Steurer J, Bachmann LM. Prediction of first coronary events with the Framingham score: a systematic review. Am Heart J 2007;153:722-31, 731.e1-8. doi:10.1016/j.ahj.2007.02.027.
17. Perel P, Edwards P, Wentz R, Roberts I. Systematic review of prognostic models in traumatic brain injury. BMC Med Inform Decis Mak 2006;6:38. doi:10.1186/1472-6947-6-38.
18. Collins GS, de Groot JA, Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014;14:40. doi:10.1186/1471-2288-14-40.
19. Riley RD, Ensor J, Snell KI, et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ 2016;353:i3140. doi:10.1136/bmj.i3140.
20. Peat G, Riley RD, Croft P, et al. PROGRESS Group. Improving the transparency of prognosis research: the role of reporting, data sharing, registration, and protocols. PLoS Med 2014;11:e1001671. doi:10.1371/journal.pmed.1001671.
21. Iorio A, Spencer FA, Falavigna M, et al. Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. BMJ 2015;350:h870. doi:10.1136/bmj.h870.
22. Vergouwe Y, Moons KGM, Steyerberg EW. External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010;172:971-80. doi:10.1093/aje/kwq223.
23. Moons KGM, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015;162:W1-73. doi:10.7326/M14-0698.
24. van Klaveren D, Gönen M, Steyerberg EW, Vergouwe Y. A new concordance measure for risk prediction models in external validation settings. Stat Med 2016;35:4136-52. doi:10.1002/sim.6997.
25. Higgins JPT, Green S. Combining Groups. http://handbook.cochrane. org/chapter-7/7-7-3-8-combining-groups.htm, 2011.
26. Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol 2005;5:13. doi:10.1186/1471-2288-5-13.
27. Bouwmeester W, Zuithoff NPA, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med 2012;9:1-12. doi:10.1371/journal.pmed.1001221.
28. Austin PC, Pencinca MJ, Steyerberg EW. Predictive accuracy of novel risk factors and markers: A simulation study of the sensitivity of different performance measures for the Cox proportional hazards regression model. Stat Methods Med Res 2015;0962280214567141.
29. Blanche P, Dartigues JF, Jacqmin-Gadda H. Review and comparison of ROC curve estimators for a time-dependent outcome with marker-dependent censoring. Biom J 2013;55:687-704. doi:10.1002/bimj.201200045.
30. Jinks RC, Royston P, Parmar MKB. Discrimination-based sample size calculations for multivariable prognostic models for time-to-event data. BMC Med Res Methodol 2015;15:82. doi:10.1186/s12874-015-0078-y.
31. Pennells L, Kaptoge S, White IR, Thompson SG, Wood AM. Emerging Risk Factors Collaboration. Assessing risk prediction models using individual participant data from multiple studies. Am J Epidemiol 2014;179:621-32. doi:10.1093/aje/kwt298.
32. Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ 2011;342:d549. doi:10.1136/bmj.d549.
33. Snell KIE. Development and application of statistical methods for prognosis research. PhD thesis, School of Health and Population Sciences, Birmingham, UK, 2015.
34. van Klaveren D, Steyerberg EW, Perel P, Vergouwe Y. Assessing discriminative ability of risk models in clustered data. BMC Med Res Methodol 2014;14:5. doi:10.1186/1471-2288-14-5.
35. Gengsheng Qin, Hotilovac L. Comparison of non-parametric confidence intervals for the area under the ROC curve of a continuous-scale diagnostic test. Stat Methods Med Res 2008;17:207-21. doi:10.1177/0962280207087173.
36. IntHout J, Ioannidis JP, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Med Res Methodol 2014;14:25. doi:10.1186/1471-2288-14-25.
37. Cornell JE, Mulrow CD, Localio R, et al. Random-effects meta-analysis of inconsistent effects: a time for change. Ann Intern Med 2014;160:267-70. doi:10.7326/M13-2886.
38. Ban JW, Emparanza JI, Urreta I, Burls A. Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study. PLoS One 2016;11:e0145779. doi:10.1371/journal.pone.0145779.
39. Deeks JJ, Higgins JPT, Altman DG. Chapter 9. Analysing data and undertaking meta-analyses.Cochrane Collaboration, 2011.
40. Berlin JA, Santanna J, Schmid CH, Szczech LA, Feldman HI. Anti-Lymphocyte Antibody Induction Therapy Study Group. Individual patient-versus group-level data meta-regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head. Stat Med 2002;21:371-87. doi:10.1002/sim.1023.
41. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009;339:b2700. doi:10.1136/bmj.b2700.
42. Stewart LA, Clarke M, Rovers M, et al. PRISMA-IPD Development Group. Preferred reporting items for a systematic review and meta-analysis of individual participant data: The PRISMA-IPD Statement. JAMA 2015;313:1657-65. doi:10.1001/jama.2015.3656.
43. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med 2015;162:55-63. doi:10.7326/M14-0697.
44. Debray TPA, Riley RD, Rovers MM, Reitsma JB, Moons KG. Cochrane IPD Meta-analysis Methods group. Individual participant data (IPD) meta-analyses of diagnostic and prognostic modeling studies: guidance on their use. PLoS Med 2015;12:e1001886. doi:10.1371/ journal.pmed.1001886.
45. Sutton AJ, Abrams KR. Bayesian methods in meta-analysis and evidence synthesis. Stat Methods Med Res 2001;10:277-303. doi:10.1191/096228001678227794.
46. Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ 2016;352:i6. doi:10.1136/bmj.i6