Correction: External validation of clinical prediction models using big datasets from e-health records or IPD meta-Analysis: Opportunities and challenges (BMJ (Online) (2016) 353 DOI: 10.1136/bmj.i3140);External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: Opportunities and challenges

The BMJ

Volumn 353, Issue , 2016, Pages

  Riley, Richard D ^a   Ensor, Joie ^a   Snell, Kym I E ^b   Debray, Thomas P A ^c   Altman, Doug G ^d   Moons, Karel G M ^c   Collins, Gary S ^d  

^a KEELE UNIVERSITY   (United Kingdom)

^b UNIVERSITY OF BIRMINGHAM   (United Kingdom)

^c UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

^d UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ACCESS TO INFORMATION; ARTICLE; CALIBRATION; CLINICAL RESEARCH; DATA BASE; ELECTRONIC MEDICAL RECORD; EXTERNAL VALIDATION; IMPACT EVALUATION; INDIVIDUAL PARTICIPANT DATA; METHODOLOGY; MODEL DEVELOPMENT; PRACTICE GUIDELINE; PREDICTION; PRIORITY JOURNAL; PROGNOSIS; REGISTER; STATISTICAL MODEL; VALIDATION STUDY; DECISION SUPPORT SYSTEM; ELECTRONIC HEALTH RECORD; HUMAN; INFORMATION PROCESSING; META ANALYSIS (TOPIC); PROCEDURES; RISK ASSESSMENT;

DATASETS AS TOPIC; DECISION SUPPORT TECHNIQUES; ELECTRONIC HEALTH RECORDS; HUMANS; META-ANALYSIS AS TOPIC; MODELS, STATISTICAL; REGISTRIES; RISK ASSESSMENT; VALIDATION STUDIES AS TOPIC;

EID: 84976645528     PISSN: 09598146     EISSN: 17561833     Source Type: Journal    
DOI: 10.1136/bmj.l4379     Document Type: Erratum

References (82)

1. Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating.Springer, 2009doi: 10.1007/978-0-387-77244-8
2. Royston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model. BMJ 2009; 338: b604. doi: 10.1136/bmj.b604
3. Steyerberg EW, Moons KG, van der Windt DA, et al. PROGRESS Group. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med 2013; 10: e1001381. doi: 10.1371/journal.pmed.1001381
4. Anderson KM, Odell PM, Wilson PW, Kannel WB. Cardiovascular disease risk profiles. Am Heart J 1991; 121: 293-8. doi: 10.1016/0002-8703(91)90861-B.
5. Hippisley-Cox J, Coupland C, Vinogradova Y, et al. Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2. BMJ 2008; 336: 1475-82. doi: 10.1136/bmj.39609.449676.25
6. Haybittle JL, Blamey RW, Elston CW, et al. A prognostic index in primary breast cancer. Br J Cancer 1982; 45: 361-6. doi: 10.1038/bjc.1982.62
7. Galea MH, Blamey RW, Elston CE, Ellis IO. The Nottingham Prognostic Index in primary breast cancer. Breast Cancer Res Treat 1992; 22: 207-19. doi: 10.1007/BF01840834
8. Wells PS, Anderson DR, Rodger M, et al. Derivation of a simple clinical model to categorize patients probability of pulmonary embolism: increasing the models utility with the SimpliRED D-dimer. Thromb Haemost 2000; 83: 416-20
9. Wells PS, Anderson DR, Bormanis J, et al. Value of assessment of pretest probability of deep-vein thrombosis in clinical management. Lancet 1997; 350: 1795-8. doi: 10.1016/S0140-6736(97)08140-3
10. Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. BMJ 2009; 338: b605. doi: 10.1136/bmj.b605
11. Moons KG, Altman DG, Vergouwe Y, Royston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 2009; 338: b606. doi: 10.1136/bmj.b606
12. Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how?BMJ 2009; 338: b375. doi: 10.1136/bmj.b375
13. Hemingway H, Croft P, Perel P, et al. PROGRESS Group. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ 2013; 346: e5595. doi: 10.1136/bmj.e5595
14. Riley RD, Hayden JA, Steyerberg EW, et al. PROGRESS Group. Prognosis Research Strategy (PROGRESS) 2: prognostic factor research. PLoS Med 2013; 10: e1001380. doi: 10.1371/journal. pmed.1001380
15. Hingorani AD, Windt DA, Riley RD, et al. PROGRESS Group. Prognosis research strategy (PROGRESS) 4: stratified medicine research. BMJ 2013; 346: e5793. doi: 10.1136/bmj.e5793
16. Pavlou M, Ambler G, Seaman SR, et al. How to develop a more accurate risk prediction model when there are few events [correction in BMJ 2016; 353: i3235]. BMJ 2015; 351: h3868. doi: 10.1136/bmj.h3868
17. Moons KG, Kengne AP, Woodward M, et al. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart 2012; 98: 683-90. doi: 10.1136/heartjnl-2011-301246
18. Harrell FE. Regression modeling strategies, with applications to linear models, logistic regression, and survival analysis.Springer, 2001
19. Bleeker SE, Moll HA, Steyerberg EW, et al. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol 2003; 56: 826-32. doi: 10.1016/S0895-4356(03)00207-5
20. Debray TP, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg EW, Moons KG. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 2015; 68: 279-89. doi: 10.1016/j.jclinepi.2014.06.018
21. Mallett S, Royston P, Waters R, Dutton S, Altman DG. Reporting performance of prognostic models in cancer: a review. BMC Med 2010; 8: 21. doi: 10.1186/1741-7015-8-21
22. Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 2006; 144: 201-9. doi: 10.7326/0003-4819-144-3-200602070-00009
23. Bouwmeester W, Zuithoff NP, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med 2012; 9: e1001221. doi: 10.1371/journal.pmed.1001221
24. Wyatt J, Altman DG. Commentary: Prognostic models: clinically useful or quickly forgotten?BMJ 1995; 311: 1539-41doi: 10.1136/bmj.311.7019.1539
25. Collins GS, Michaëlsson K. Fracture risk assessment: state of the art, methodologically unsound, or poorly reported?Curr Osteoporos Rep 2012; 10: 199-207. doi: 10.1007/s11914-012-0108-1
26. Steyerberg EW, Harrell FEJ Jr, , Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 2001; 54: 774-81. doi: 10.1016/S0895-4356(01)00341-9
27. Riley RD, Lambert PC, Abo-Zaid G. Meta-Analysis of individual participant data: rationale, conduct, and reporting. BMJ 2010; 340: c221. doi: 10.1136/bmj.c221
28. Ahmed I, Debray TP, Moons KG, Riley RD. Developing and validating risk prediction models in an individual participant data meta-Analysis. BMC Med Res Methodol 2014; 14: 3. doi: 10.1186/1471-2288-14-3
29. Debray TPA, Riley RD, Rovers MM, Reitsma JB, Moons KG. Cochrane IPD Meta-Analysis Methods group. Individual participant data (IPD) meta-Analyses of diagnostic and prognostic modeling studies: guidance on their use. PLoS Med 2015; 12: e1001886. doi: 10.1371/journal.pmed.1001886
30. Pennells L, Kaptoge S, White IR, Thompson SG, Wood AM. Emerging Risk Factors Collaboration. Assessing risk prediction models using individual participant data from multiple studies. Am J Epidemiol 2014; 179: 621-32. doi: 10.1093/aje/kwt298
31. Cook JA, Collins GS. The rise of big clinical databases. Br J Surg 2015; 102: e93-101. doi: 10.1002/bjs.9723
32. Collins GS, Altman DG. An independent and external validation of QRISK2 cardiovascular disease risk score: a prospective open cohort study. BMJ 2010; 340: c2442. doi: 10.1136/bmj.c2442
33. Steyerberg EW, Mushkudiani N, Perel P, et al. Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med 2008; 5: e165. doi: 10.1371/journal.pmed.0050165
34. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med 2015; 162: 55-63. doi: 10.7326/M14-0697
35. Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015; 162: W1-73. doi: 10.7326/M14-0698
36. Royston P, Sauerbrei W. A new measure of prognostic separation in survival data. Stat Med 2004; 23: 723-48. doi: 10.1002/sim.1621
37. Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol 2016; S0895-4356(15)00581-8. doi: 10.1016/j.jclinepi.2015.12.005 PubMed
38. Royston P, Altman DG. External validation of a Cox prognostic model: principles and methods. BMC Med Res Methodol 2013; 13: 33. doi: 10.1186/1471-2288-13-33
39. Collins GS, de Groot JA, Dutton S, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 2014; 14: 40. doi: 10.1186/1471-2288-14-40
40. Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 2005; 58: 475-83. doi: 10.1016/j.jclinepi.2004.06.017
41. Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med 2016; 35: 214-26. doi: 10.1002/sim.6787
42. Royston P, Parmar MKB, Sylvester R. Construction and validation of a prognostic model across several studies, with an application in superficial bladder cancer. Stat Med 2004; 23: 907-26. doi: 10.1002/sim.1691
43. Vergouwe Y, Moons KG, Steyerberg EW. External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010; 172: 971-80. doi: 10.1093/aje/kwq223
44. Collins GS, Altman DG. Predicting the 10 year risk of cardiovascular disease in the United Kingdom: independent and external validation of an updated version of QRISK2. BMJ 2012; 344: e4181. doi: 10.1136/bmj.e4181
45. Debray TP, Moons KG, Ahmed I, Koffijberg H, Riley RD. A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-Analysis. Stat Med 2013; 32: 3158-80. doi: 10.1002/sim.5732
46. Mulherin SA, Miller WC. Spectrum bias or spectrum effect?. Subgroup variation in diagnostic test evaluation. Ann Intern Med 2002; 137: 598-602. doi: 10.7326/0003-4819-137-7-200210010-00011
47. Ransohoff DF, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med 1978; 299: 926-30. doi: 10.1056/NEJM197810262991705
48. Knottnerus JA. Between iatrotropic stimulus and interiatric referral: the domain of primary care research. J Clin Epidemiol 2002; 55: 1201-6. doi: 10.1016/S0895-4356(02)00528-0
49. Oudega R, Hoes AW, Moons KG. The Wells rule does not adequately rule out deep venous thrombosis in primary care patients. Ann Intern Med 2005; 143: 100-7. doi: 10.7326/0003-4819-143-2-200507190-00008
50. Sauerbrei W. Prognostic factors-confusion caused by bad quality of design, analysis and reporting of many studies. In: Bier H, ed. Current research in head and neck cancer advances in oto-rhinolaryngology. Karger, 2005: 184-200
51. Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y. Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 2008; 61: 76-86. doi: 10.1016/j. jclinepi.2007.04.018
52. Riley RD, Higgins JP, Deeks JJ. Interpretation of random effects meta-Analyses. BMJ 2011; 342: d549. doi: 10.1136/bmj.d549
53. Snell KI, Hua H, Debray TP, et al. Multivariate meta-Analysis of individual participant data helped externally validate the performance and implementation of a prediction model. J Clin Epidemiol 2016; 69: 40-50. doi: 10.1016/j.jclinepi.2015.05.009
54. van Klaveren D, Steyerberg EW, Perel P, Vergouwe Y. Assessing discriminative ability of risk models in clustered data. BMC Med Res Methodol 2014; 14: 5. doi: 10.1186/1471-2288-14-5
55. Rücker G, Schwarzer G, Carpenter JR, Schumacher M. Undue reliance on I(2) in assessing heterogeneity may mislead. BMC Med Res Methodol 2008; 8: 79. doi: 10.1186/1471-2288-8-79
56. Geersing GJ, Zuithoff NP, Kearon C, et al. Exclusion of deep vein thrombosis using the Wells rule in clinically important subgroups: individual patient data meta-Analysis. BMJ 2014; 348: g1340. doi: 10.1136/bmj.g1340
57. Gengsheng Qin , Hotilovac L. Comparison of non-parametric confidence intervals for the area under the ROC curve of a continuous-scale diagnostic test. Stat Methods Med Res 2008; 17: 207-21. doi: 10.1177/0962280207087173
58. Tillin T, Hughes AD, Whincup P, et al. QRISK2 validation by ethnic group. Heart 2014; 100: 437. doi: 10.1136/heartjnl-2013-305333
59. Dalton AR, Bottle A, Soljak M, Majeed A, Millett C. Ethnic group differences in cardiovascular risk assessment scores: national cross-sectional study. Ethn Health 2014; 19: 367-84. doi: 10.1080/13557858.2013.797568
60. Hippisley-Cox J, Coupland C, Brindle P. Validation of QRISK2 (2014) in patients with diabetes. Online report http://eprints.nottingham. ac.uk/3602/2014
61. Riley RD, Ahmed I, Debray TP, et al. Summarising and validating test accuracy results across multiple studies for use in clinical practice. Stat Med 2015; 34: 2081-103. doi: 10.1002/sim.6471
62. Willis BH, Hyde CJ. Estimating a test's accuracy using tailored meta-Analysis-How setting-specific data may aid study selection. J Clin Epidemiol 2014; 67: 538-46. doi: 10.1016/j.jclinepi.2013.10.016
63. Leeflang MM, Rutjes AW, Reitsma JB, Hooft L, Bossuyt PM. Variation of a test's sensitivity and specificity with disease prevalence. CMAJ 2013; 185: E537-44. doi: 10.1503/cmaj.121286
64. Schuetz P, Koller M, Christ-Crain M, et al. Predicting mortality with pneumonia severity scores: importance of model recalibration to local settings. Epidemiol Infect 2008; 136: 1628-37. doi: 10.1017/S0950268808000435
65. Abo-Zaid G, Sauerbrei W, Riley RD. Individual participant data meta-Analysis of prognostic factor studies: state of the art?BMC Med Res Methodol 2012; 12: 56. doi: 10.1186/1471-2288-12-56
66. Jolani S, Debray TP, Koffijberg H, van Buuren S, Moons KG. Imputation of systematically missing predictors in an individual participant data meta-Analysis: a generalized approach using MICE. Stat Med 2015; 34: 1841-63. doi: 10.1002/sim.6451
67. Resche-Rigon M, White IR, Bartlett JW, Peters SA, Thompson SG. PROG-IMT Study Group. Multiple imputation for handling systematically missing confounders in meta-Analysis of individual participant data. Stat Med 2013; 32: 4890-905. doi: 10.1002/sim.5894
68. Herrett E, Gallagher AM, Bhaskaran K, et al. Data Resource Profile: Clinical Practice Research Datalink (CPRD). Int J Epidemiol 2015; 44: 827-36. doi: 10.1093/ije/dyv098
69. Tierney JF, Vale C, Riley R, et al. Individual Participant Data (IPD) Meta-Analyses of Randomised Controlled Trials: Guidance on Their Use. PLoS Med 2015; 12: e1001855. doi: 10.1371/journal. pmed.1001855
70. Altman DG, Trivella M, Pezzella F, et al. Systematic review of multiple studies of prognosis: the feasibility of obtaining individual patient data. In: Auget J-L, Balakrishnan N, Mesbah M, et al, eds. Advances in statistical methods for the health sciences. Birkhäuser, 2006: 3-18
71. Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta-Analyses using individual participant data: a database survey. BMJ 2012; 344: d7762. doi: 10.1136/bmj.d7762
72. Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996; 49: 1373-9. doi: 10.1016/S0895-4356(96)00236-3
73. Jinks RC, Royston P, Parmar MK. Discrimination-based sample size calculations for multivariable prognostic models for time-to-event data. BMC Med Res Methodol 2015; 15: 82. doi: 10.1186/s12874-015-0078-y.
74. Ogundimu EO, Altman DG, Collins GS. Adequate sample size for developing prediction models is not simply related to events per variable. J Clin Epidemiol 2016; S0895-4356(16)30011-7
75. Wynants L, Timmerman D, Bourne T, Van Huffel S, Van Calster B. Screening for data clustering in multicenter studies: the residual intraclass correlation. BMC Med Res Methodol 2013; 13: 128. doi: 10.1186/1471-2288-13-128
76. Stewart LA, Clarke M, Rovers M, et al. PRISMA-IPD Development Group. Preferred Reporting Items for Systematic Review and Meta-Analyses of individual participant data: the PRISMA-IPD Statement. JAMA 2015; 313: 1657-65. doi: 10.1001/jama.2015.3656
77. Hippisley-Cox J, Coupland C, Robson J, Sheikh A, Brindle P. Predicting risk of type 2 diabetes in England and Wales: prospective derivation and validation of QDScore. BMJ 2009; 338: b880. doi: 10.1136/bmj. b880
78. Debray TP, Koffijberg H, Vergouwe Y, Moons KG, Steyerberg EW. Aggregating published prediction models with individual participant data: a comparison of different approaches. Stat Med 2012; 31: 2697-712. doi: 10.1002/sim.5412
79. Collins GS, Moons KG. Comparing risk prediction models. BMJ 2012; 344: e3186. doi: 10.1136/bmj.e3186
80. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 2006; 26: 565-74. doi: 10.1177/0272989X06295361
81. Krumholz HM. Why data sharing should be the expected norm. BMJ 2015; 350: h599. doi: 10.1136/bmj.h599
82. Vale CL, Rydzewska LH, Rovers MM, Emberson JR, Gueyffier F, Stewart LA. Cochrane IPD Meta-Analysis Methods Group. Uptake of systematic reviews and meta-Analyses based on individual participant data in clinical practice guidelines: descriptive study. BMJ 2015; 350: h1088. doi: 10.1136/bmj.h1088