Design characteristics influence performance of clinical prediction rules in validation: A meta-epidemiological study

PLoS ONE

Volumn 11, Issue 1, 2016, Pages

  Ban, Jong Wook ^a   Emparanza, José Ignacio ^b   Urreta, Iratxe ^b   Burls, Amanda ^c  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b HOSPITAL DONOSTIA   (Spain)

^c CITY UNIVERSITY   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

DATA BASE; HUMAN; META ANALYSIS; POPULATION BASED CASE CONTROL STUDY; PREDICTION; RELIABILITY; RISK; SAMPLE SIZE; SYSTEMATIC REVIEW; VALIDATION PROCESS; VALIDATION STUDY; DECISION SUPPORT SYSTEM; EPIDEMIOLOGY; MEDICAL RESEARCH; METHODOLOGY; ODDS RATIO; PROCEDURES; REPRODUCIBILITY; RESEARCH; STANDARDS; STATISTICS AND NUMERICAL DATA;

BIOMEDICAL RESEARCH; DECISION SUPPORT TECHNIQUES; EPIDEMIOLOGIC STUDIES; HUMANS; ODDS RATIO; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; RESEARCH REPORT; SAMPLE SIZE; VALIDATION STUDIES AS TOPIC;

EID: 84953776489     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0145779     Document Type: Article

References (59)

1. Wasson JH, Sox HC, Neff RK, Goldman L. Clinical prediction rules. Applications and methodological standards. The New England journal of medicine. 1985; 313(13):793-9. Epub 1985/09/26. doi: 10. 1056/NEJM198509263131306 .
2. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA: the journal of the American Medical Association. 1997; 277(6):488-94. Epub 1997/02/12. .
3. McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA: the journal of the American Medical Association. 2000; 284(1):79-84. Epub 2000/06/29.
4. Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Annals of internal medicine. 1999; 130(6):515-24. .
5. Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Annals of internal medicine. 2006; 144(3):201-9. Epub 2006/02/08. .
6. Toll DB, Janssen KJ, Vergouwe Y, Moons KG. Validation, updating and impact of clinical prediction rules: a review. Journal of clinical epidemiology. 2008; 61(11):1085-94. Epub 2009/02/12. doi: 10. 1016/j.jclinepi.2008.04.008 .
7. Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. Bmj. 2009; 338:b605. Epub 2009/05/30. doi: 10.1136/bmj.b605 .
8. Hendriksen JM, Geersing GJ, Moons KG, de Groot JA. Diagnostic and prognostic prediction models. Journal of thrombosis and haemostasis: JTH. 2013; 11 Suppl 1:129-41. doi: 10.1111/jth.12262 .
9. Bleeker SE, Moll HA, Steyerberg EW, Donders AR, Derksen-Lubsen G, Grobbee DE, et al. External validation is necessary in prediction research: a clinical example. Journal of clinical epidemiology. 2003; 56(9):826-32. .
10. Steyerberg EW, Bleeker SE, Moll HA, Grobbee DE, Moons KG. Internal and external validation of predictive models: a simulation study of bias and precision in small samples. Journal of clinical epidemiology. 2003; 56(5):441-7. Epub 2003/06/19. .
11. Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. New York: Springer; 2009. xxviii, 497 p. p.
12. Wallace E, Smith SM, Perera-Salazar R, Vaucher P, McCowan C, Collins G, et al. Framework for the impact analysis and implementation of Clinical Prediction Rules (CPRs). BMC medical informatics and decision making. 2011; 11:62. Epub 2011/10/18. doi: 10.1186/1472-6947-11-62 ; PubMed Central PMCID: PMC3216240.
13. Stiell IG, Wells GA. Methodologic standards for the development of clinical decision rules in emergency medicine. Annals of emergency medicine. 1999; 33(4):437-47. .
14. Mallett S, Royston P, Dutton S, Waters R, Altman DG. Reporting methods in studies developing prognostic models in cancer: a review. BMC medicine. 2010; 8:20. doi: 10.1186/1741-7015-8-20 ; PubMed Central PMCID: PMC2856521.
15. Mallett S, Royston P, Waters R, Dutton S, Altman DG. Reporting performance of prognostic models in cancer: a review. BMC medicine. 2010; 8:21. doi: 10.1186/1741-7015-8-21 ; PubMed Central PMCID: PMC2857810.
16. Collins GS, Mallett S, Omar O, Yu LM. Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. BMC medicine. 2011; 9:103. doi: 10.1186/1741-7015-9-103 ; PubMed Central PMCID: PMC3180398.
17. Bouwmeester W, Zuithoff NP, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS medicine. 2012; 9(5):1-12. Epub 2012/05/26. doi: 10.1371/journal.pmed.1001221 ; PubMed Central PMCID: PMC3358324.
18. Collins GS, de Groot JA, Dutton S, Omar O, Shanyinde M, Tajar A, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC medical research methodology. 2014; 14:40. doi: 10.1186/1471-2288-14-40 ; PubMed Central PMCID: PMC3999945.
19. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA: the journal of the American Medical Association. 1995; 273(5):408-12. .
20. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. Bmj. 2008; 336(7644):601-5. doi: 10.1136/bmj.39465.451748.AD
21. Odgaard-Jensen J, Vist GE, Timmer A, Kunz R, Akl EA, Schunemann H, et al. Randomisation to protect against selection bias in healthcare trials. The Cochrane database of systematic reviews. 2011;(4: ):MR000012. doi: 10.1002/14651858.MR000012.pub3 .
22. Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Annals of internal medicine. 2012; 157(6):429-38. doi: 10.7326/0003-4819-157-6-201209180-00537 .
23. Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA: the journal of the American Medical Association. 1999; 282(11):1061-6. .
24. Rutjes AW, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ: Canadian Medical Association journal = journal de l'Association medicale canadienne. 2006; 174(4):469-76. doi: 10.1503/cmaj.050090 ; PubMed Central PMCID: PMC1373751.
25. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Annals of internal medicine. 2003; 138(1):W1-12. .
26. Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC medical research methodology. 2003; 3:25. doi: 10.1186/1471-2288-3-25 ; PubMed Central PMCID: PMC305345.
27. Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of internal medicine. 2011; 155 (8):529-36. doi: 10.7326/0003-4819-155-8-201110180-00009 .
28. Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. Journal of clinical epidemiology. 2005; 58(5):475-83. Epub 2005/04/23. doi: 10.1016/j.jclinepi.2004.06.017 .
29. Whiting P, Rutjes AW, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med. 2004; 140(3):189-202. Epub 2004/02/06. .
30. Whiting PF, Rutjes AW, Westwood ME, Mallett S, Group Q-S. A systematic review classifies sources of bias and variation in diagnostic test accuracy studies. J Clin Epidemiol. 2013; 66(10):1093-104. doi: 10.1016/j.jclinepi.2013.05.014
31. Rutjes AW, Reitsma JB, Vandenbroucke JP, Glas AS, Bossuyt PM. Case-control and two-gate designs in diagnostic accuracy studies. Clin Chem. 2005; 51(8):1335-41. doi: 10.1373/clinchem.2005.048595 .
32. Wong SS, Wilczynski NL, Haynes RB, Ramkissoonsingh R, Hedges T. Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE. AMIA Annual Symposium proceedings /AMIA Symposium AMIA Symposium. 2003:728-32
33. Holland JL, Wilczynski NL, Haynes RB, Hedges T. Optimal search strategies for identifying sound clinical prediction studies in EMBASE. BMC medical informatics and decision making. 2005; 5:11. doi: 10. 1186/1472-6947-5-11 ; PubMed Central PMCID: PMC1097733.
34. Montori VM, Wilczynski NL, Morgan D, Haynes RB, Hedges T. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. Bmj. 2005; 330(7482):68. doi: 10.1136/bmj.38336. 804167.47
35. Wilczynski NL, Haynes RB, Hedges T. EMBASE search strategies achieved high sensitivity and specificity for retrieving methodologically sound systematic reviews. Journal of clinical epidemiology. 2007; 60(1):29-33. doi: 10.1016/j.jclinepi.2006.04.001 .
36. Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Statistics in medicine. 2002; 21(11):1513-24. doi: 10.1002/sim.1184 .
37. Zamora J, Abraira V, Muriel A, Khan K, Coomarasamy A. Meta-DiSc: a software for meta-Analysis of test accuracy data. BMC medical research methodology. 2006; 6:31. doi: 10.1186/1471-2288-6-31 ; PubMed Central PMCID: PMC1552081.
38. Littenberg B, Moses LE. Estimating diagnostic accuracy from multiple conflicting reports: a new metaanalytic method. Medical decision making: an international journal of the Society for Medical Decision Making. 1993; 13(4):313-21. .
39. Moses LE, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test into a summary ROC curve: data-Analytic approaches and some additional considerations. Statistics in medicine. 1993; 12(14):1293-316. .
40. StataCorp. Stata Statistical Software: Release 12. College Station, TX: StataCorp LP; 2011.
41. Kontopantelis E, Reeves D. metaan: Random-effects meta-Analysis. Stata J. 2010; 10(3):395-407. PMID: WOS:000282853700007.
42. Gilbody S, Richards D, Brealey S, Hewitt C. Screening for depression in medical settings with the Patient Health Questionnaire (PHQ): a diagnostic meta-Analysis. Journal of general internal medicine. 2007; 22(11):1596-602. Epub 2007/09/18. doi: 10.1007/s11606-007-0333-y ; PubMed Central PMCID: PMC2219806.
43. Warnick EM, Bracken MB, Kasl S. Screening efficiency of the child behavior checklist and strengths and difficulties questionnaire: A systematic review. Child and Adolescent Mental Health. 2008; 13 (3):140-7. doi: http://dx.doi.org/10.1111/j.1475-3588.2007.00461.x. 44.
44. Banal F, Dougados M, Combescure C, Gossec L. Sensitivity and specificity of the American College of Rheumatology 1987 criteria for the diagnosis of rheumatoid arthritis according to disease duration: a systematic literature review and meta-Analysis. Annals of the rheumatic diseases. 2009; 68(7):1184-91. Epub 2008/08/30. doi: 10.1136/ard.2008.093187 .
45. Dowling S, Spooner CH, Liang Y, Dryden DM, Friesen C, Klassen TP, et al. Accuracy of Ottawa Ankle Rules to exclude fractures of the ankle and midfoot in children: a meta-Analysis. Academic emergency medicine: official journal of the Society for Academic Emergency Medicine. 2009; 16(4):277-87. Epub 2009/02/04. doi: 10.1111/j.1553-2712.2008.00333.x .
46. Geomini P, Kruitwagen R, Bremer GL, Cnossen J, Mol BW. The accuracy of risk scores in predicting ovarian malignancy: a systematic review. Obstetrics and gynecology. 2009; 113(2 Pt 1):384-94. Epub 2009/01/22. doi: 10.1097/AOG.0b013e318195ad17 .
47. Hewitt C, Gilbody S, Brealey S, Paulden M, Palmer S, Mann R, et al. Methods to identify postnatal depression in primary care: an integrated evidence synthesis and value of information analysis. Health Technol Assess. 2009; 13(36):1-145, 7-230. Epub 2009/07/25. doi: 10.3310/hta13360 .
48. Brennan C, Worrall-Davies A, McMillan D, Gilbody S, House A. The Hospital Anxiety and Depression Scale: a diagnostic meta-Analysis of case-finding ability. Journal of psychosomatic research. 2010; 69 (4):371-8. Epub 2010/09/18. doi: 10.1016/j.jpsychores.2010.04.006 .
49. Ceriani E, Combescure C, Le Gal G, Nendaz M, Perneger T, Bounameaux H, et al. Clinical prediction rules for pulmonary embolism: a systematic review and meta-Analysis. Journal of thrombosis and haemostasis: JTH. 2010; 8(5):957-70. Epub 2010/02/13. doi: 10.1111/j.1538-7836.2010.03801.x .
50. Giles MF, Rothwell PM. Systematic review and pooled analysis of published and unpublished validations of the ABCD and ABCD2 transient ischemic attack risk scores. Stroke; a journal of cerebral circulation. 2010; 41(4):667-73. Epub 2010/02/27. doi: 10.1161/STROKEAHA.109.571174 .
51. Harrington L, Luquire R, Vish N, Winter M, Wilder C, Houser B, et al. Meta-Analysis of fall-risk tools in hospitalized adults. The Journal of nursing administration. 2010; 40(11):483-8. Epub 2010/10/28. doi: 10.1097/NNA.0b013e3181f88fbd .
52. McPhail MJ, Wendon JA, Bernal W. Meta-Analysis of performance of Kings's College Hospital Criteria in prediction of outcome in non-paracetamol-induced acute liver failure. Journal of hepatology. 2010; 53(3):492-9. Epub 2010/06/29. doi: 10.1016/j.jhep.2010.03.023 .
53. Mitchell AJ, Bird V, Rizzo M, Meader N. Diagnostic validity and added value of the Geriatric Depression Scale for depression in primary care: a meta-Analysis of GDS30 and GDS15. Journal of affective disorders. 2010; 125(1-3):10-7. Epub 2009/10/06. doi: 10.1016/j.jad.2009.08.019 .
54. Mitchell AJ, Bird V, Rizzo M, Meader N. Which version of the geriatric depression scale is most useful in medical settings and nursing homes? Diagnostic validity meta-Analysis. The American journal of geriatric psychiatry: official journal of the American Association for Geriatric Psychiatry. 2010; 18(12):1066-77. Epub 2010/12/16. .
55. Mitchell AJ, Meader N, Symonds P. Diagnostic validity of the Hospital Anxiety and Depression Scale (HADS) in cancer and palliative settings: a meta-Analysis. Journal of affective disorders. 2010; 126 (3):335-48. doi: 10.1016/j.jad.2010.01.067 .
56. Serrano LA, Hess EP, Bellolio MF, Murad MH, Montori VM, Erwin PJ, et al. Accuracy and quality of clinical decision rules for syncope in the emergency department: a systematic review and meta-Analysis. Annals of emergency medicine. 2010; 56(4):362-73 e1. Epub 2010/09/28. doi: 10.1016/j. annemergmed.2010.05.013 ; PubMed Central PMCID: PMC2946941.
57. Stengel D, Bauwens K, Rademacher G, Mutze S, Ekkernkamp A. Association between compliance with methodological standards of diagnostic research and reported test accuracy: meta-Analysis of focused assessment of US for trauma. Radiology. 2005; 236(1):102-11. doi: 10.1148/radiol. 2361040791 .
58. Hulsemann JL, Zeidler H. Diagnostic evaluation of classification criteria for rheumatoid arthritis and reactive arthritis in an early synovitis outpatient clinic. Annals of the rheumatic diseases. 1999; 58 (5):278-80. ; PubMed Central PMCID: PMC1752884.
59. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Annals of internal medicine. 2015; 162(1):55-63. doi: 10.7326/M14-0697