External validation of multivariable prediction models: A systematic review of methodological conduct and reporting

BMC Medical Research Methodology

Volumn 14, Issue 1, 2014, Pages

  Collins, Gary S ^a   De Groot, Joris A ^b   Dutton, Susan ^a   Omar, Omar ^a   Shanyinde, Milensu ^a   Tajar, Abdelouahid ^a   Voysey, Merryn ^a   Wharton, Rose ^a   Yu, Ly Mee ^a   Moons, Karel G ^b   Altman, Douglas G ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

HUMAN; OUTCOME ASSESSMENT; PROGNOSIS; REPRODUCIBILITY; REVIEW; STATISTICAL ANALYSIS; VALIDATION STUDY;

DATA INTERPRETATION, STATISTICAL; HUMANS; OUTCOME ASSESSMENT (HEALTH CARE); PROGNOSIS; REPRODUCIBILITY OF RESULTS; VALIDATION STUDIES AS TOPIC;

EID: 84899459258     PISSN: None     EISSN: 14712288     Source Type: Journal    
DOI: 10.1186/1471-2288-14-40     Document Type: Review

References (62)

1. Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. Collins GS, Mallett S, Omar O, Yu LM, BMC Med 2011 9 103 10.1186/1741-7015-9-103 21902820
2. Reporting methods in studies developing prognostic models in cancer: a review. Mallett S, Royston P, Dutton S, Waters R, Altman DG, BMC Med 2010 8 20 10.1186/1741-7015-8-20 20353578
3. An updated catalog of prostate cancer predictive tools. Shariat SF, Karakiewicz PI, Roehrborn CG, Kattan MW, Cancer 2008 113 3075 3099 10.1002/cncr.23908 18823041
4. Risk assessment of fragility fractures: summary of NICE guidance. Rabar S, Lau R, O'Flynn N, Li L, Barry P, Guideline Development Group, BMJ 2012 345 3698 10.1136/bmj.e3698 22875946
5. European Guidelines on cardiovascular disease prevention in clinical practice (version 2012). The Fifth Joint Task Force of the European Society of Cardiology and Other Societies on Cardiovascular Disease Prevention in Clinical Practice (constituted by representatives of nine societies and by invited experts). Perk J, De Backer G, Gohlke H, Graham I, Reiner Z, Verschuren M, Albus C, Benlian P, Boysen G, Cifkova R, Deaton C, Ebrahim S, Fisher M, Germano G, Hobbs R, Hoes A, Karadeniz S, Mezzani A, Prescott E, Ryden L, Scherer M, Syvanne M, op Reimer WJ S, Vrints C, Wood D, Zamorano JL, Zannad F, Eur Heart J 2012 33 1635 1701 22555213
6. American association of Clinical Endocrinologists' Guidelines for Management of Dyslipidemia and Prevention of Atherosclerosis: executive summary. Jellinger PS, Smith DA, Mehta AE, Ganda O, Handelsman Y, Rodbard HW, Shepherd MD, Seibel JA, Endocr Pract 2012 18 269 293 10.4158/EP.18.2.269 22507559
7. Clinicians versus nomogram: predicting future technetium-99 m bone scan positivity in patients with rising prostate-specific antigen after radical prostatectomy for prostate cancer. Kattan MW, Yu C, Stephenson AJ, Sartor O, Tombal B, Urology 2013 81 956 961 10.1016/j.urology.2012.12.010 23375915
8. Comparisons of nomograms and urologists' predictions in prostate cancer. Ross PL, Gerigk C, Gonen M, Yossepowitch O, Cagiannos I, Sogani PC, Scardino PT, Kattan MW, Semin Urol Oncol 2002 20 82 88 10.1053/suro.2002.32490 12012293
9. Everything you always wanted to know about evaluating prediction models (but were too afraid to ask). Vickers AJ, Cronin AM, Urology 2010 76 6 1298 1301 10.1016/j.urology.2010.06.019 21030068
10. Avoidable waste in the production and reporting of research evidence. Chalmers I, Glasziou P, Lancet 2009 374 86 89 10.1016/S0140-6736(09)60329-9 19525005
11. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. Steyerberg EW, Moons KGM, van der Windt DA, Hayden JA, Perel P, Schroter S, Riley RD, Hemingway H, Altman DG, PLoS Med 2013 10 2 1001381 10.1371/journal.pmed.1001381 23393430
12. Prognosis and prognostic research: validating a prognostic model. Altman DG, Vergouwe Y, Royston P, Moons KGM, BMJ 2009 338 605 10.1136/bmj.b605 19477892
13. What do we mean by validating a prognostic model? Altman DG, Royston P, Stat Med 2000 19 4 453 473 10.1002/(SICI)1097-0258(20000229)19:4<453::AID- SIM350>3.0.CO;2-5 10694730
14. Steyerberg EW, Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating New York: Springer 2009
15. Assessing the performance of prediction models: a framework for traditional and novel measures. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW, Epidemiology 2010 21 1 128 138 10.1097/EDE.0b013e3181c30fb2 20010215
16. Striving for a more perfect peer review editors confront strengths, flaws of biomedical literature. Kuehn MB, JAMA 2013 310 1781 1783 10.1001/jama.2013.280660 24193063
17. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. Boutron I, Dutton S, Ravaud P, Altman DG, JAMA 2010 303 2058 2064 10.1001/jama.2010.651 20501928
18. Overinterpretation and misreporting of diagnostic accuracy studies: evidence of "spin" Ochodo EA, de Haan MC, Reitsma JB, Hooft L, Bossuyt PM, Leeflang MM, Radiology 2013 267 581 588 10.1148/radiol.12120527 23360738
19. Improving validation practices in "Omics" research. Ioannidis JPA, Khoury MJ, Science 2011 334 1230 1232 10.1126/science.1211811 22144616
20. Increasing value and reducing waste in research design, conduct, and analysis. Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R, Lancet 2014 383 166 175 10.1016/S0140-6736(13)62227-8 24411645
21. Scientific inbreeding and same-team replication: type D personality as an example. Ioannidis JPA, J Psychosom Res 2012 73 408 410 10.1016/j.jpsychores. 2012.09.014 23148806
22. A systematic review finds prediction models for chronic kidney were poorly reported and often developed using inappropriate methods. Collins GS, Omar O, Shanyinde M, Yu LM, J Clin Epidemiol 2013 66 268 277 10.1016/j.jclinepi.2012.06.020 23116690
23. Reporting and methods in clinical prediction research: a systematic review. Bouwmeester W, Zuithoff NP, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KG, PLoS Med 2012 9 5 1001221 10.1371/journal.pmed.1001221
24. Clinical prediction models for aneurysmal subarachnoid hemorrhage: a systematic review. Jaja BN, Cusimano MD, Etminan N, Hanggi D, Hasan D, Ilodigwe D, Lantigua H, Le Roux P, Lo B, Louffat-Olivares A, Mayer S, Molyneaux A, Quinn A, Schweizer TA, Schenk T, Spears J, Todd M, Torner J, Vergouwen MD, Wong GK, Singh J, Macdonald RL, Neurocrit Care 2013 18 1 143 153 10.1007/s12028-012-9792- z 23138544
25. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. Steyerberg EW, Harrell FE Jr, Borsboom GJJM, Eijkemans MJC, Vergouwe Y, Habbema JDF, J Clin Epidemiol 2001 54 774 781 10.1016/S0895-4356(01)00341-9 11470385
26. Risk prediction models: II. External validation, model updating, and impact assessment. Moons KG, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, Woodward M, Heart 2012 98 691 698 10.1136/heartjnl-2011-301247 22397946
27. Reporting performance of prognostic models in cancer: a review. Mallett S, Royston P, Waters R, Dutton S, Altman DG, BMC Med 2010 8 21 10.1186/1741-7015-8-21 20353579
28. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. Vergouwe Y, Steyerberg EW, Eijkemans MJC, Habbema JDF, J Clin Epidemiol 2005 58 5 475 483 10.1016/j.jclinepi.2004.06.017 15845334
29. Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines. Burton A, Altman DG, Br J Cancer 2004 91 1 4 8 10.1038/sj.bjc.6601907 15188004
30. Missing covariate data in medical research: to impute is better than to ignore. Janssen KJ, Donders AR, Harrell FE Jr, Vergouwe Y, Chen Q, Grobbee DE, Moons KG, J Clin Epidemiol 2010 63 7 721 727 10.1016/j.jclinepi.2009.12.008 20338724
31. Multiple imputation using chained equations: issues and guidance for practice. White IR, Royston P, Wood AM, Stat Med 2011 30 4 377 399 10.1002/sim.4067 21225900
32. Development and validation of a patient self-assessment score for diabetes risk. Bang H, Edwards AM, Bomback AS, Ballantyne CM, Brillon D, Callahan MA, Teutsch SM, Mushlin AI, Kern LM, Ann Intern Med 2009 151 775 783 10.7326/0003-4819-151-11-200912010-00005 19949143
33. Global Registry of Acute Coronary Events (GRACE) hospital discharge risk score accurately predicts long-term mortality post acute coronary syndrome. Tang EW, Wong CK, Herbison P, Am Heart J 2007 153 29 35 10.1016/j.ahj.2006.10.004 17174633
34. External validation of a cox prognostic model: principles and methods. Royston P, Altman DG, BMC Med Res Methodol 2013 13 1 33 10.1186/1471-2288-13-33 23496923
35. External validation is necessary in prediction research: a clinical example. Bleeker SE, Moll HA, Steyerberg EW, Donders ART, Derksen-Lubsen G, Grobbee DE, Moons KGM, J Clin Epidemiol 2003 56 9 826 832 10.1016/S0895-4356(03) 00207-5 14505766
36. Predicting risk of osteoporotic and hip fracture in the United Kingdom: prospective independent and external validation of QFractureScores. Collins GS, Mallett S, Altman DG, BMJ 2011 342 3651 10.1136/bmj.d3651 21697214
37. Fracture risk assessment: state of the art, methodologically unsound, or poorly reported? Collins GS, Michaëlsson K, Curr Osteoporos Rep 2013 10 199 207
38. Clinician's guide to prevention and treatment of osteoporosis. National Osteoporosis Foundation, 2010 (available at http://nof.org/hcp/clinicians-guide)
39. The use of clinical risk factors enhances the performance of BMD in the prediction of hip and osteoporotic fractures in men and women. Kanis JA, Oden A, Johnell O, Johansson H, De Laet C, Brown J, Burckhardt P, Cooper C, Christiansen C, Cummings S, Eisman JA, Fujiwara S, Gluer C, Goltzman D, Krieg MA HD, La Croix A, McCloskey E, Mellstrom D, Melton LJ, Pols H, Reeve J, Sanders K, Schott AM, Silman A, Torgerson D, van Staa T, Watts NB, Yoshimura N, Osteoporos Int 2007 18 1033 1046 10.1007/s00198-007-0343-y 17323110
40. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. Sterne JAC, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, Wood AM, Carpenter JR, BMJ 2009 338 2393 10.1136/bmj.b2393 19564179
41. Issues in multiple imputation of missing data for large general practice clinical databases. Marston L, Carpenter JR, Walters KR, Morris RW, Nazareth I, Petersen I, Pharmacoepidemiol Drug Saf 2010 19 6 618 626 10.1002/pds.1934 20306452
42. The art versus science of predicting prognosis: can a prognostic index predict short-term mortality better than experienced nurses do? Casarett DJ, Farrington S, Craig T, Slattery J, Harrold J, Oldanie B, Roy J, Biehl R, Teno J, J Palliat Med 2012 15 6 703 708 10.1089/jpm.2011.0531 22583382
43. Dealing with missing outcome data in randomized trials and observational studies. Groenwold RH, Donders AR, Roes KC, Harrell FE Jr, Moons KG, Am J Epidemiol 2012 175 3 210 217 10.1093/aje/kwr302 22262640
44. Missing data and imputation: a practical illustration in a prognostic study on low back pain. Vergouw D, Heymans MW, van der Windt DA, Foster NE, Dunn KM, van der Horst HE, de Vet HC, J Manipulative Physiol Ther 2012 35 6 464 471 10.1016/j.jmpt.2012.07.002 22964020
45. Prediction models in cancer care. Vickers AJ, CA Cancer J Clin 2011 61 5 315 326
46. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio) marker. Moons KG, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, Grobbee DE, Heart 2012 98 683 690 10.1136/heartjnl-2011-301246 22397945
47. Risk prediction models for patients with chronic kidney disease: a systematic review. Tangri N, Kitsios GD, Inker LA, Griffith J, Naimark DM, Walker S, Rigatto C, Uhlig K, Kent DM, Levey AS, Ann Intern Med 2013 158 596 603 10.7326/0003-4819-158-8-201304160-00004 23588748
48. Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited. Kramer AA, Zimmerman JE, Crit Care Med 2007 35 9 2052 2056 10.1097/01.CCM.0000275267.64078.B0 17568333
49. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. Moons KGM, Altman DG, Vergouwe Y, Royston P, BMJ 2009 338 606 10.1136/bmj.b606 19502216
50. Decision curve analysis: a novel method for evaluating prediction models. Vickers AJ, Elkin EB, Med Decis Making 2006 26 6 565 574 10.1177/ 0272989X06295361 17099194
51. Using relative utility curves to evaluate risk prediction. Baker SG, Cook NR, Vickers A, Kramer BS, J R Stat Soc Ser A Stat Soc 2009 172 729 748 10.1111/j.1467-985X.2009.00592.x 20069131
52. Impact of recent screening on predicting the outcome of prostate cancer biopsy in men with elevated prostate-specific antigen: data from the European Randomized Study of Prostate Cancer Screening in Gothenburg, Sweden. Vickers AJ, Cronin AM, Aus G, Pihl CG, Becker C, Pettersson K, Scardino PT, Hugosson J, Lilja H, Cancer 2010 116 11 2612 2620 20336781
53. Search filters for finding prognostic and diagnostic prediction studies in medline to enhance systematic reviews. Geersing GJ, Bouwmeester W, Zuithoff P, Spijker R, Leeflang M, Moons K, PLoS One 2012 7 2 32844 10.1371/journal.pone. 0032844 22393453
54. Searching for clinical prediction rules in MEDLINE. Ingui BJ, Rogers MAM, J Am Med Inform Assoc 2001 8 4 391 397 10.1136/jamia.2001.0080391 11418546
55. Optimal search strategies for detecting clinically sound prognostic studies in EMBASE: an analytic survery. Wilczynski NL, Haynes RB, J Am Med Inform Assoc 2005 12 4 481 485 10.1197/jamia.M1752 15802476
56. Prediction models for prolonged intensive care unit stay after cardiac surgery: systematic review and validation study. Ettema RG, Peelen LM, Schuurmans MJ, Nierich AP, Kalkman CJ, Moons KG, Circulation 2010 122 7 682 689 687 p following p 689 10.1161/CIRCULATIONAHA.109.926808 20679549
57. Risk models to predict chronic kidney disease and its progression: a systematic review. Echouffo-Tcheugui JB, Kengne AP, PLoS Med 2012 9 11 1001344 10.1371/journal.pmed.1001344 23185136
58. Inventory of prostate cancer predictive tools. Shariat SF, Karakiewicz PI, Margulis V, Kattan MW, Curr Opin Urol 2008 18 279 296 10.1097/MOU. 0b013e3282f9b3e5 18382238
59. Clinical value of prognostic instruments to identify patients with an increased risk for osteoporotic fractures: systematic review. Steurer J, Haller C, Hauselmann H, Brunner F, Bachmann LM, PLoS One 2011 6 5 19994 10.1371/journal.pone.0019994 21625596
60. Aggregating published prediction models with individual participant data: a comparison of different approaches. Debray TP, Koffijberg H, Vergouwe Y, Moons KG, Steyerberg EW, Stat Med 2012 31 2697 2712 10.1002/sim.5412 22733546
61. A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis. Debray TP, Moons KG, Ahmed I, Koffijberg H, Riley RD, Stat Med 2013 32 3158 3180 10.1002/sim.5732 23307585
62. Opening up multivariable prediction models: consensus-based guidelines for transparent reporting. [ http://blogs.bmj.com/bmj/2011/08/03/gary-collins- opening-up-multivariable-prediction-models/ ]