A new framework to enhance the interpretation of external validation studies of clinical prediction models

Journal of Clinical Epidemiology

Volumn 68, Issue 3, 2015, Pages 279-289

  Debray, Thomas P A ^a   Vergouwe, Yvonne ^b   Koffijberg, Hendrik ^a   Nieboer, Daan ^b   Steyerberg, Ewout W ^b   Moons, Karel G M ^a  

^a UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

^b ERASMUS MEDICAL CENTER   (Netherlands)

Author keywords

Case mix; Generalizability; Prediction model; Reproducibility; Transportability; Validation

Indexed keywords

D DIMER;

ARTICLE; CASE MIX; CONFIDENCE INTERVAL; DEEP VEIN THROMBOSIS; DISCRIMINANT ANALYSIS; FEMALE; HUMAN; LOGISTIC REGRESSION ANALYSIS; MAJOR CLINICAL STUDY; MALE; PREDICTION; PRIORITY JOURNAL; RECEIVER OPERATING CHARACTERISTIC; REPRODUCIBILITY; VALIDATION STUDY; OUTCOME ASSESSMENT; PREDICTIVE VALUE; STATISTICAL ANALYSIS; STATISTICAL MODEL;

DATA INTERPRETATION, STATISTICAL; FEMALE; HUMANS; MALE; MODELS, STATISTICAL; OUTCOME ASSESSMENT (HEALTH CARE); PREDICTIVE VALUE OF TESTS; REPRODUCIBILITY OF RESULTS; VALIDATION STUDIES AS TOPIC;

EID: 84923527988     PISSN: 08954356     EISSN: 18785921     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2014.06.018     Document Type: Article

References (54)

1. M.H. Gail, L.A. Brinton, D.P. Byar, D.K. Corle, S.B. Green, and C. Schairer Projecting individualized probabilities of developing breast cancer for white females who are being examined annually J Natl Cancer Inst 81 1989 1879 1886
2. P.W. Wilson, R.B. DAgostino, D. Levy, A.M. Belanger, H. Silbershatz, and W.B. Kannel Prediction of coronary heart disease using risk factor categories Circulation 97 1998 1837 1847
3. K.G.M. Moons, A.P. Kengne, D.E. Grobbee, P. Royston, Y. Vergouwe, and D.G. Altman Risk prediction models: II. External validation, model updating, and impact assessment Heart 98 9 2012 691 698
4. D.G. Altman, Y. Vergouwe, P. Royston, and K.G.M. Moons Prognosis and prognostic research: validating a prognostic model BMJ 338 2009 b605
5. B.M. Reilly, and A.T. Evans Translating clinical research into clinical practice: impact of using prediction rules to make decisions Ann Intern Med 144 2006 201209
6. D.G. Altman, and P. Royston What do we mean by validating a prognostic model? Stat Med 19 2000 453 473
7. A.C. Justice, K.E. Covinsky, and J.A. Berlin Assessing the generalizability of prognostic information Ann Intern Med 130 1999 515 524
8. E.W. Steyerberg, K.G.M. Moons, D.A. van der Windt, J.A. Hayden, P. Perel, and S. Schroter Prognosis research strategy (PROGRESS) 3: prognostic model research PLoS Med 10 2 2012 e1001381
9. K.G.M. Moons, P. Royston, Y. Vergouwe, D.E. Grobbee, and D.G. Altman Prognosis and prognostic research: what, why, and how? BMJ 338 2009 b375
10. K.G.M. Moons, D.G. Altman, Y. Vergouwe, and P. Royston Prognosis and prognostic research: application and impact of prognostic models in clinical practice BMJ 338 2009 b606
11. E.W. Steyerberg Clinical prediction models: a practical approach to development, validation, and updating 2009 Springer New York, NY
12. F.E. Harrell Jr. Regression modeling strategies with applications to linear models, logistic regression and survival analysis 2001 Springer New York, NY
13. D.B. Toll, K.J.M. Janssen, Y. Vergouwe, and K.G.M. Moons Validation, updating and impact of clinical prediction rules: a review J Clin Epidemiol 61 2008 1085 1094
14. J.A. Knottnerus Prediction rules: statistical reproducibility and clinical similarity Med Decis Making 12 1992 286 287
15. J.A. Knottnerus Diagnostic prediction rules: principles, requirements and pitfalls Prim Care 22 1995 341 363
16. C.M. Smith-Spangler Transparency and reproducible research in modeling Med Decis Making 32 2012 663 666
17. D.B. Toll, R. Oudega, Y. Vergouwe, K.G.M. Moons, and A.W. Hoes A new diagnostic rule for deep vein thrombosis: safety and efficiency in clinically relevant subgroups Fam Pract 25 1 2008 3 8
18. G.J. Geersing, N.P.A. Zuithoff, C. Kearon, D.R. Anderson, A.J. Ten Cate-Hoek, and J.L. Elf Exclusion of deep vein thrombosis using the Wells-rule in various clinically important subgroups; an individual patient data meta-analysis BMJ 348 2014 g1340
19. T.P.A. Debray, K.G.M. Moons, I. Ahmed, H. Koffijberg, and R.D. Riley A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis Stat Med 32 2013 3158 3180
20. Y. Vergouwe, K.G.M. Moons, and E.W. Steyerberg External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients Am J Epidemiol 172 2010 971 980
21. Y. Vergouwe, E.W. Steyerberg, M.J.C. Eijkemans, and J.D.F. Habbema Validity of prognostic models: when is a model clinically useful? Semin Urol Oncol 20 2 2002 96 107
22. G.S. Collins, and D.G. Altman Identifying patients with undetected colorectal cancer: an independent validation of QCancer (colorectal) Br J Cancer 107 2012 260 265
23. P.C. OBrien Comparing two samples: extensions of the t, rank-sum, and log-rank tests J Am Stat Assoc 83 1988 52 61
24. B. Roozenbeek, H.F. Lingsma, F.E. Lecky, J. Lu, J. Weir, and I. Butcher Prediction of outcome after moderate and severe traumatic brain injury: external validation of the International Mission on Prognosis and Analysis of Clinical Trials (IMPACT) and Corticoid Randomisation after Significant Head injury (CRASH) prognostic models Crit Care Med 40 2012 1609 1617
25. S.M. Hailpern, and P.F. Visintainer Odds ratios and logistic regression: further examples of their use and interpretation Stata J 3 3 2003 213 225
26. M.H. Gail, and R.M. Pfeiffer On criteria for evaluating models of absolute risk Biostatistics 6 2 2005 227 239
27. E.W. Steyerberg, A.J. Vickers, N.R. Cook, T. Gerds, M. Gonen, and N. Obuchowski Assessing the performance of prediction models: a framework for traditional and novel measures Epidemiology 21 2010 128 138
28. T. Fawcett ROC graphs: notes and practical considerations for researchers 2004 Kluwer Academic Publishers, HP Labs Tech Report The Netherlands HPL-2003-4
29. M.E. Miller, C.D. Langefeld, W.M. Tierney, S.L. Hui, and C.J. McDonald Validation of probabilistic predictions Med Decis Making 13 1993 49 58
30. J. Hilden The area under the ROC curve and its competitors Med Decis Making 11 1991 95 101
31. K.G.M. Moons, A.P. Kengne, M. Woodward, P. Royston, Y. Vergouwe, and D.G. Altman Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker Heart 98 9 2012 683 690
32. S.D. Walter Variation in baseline risk as an explanation of heterogeneity in meta-analysis Stat Med 16 1997 2883 2900
33. D.R. Cox Two further applications of a model for binary regression Biometrika 45 3 1958 562 565
34. K.J.M. Janssen, K.G.M. Moons, C.J. Kalkman, D.E. Grobbee, and Y. Vergouwe Updating methods improved the performance of a clinical prediction model in new patients J Clin Epidemiol 61 2008 76 86
35. E.W. Steyerberg, G.J.J. Borsboom, H.C. van Houwelingen, M.J.C. Eijkemans, and J.D.F. Habbema Validation and updating of predictive logistic regression models: a study on sample size and shrinkage Stat Med 23 2004 2567 2586
36. H.C. van Houwelingen Validation, calibration, revision and combination of prognostic survival models Stat Med 19 2000 3401 3415
37. K.J.M. Janssen, Y. Vergouwe, C.J. Kalkman, D.E. Grobbee, and K.G.M. Moons A simple method to adjust clinical prediction models to local circumstances Can J Anesth 56 3 2009 194 201
38. J. Locke. An essay concerning human understanding. 1689.
39. B.J. Calder, L.W. Phillips, and A.M. Tybout The concept of external validity J Cons Res 9 3 1982 240 244
40. M. Maclure Popperian refutation in epidemiology Am J Epidemiol 121 1985 343 350
41. J.W. Lucas Theory-testing, generalization, and the problem of external validity Soc Theor 21 3 2003 236 253
42. E.A. Ochodo, M.C. de Haan, J.B. Reitsma, L. Hooft, P.M. Bossuyt, and M.M.G. Leeflang Overinterpretation and misreporting of diagnostic accuracy studies: evidence of "spin" Radiology 267 2013 581 588
43. G.S. Collins, J.A. de Groot, S. Dutton, O. Omar, M. Shanyinde, and A. Tajar External validation of multivariable prediction models: a systematic review of methodological conduct and reporting BMC Med Res Methodol 14 2014 40
44. J.P.A. Ioannidis, S. Greenland, M.A. Hlatky, M.J. Khoury, M.R. Macleod, and D. Moher Increasing value and reducing waste in research design, conduct, and analysis Lancet 383 2014 166 175
45. B.H. Willis Spectrum bias - why clinicians need to be cautious when applying diagnostic test studies Fam Pract 25 2008 390 396
46. R. Oudega, A.W. Hoes, and K.G.M. Moons The Wells rule does not adequately rule out deep venous thrombosis in primary care patients Ann Intern Med 143 2005 100 107
47. W. Bouwmeester, N.P.A. Zuithoff, S. Mallett, M.I. Geerlings, Y. Vergouwe, and E.W. Steyerberg Reporting and methods in clinical prediction research: a systematic review PLoS Med 9 5 2012 1 12
48. C.P. Gross, R. Mallory, A. Heiat, and H.M. Krumholz Reporting the recruitment process in clinical trials: who are these patients and how did they get there? Ann Intern Med 137 2002 10 16
49. K.F. Schulz, D.G. Altman, and D. Moher CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials BMJ 340 2010 c332
50. G.S. Collins Opening up multivariable prediction models: consensus-based guidelines for transparent reporting 2011 BMJ Blogs London, UK, Available at http://blogs.bmj.com/bmj/2011/08/03/gary-collins-opening-up-multivariable-prediction-models/. Accessed April 4, 2013
51. T.E. Clemons, and E.L. Bradley Jr. A nonparametric measure of the overlapping coefficient Comput Stat Data 34 1 2000 51 61
52. S. Mizuno, T. Yamaguchi, A. Fukushima, Y. Matsuyama, and Y. Ohashi Overlap coefficient for assessing the similarity of pharmacokinetic data between ethnically different populations Clin Trials 2 2005 174 181
53. A.R. de Leon, and K.C. Carriere A generalized Mahalanobis distance for mixed data J Multivariate Anal 92 2005 174 185
54. J.C. Wyatt, and D.G. Altman Prognostic models: clinically useful or quickly forgotten? BMJ 311 1995 1539