Poor performance of clinical prediction models: the harm of commonly applied methods

Journal of Clinical Epidemiology

Volumn 98, Issue , 2018, Pages 133-143

  Steyerberg, Ewout W ^a,b   Uno, Hajime ^c   Ioannidis, John P A ^d,e   van Calster, Ben ^a,f   Ukaegbu, Chinedu ^c   Dhingra, Tara ^c   Syngal, Sapna ^c   Kastrinos, Fay ^g  

^a LEIDEN UNIVERSITY MEDICAL CENTER   (Netherlands)

^b ERASMUS MEDICAL CENTER   (Netherlands)

^c DANA FARBER CANCER INSTITUTE   (United States)

^d STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

^e STANFORD UNIVERSITY   (United States)

^f UNIVERSITY OF LEUVEN   (Belgium)

^g Department of Medicine   (United States)

Author keywords

Events per variable; Prediction model; Regression analysis; Sample size; Simulation; Validation

Indexed keywords

DNA MISMATCH REPAIR PROTEIN MSH2; MISMATCH REPAIR PROTEIN PMS2; MUTL PROTEIN HOMOLOG 1; PROTEIN MSH6;

ADULT; ARTICLE; CANCER PREVENTION; CANCER RISK; COHORT ANALYSIS; ENDOMETRIUM CANCER; FAMILY HISTORY; FEMALE; FIRST-DEGREE RELATIVE; GERMLINE MUTATION; HEREDITARY NONPOLYPOSIS COLORECTAL CANCER; HUMAN; MAJOR CLINICAL STUDY; MALE; MISMATCH REPAIR; PREDICTION; PRIORITY JOURNAL; REGRESSION ANALYSIS; SECOND-DEGREE RELATIVE; SIMULATION; STATISTICAL ANALYSIS; STATISTICAL MODEL; STATISTICAL SIGNIFICANCE; VALIDATION STUDY; AGE; BIOLOGICAL MODEL; GENETIC SCREENING; GENETICS; MIDDLE AGED; PERSONALIZED MEDICINE; PREDICTIVE VALUE; SAMPLE SIZE; SEX FACTOR;

AGE FACTORS; COLORECTAL NEOPLASMS, HEREDITARY NONPOLYPOSIS; DNA MISMATCH REPAIR; FEMALE; GENETIC TESTING; GERM-LINE MUTATION; HUMANS; MALE; MIDDLE AGED; MODELS, GENETIC; PRECISION MEDICINE; PREDICTIVE VALUE OF TESTS; REGRESSION ANALYSIS; SAMPLE SIZE; SEX FACTORS;

EID: 85039444017     PISSN: 08954356     EISSN: 18785921     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2017.11.013     Document Type: Article

References (73)

1. Kattan, M.W., Hess, K.R., Amin, M.B., Lu, Y., Moons, K.G., Gershenwald, J.E., et al. American Joint Committee on Cancer acceptance criteria for inclusion of risk models for individualized prognosis in the practice of precision medicine. CA Cancer J Clin 66 (2016), 370–374.
2. Moons, K.G., Altman, D.G., Reitsma, J.B., Ioannidis, J.P., Macaskill, P., Steyerberg, E.W., et al. Transparent Reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 162 (2015), W1–W73.
3. Mushkudiani, N.A., Hukkelhoven, C.W., Hernandez, A.V., Murray, G.D., Choi, S.C., Maas, A.I., et al. A systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomes. J Clin Epidemiol 61 (2008), 331–343.
4. Altman, D.G., Prognostic models: a methodological framework and review of models for breast cancer. Cancer Invest 27 (2009), 235–243.
5. Mallett, S., Royston, P., Dutton, S., Waters, R., Altman, D.G., Reporting methods in studies developing prognostic models in cancer: a review. BMC Med, 8, 2010, 20.
6. Collins, G.S., Mallett, S., Omar, O., Yu, L.M., Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. BMC Med, 9, 2011, 103.
7. Bouwmeester, W., Zuithoff, N.P., Mallett, S., Geerlings, M.I., Vergouwe, Y., Steyerberg, E.W., et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med 9 (2012), 1–12.
8. Collins, G.S., Omar, O., Shanyinde, M., Yu, L.M., A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods. J Clin Epidemiol 66 (2013), 268–277.
9. Steyerberg, E.W., Clinical prediction models: a practical approach to development, validation, and updating. 2009, Springer, New York.
10. Vergouwe, Y., Steyerberg, E.W., Eijkemans, M.J., Habbema, J.D., Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 58 (2005), 475–483.
11. Collins, G.S., Ogundimu, E.O., Altman, D.G., Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med 35 (2016), 214–226.
12. Van Calster, B., Steyerberg, E.W., Bourne, T., Timmerman, D., Collins, G.S., Flawed external validation study of the ADNEX model to diagnose ovarian cancer. Gynecol Oncol Rep 18 (2016), 49–50.
13. Collins, G.S., de Groot, J.A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol, 14, 2014, 40.
14. Siontis, G.C., Tzoulaki, I., Castaldi, P.J., Ioannidis, J.P., External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 68 (2015), 25–34.
15. Starmans, R., Muris, J.W., Fijten, G.H., Schouten, H.J., Pop, P., Knottnerus, J.A., The diagnostic value of scoring models for organic and non-organic gastrointestinal disease, including the irritable-bowel syndrome. Med Decis Making 14 (1994), 208–216.
16. Barnetson, R.A., Tenesa, A., Farrington, S.M., Nicholl, I.D., Cetnarskyj, R., Porteous, M.E., et al. Identification and survival of carriers of mutations in DNA mismatch-repair genes in colon cancer. N Engl J Med 354 (2006), 2751–2763.
17. Kastrinos, F., Ojha, R.P., Leenen, C., Alvero, C., Mercado, R.C., Balmana, J., et al. Comparison of prediction models for Lynch syndrome among individuals with colorectal cancer. J Natl Cancer Inst, 108(2), 2015, 18.
18. Giardiello, F.M., Allen, J.I., Axilbund, J.E., Boland, C.R., Burke, C.A., Burt, R.W., et al. Guidelines on genetic evaluation and management of Lynch syndrome: a consensus statement by the US Multi-Society Task Force on colorectal cancer. Gastroenterology 147 (2014), 502–526.
19. Syngal, S., Brand, R.E., Church, J.M., Giardiello, F.M., Hampel, H.L., Burt, R.W., ACG clinical guideline: genetic testing and management of hereditary gastrointestinal cancer syndromes. Am J Gastroenterol 110 (2015), 223–262.
20. Balmana, J., Stockwell, D.H., Steyerberg, E.W., Stoffel, E.M., Deffenbaugh, A.M., Reid, J.E., et al. Prediction of MLH1 and MSH2 mutations in Lynch syndrome. JAMA 296 (2006), 1469–1478.
21. Kastrinos, F., Steyerberg, E.W., Mercado, R., Balmana, J., Holter, S., Gallinger, S., et al. The PREMM(1,2,6) model predicts risk of MLH1, MSH2, and MSH6 germline mutations based on cancer history. Gastroenterology 140 (2011), 73–81.
22. Kastrinos, F., Uno, H., Ukaegbu, C., Alvero, C., McFarland, A., Yurgelun, M.B., et al. Development and validation of the PREMM5 model for comprehensive risk assessment of Lynch syndrome. J Clin Oncol 35 (2017), 2165–2172.
23. Barnetson RA, Appendix, Available at http://www.nejm.org/doi/suppl/10.1056/NEJMoa053493/suppl_file/nejm_barnetson_2751sa1.pdf. 2006. Accessed May 1, 2017.
24. Ioannidis, J.P., Why most published research findings are false. PLoS Med, 2, 2005, e124.
25. Sun, G.W., Shook, T.L., Kay, G.L., Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol 49 (1996), 907–916.
26. Ioannidis, J.P., Why most discovered true associations are inflated. Epidemiology 19 (2008), 640–648.
27. Chatfield, C., Model uncertainty, data mining and statistical inference. J R Stat Soc Ser A 158 (1995), 419–466.
28. Steyerberg, E.W., Eijkemans, M.J., Habbema, J.D., Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol 52 (1999), 935–942.
29. Steyerberg, E.W., Eijkemans, M.J., Harrell, F.E. Jr., Habbema, J.D., Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med 19 (2000), 1059–1079.
30. Royston, P., Altman, D.G., Sauerbrei, W., Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 25 (2006), 127–141.
31. Collins, G.S., Ogundimu, E.O., Cook, J.A., Manach, Y.L., Altman, D.G., Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model. Stat Med 35 (2016), 4124–4135.
32. Babyak, M.A., What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models. Psychosom Med 66:3 (2004), 411–421.
33. van der Ploeg, T., Austin, P.C., Steyerberg, E.W., Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Med Res Methodol, 14, 2014, 137.
34. Dekker, F.W., Ramspek, C.L., van Diepen, M., Con: most clinical risk scores are useless. Nephrol Dial Transplant 32 (2017), 752–755.
35. Steyerberg, E.W., Balmana, J., Stockwell, D.H., Syngal, S., Data reduction for prediction: robust coding of age and family history for the risk of having a genetic mutation. Stat Med 26 (2007), 5545–5556.
36. Harrell, F.E., Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. 2015, Springer, New York.
37. Sauerbrei, W., Royston, P., Binder, H., Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat Med 26 (2007), 5512–5528.
38. Steyerberg, E.W., Harrell, F.E. Jr., Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol 69 (2016), 245–247.
39. Harrell, F.E. Jr., Lee, K.L., Mark, D.B., Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 15 (1996), 361–387.
40. Steyerberg, E.W., Vickers, A.J., Cook, N.R., Gerds, T., Gonen, M., Obuchowski, N., et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21 (2010), 128–138.
41. Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M.J., Steyerberg, E.W., A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol 74 (2016), 167–176.
42. Steyerberg, E.W., Vergouwe, Y., Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J 35 (2014), 1925–1931.
43. Pavlou, M., Ambler, G., Seaman, S., De Iorio, M., Omar, R.Z., Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events. Stat Med 35 (2016), 1159–1177.
44. Rahman, M.S., Sultana, M., Performance of Firth-and logF-type penalized methods in risk prediction for small or sparse binary data. BMC Med Res Methodol, 17, 2017, 33.
45. Altman, D.G., Andersen, P.K., Bootstrap investigation of the stability of a Cox regression model. Stat Med 8 (1989), 771–783.
46. Derksen, S., Keselman, H.J., Backward, forward and stepwise automated subset selection algorithms: frequency of obtaining authentic and noise variables. Br J Math Stat Psychol 45 (1992), 265–282.
47. Austin, P.C., Tu, J.V., Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality. J Clin Epidemiol 57 (2004), 1138–1146.
48. Wasserstein, R.L., Lazar, N.A., The ASA's statement on p-values: context, process, and purpose. Am Stat 70 (2016), 129–133.
49. Greenland, S., Senn, S.J., Rothman, K.J., Carlin, J.B., Poole, C., Goodman, S.N., et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol 31:4 (2016), 337–350.
50. Stang, A., Deckert, M., Poole, C., Rothman, K.J., Statistical inference in abstracts of major medical and epidemiology journals 1975-2014: a systematic review. Eur J Epidemiol 32:1 (2017), 21–29.
51. Irwin, J.R., McClelland, G.H., Negative consequences of dichotomizing continuous predictor variables. J Marketing Res 40 (2003), 366–371.
52. Altman, D.G., Royston, P., The cost of dichotomising continuous variables. BMJ, 332, 2006, 1080.
53. Naggara, O., Raymond, J., Guilbert, F., Roy, D., Weill, A., Altman, D.G., Analysis by categorizing or dichotomizing continuous variables is inadvisable: an example from the natural history of unruptured aneurysms. AJNR Am J Neuroradiol 32 (2011), 437–440.
54. Dawson, N.V., Weiss, R., Dichotomizing continuous variables in statistical analysis. Med Decis Making 32 (2012), 225–226.
55. Wynants, L., Timmerman, D., Verbakel, J.Y., Testa, A., Savelli, L., Fischerova, D., et al. Clinical utility of risk models to refer patients with adnexal masses to specialized oncology care: multicenter external validation using decision curve analysis. Clin Cancer Res 23 (2017), 5082–5090.
56. Steyerberg, E.W., Eijkemans, M.J., Harrell, F.E. Jr., Habbema, J.D., Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets. Med Decis Making 21 (2001), 45–56.
57. Ye, J., On measuring and correcting the effects of data mining and model selection. J Am Stat Assoc 93 (1998), 120–131.
58. Hastie, T., Tibshirani, R., Friedman, J.H., The elements of statistical learning: data mining, inference, and prediction. 2001, Springer, New York.
59. Van Belle, V., Van Calster, B., Visualizing risk prediction models. PLoS One, 10(7), 2015, e0132614.
60. Lee, K.L., Woodlief, L.H., Topol, E.J., Weaver, W.D., Betriu, A., Col, J., et al. Predictors of 30-day mortality in the era of reperfusion for acute myocardial infarction. Results from an international trial of 41,021 patients. GUSTO-I Investigators. Circulation 91 (1995), 1659–1668.
61. Steyerberg, E.W., Bleeker, S.E., Moll, H.A., Grobbee, D.E., Moons, K.G., Internal and external validation of predictive models: a simulation study of bias and precision in small samples. J Clin Epidemiol 56 (2003), 441–447.
62. Copas, J.B., Regression, prediction and shrinkage. J R Stat Soc Ser B 45 (1983), 311–354.
63. Tibshirani, R., Regression and shrinkage via the Lasso. J R Stat Soc Ser B 58 (1996), 267–288.
64. Greenland, S., Bayesian perspectives for epidemiological research. II. Regression analysis. Int J Epidemiol 36 (2007), 195–202.
65. Moons, K.G., Donders, A.R., Steyerberg, E.W., Harrell, F.E., Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example. J Clin Epidemiol 57 (2004), 1262–1270.
66. Riley, R.D., Ensor, J., Snell, K.I., Debray, T.P., Altman, D.G., Moons, K.G., et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ, 353, 2016, i3140.
67. Debray, T.P., Vergouwe, Y., Koffijberg, H., Nieboer, D., Steyerberg, E.W., Moons, K.G., A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 68 (2015), 279–289.
68. Debray, T.P., Moons, K.G., Ahmed, I., Koffijberg, H., Riley, R.D., A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis. Stat Med 32 (2013), 3158–3180.
69. Damen, J.A., Hooft, L., Schuit, E., Debray, T.P., Collins, G.S., Tzoulaki, I., et al. Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ, 353, 2016, i2416.
70. Reilly, B.M., Evans, A.T., Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 144 (2006), 201–209.
71. Ioannidis, J.P., How to make more published research true. PLoS Med, 11, 2014, e1001747.
72. Knottnerus, J.A., Muris, J.W., Assessment of the accuracy of diagnostic tests: the cross-sectional study. J Clin Epidemiol 56 (2003), 1118–1128.
73. Van Calster, B., Vickers, A.J., Calibration of risk prediction models: impact on decision-analytic performance. Med Decis Making 35 (2015), 162–169.