Modern modeling techniques had limited external validity in predicting mortality from traumatic brain injury

Journal of Clinical Epidemiology

Volumn 78, Issue , 2016, Pages 83-89

  van der Ploeg, Tjeerd ^a,b,c   Nieboer, Daan ^c   Steyerberg, Ewout W ^c  

^a Medical Centre Alkmaar   (Netherlands)

^b INHOLLAND UNIVERSITY OF APPLIED SCIENCES   (Netherlands)

^c ERASMUS MEDICAL CENTER   (Netherlands)

Author keywords

Calibration; Discrimination; External validation; Internal validation; Modeling techniques; Prediction models

Indexed keywords

GLUCOSE; HEMOGLOBIN;

ARTICLE; ARTIFICIAL NEURAL NETWORK; BRAIN RADIOGRAPHY; CLASSIFICATION; COMPUTER ASSISTED TOMOGRAPHY; CONTROLLED STUDY; DIAGNOSTIC TEST ACCURACY STUDY; DISCRIMINANT VALIDITY; EXTERNAL VALIDITY; GLUCOSE BLOOD LEVEL; HEMOGLOBIN BLOOD LEVEL; HUMAN; LABORATORY TEST; LOGISTIC REGRESSION ANALYSIS; MAJOR CLINICAL STUDY; MORTALITY; MOTOR PERFORMANCE; PREDICTIVE VALUE; PRIORITY JOURNAL; PUPIL REFLEX; RANDOM FOREST; RECEIVER OPERATING CHARACTERISTIC; REGRESSION ANALYSIS; SCORING SYSTEM; SENSITIVITY AND SPECIFICITY; STATISTICAL MODEL; SUPPORT VECTOR MACHINE; TRAUMATIC BRAIN INJURY; REPRODUCIBILITY; THEORETICAL MODEL;

BRAIN INJURIES, TRAUMATIC; HUMANS; MODELS, THEORETICAL; REPRODUCIBILITY OF RESULTS; ROC CURVE; SUPPORT VECTOR MACHINE;

EID: 84963545841     PISSN: 08954356     EISSN: 18785921     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2016.03.002     Document Type: Article

References (38)

1. [1] Hastie, T., Tibshirani, R., Friedman, J., Hastie, T., Friedman, J., Tibshirani, R., The elements of statistical learning, Volume 2, 2009, Springer, New York.
2. [2] Harrell, F.E., Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. 2001, Springer, New York.
3. [3] Steyerberg, E.W., Clinical prediction models: a practical approach to development, validation, and updating, Volume 19, 2009, Springer, New York.
4. [4] Young, N.H., Andrews, P.J.D., Developing a prognostic model for traumatic brain injury—a missed opportunity?. PLoS Med, 5, 2008, e168.
5. [5] Colombet, I., Ruelland, A., Chatellier, G., Gueyffier, F., Degoulet, P., Jaulent, M.C., Models to predict cardiovascular risk: comparison of CART, multilayer perceptron and logistic regression. Proc AMIA Symp, 2000, 156–160.
6. [6] Maroco, J., Silva, D., Rodrigues, A., Guerreiro, M., Santana, I., de Mendonça, A., Data mining methods in the prediction of dementia: a real-data comparison of the accuracy, sensitivity and specificity of linear discriminant analysis, logistic regression, neural networks, support vector machines, classification trees and random forests. BMC Res Notes, 4, 2011, 299.
7. [7] Terrin, N., Schmid, C.H., Griffith, J.L., D'Agostino, R.B., Selker, H.P., External validity of predictive models: a comparison of logistic regression, classification trees, and neural networks. J Clin Epidemiol 56 (2003), 721–729.
8. [8] Ecke, T.H., Hallmann, S., Koch, S., Ruttloff, J., Cammann, H., Gerullis, H., et al. External validation of an artificial neural network and two nomograms for prostate cancer detection. ISRN Urol, 2012, 2012, 643181.
9. [9] König, I.R., Malley, J.D., Weimar, C., Diener, H.-C., Ziegler, A., Practical experiences on the necessity of external validation. Stat Med 26 (2007), 5499–5511.
10. [10] Steyerberg, E.W., Vickers, A.J., Cook, N.R., Gerds, T., Gonen, M., Obuchowski, N., et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21 (2010), 128–138.
11. [11] Austin, P.C., Steyerberg, E.W., Predictive accuracy of risk factors and markers: a simulation study of the effect of novel markers on different performance measures for logistic regression models. Stat Med 32 (2013), 661–672.
12. [12] Vergouwe, Y., Moons, K.G.M., Steyerberg, E.W., External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 172 (2010), 971–980.
13. [13] Lingsma, H.F., Roozenbeek, B., Steyerberg, E.W., Murray, G.D., Maas, A.I., Early prognosis in traumatic brain injury: from prophecies to predictions. Lancet Neurol 9 (2010), 543–554.
14. [14] Steyerberg, E.W., Mushkudiani, N., Perel, P., Butcher, I., Lu, J., McHugh, G.S., et al. Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med 5 (2008), 1251–1261.
15. [15] Jennett, B., Teasdale, G., Braakman, R., Minderhoud, J., Knill-Jones, R., Predicting outcome in individual patients after severe head injury. Lancet 307 (1976), 1031–1034.
16. [16] Mushkudiani, N.A., Hukkelhoven, C.W., Hernández, A.V., Murray, G.D., Choi, S.C., Maas, A.I., et al. A systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomes. J Clin Epidemiol 61 (2008), 331–343.
17. [17] Cortes, C., Vapnik, V., Support-vector networks. Mach Learn 20 (1995), 273–297.
18. [18] Marmarou, A., Lu, J., Butcher, I., McHugh, G.S., Mushkudiani, N.A., Murray, G.D., et al. IMPACT database of traumatic brain injury: design and description. J Neurotrauma 24 (2007), 239–250.
19. [19] Roozenbeek, B., Chiu, Y.-L., Lingsma, H.F., Gerber, L.M., Steyerberg, E.W., Ghajar, J., et al. Predicting 14-day mortality after severe traumatic brain injury: application of the IMPACT models in the brain trauma foundation TBI-trac New York State database. J Neurotrauma 29 (2012), 1306–1312.
20. [20] Tufféry, S., Data mining and statistics for decision making. 2011, John Wiley & Sons, New York.
21. [21] Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J., Classification and regression trees, Volume 19, 1984, Wadsworth, Monterey.
22. [22] R Development Core Team R. R: a language and environment for statistical computing. 2011, R Found Stat Comput, Vienna, Austria, 409.
23. [23] Maas, A.I., Marmarou, A., Murray, G.D., Teasdale, S.G., Steyerberg, E.W., Prognosis and clinical trial design in traumatic brain injury: the IMPACT study. J Neurotrauma 24 (2007), 232–238.
24. [24] Hanley, J.A., McNeil, B.J., The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143 (1982), 29–36.
25. [25] Cox, D.R., Two further applications of a model for binary regression. Biometrika 45 (1958), 562–565.
26. [26] van Houwelingen, J.C., Le Cessie, S., Predictive value of statistical models. Stat Med 9 (1990), 1303–1325.
27. [27] Collins, G.S., de Groot, J.A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol, 14, 2014, 40.
28. [28] Siontis, G.C.M., Tzoulaki, I., Castaldi, P.J., Ioannidis, J.P.A., External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 68 (2015), 25–34.
29. [29] Steyerberg, E.W., Harrell, F.E., Borsboom, G.J., Eijkemans, M.J., Vergouwe, Y., Habbema, J.D., Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 54 (2001), 774–781.
30. [30] Bleeker, S.E., Moll, H.A., Steyerberg, E.W., Donders, A.R.T., Derksen-Lubsen, G., Grobbee, D.E., et al. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol 56 (2003), 826–832.
31. [31] Justice, A.C., Covinsky, K.E., Berlin, J.A., Assessing the generalizability of prognostic information. Ann Intern Med 130 (1999), 515–524.
32. [32] Lang, E.W., Pitts, L.H., Damron, S.L., Rutledge, R., Outcome after severe head injury: an analysis of prediction based upon comparison of neural network versus logistic regression analysis. Neurol Res 19 (1997), 274–280.
33. [33] Eftekhar, B., Mohammad, K., Ardebili, H.E., Ghodsi, M., Ketabchi, E., Comparison of artificial neural network and logistic regression models for prediction of mortality in head trauma based on initial clinical data. BMC Med Inform Decis Mak, 5, 2005, 3.
34. [34] van der Ploeg, T., Datema, F., de Jong, R.B., Steyerberg, E.W., Prediction of survival with alternative modeling techniques using pseudo values. PLoS One, 9, 2014, e100234.
35. [35] van der Ploeg, T., Austin, P.C., Steyerberg, E.W., Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Med Res Methodol, 14, 2014, 137.
36. [36] Ennis, M., Hinton, G., Naylor, D., Revow, M., Tibshirani, R., A comparison of statistical learning methods on the Gusto database. Stat Med 17 (1998), 2501–2508.
37. [37] Hukkelhoven, C., Steyerberg, E., Habbema, J., Farace, E., Marmarou, A., Murray, G., et al. Predicting outcome after traumatic brain injury: admission characteristics. J Neurotrauma 22 (2005), 1025–1039.
38. [38] Steyerberg, E.W., Eijkemans, M.J., Harrell, F.E., Habbema, J.D., Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med 19 (2000), 1059–1079.