Internal and external validation of predictive models: A simulation study of bias and precision in small samples

Journal of Clinical Epidemiology

Volumn 56, Issue 5, 2003, Pages 441-447

  Steyerberg, Ewout W ^a   Bleeker, Sacha E ^b   Moll, Henriëtte A ^b   Grobbee, Diederick E ^c   Moons, Karel G M ^c  

^a ERASMUS MEDICAL CENTER   (Netherlands)

^b SOPHIA CHILDREN S HOSPITAL   (Netherlands)

^c UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

Author keywords

Bootstrap; External validation; Internal validation; Logistic Regression; Prediction models

Indexed keywords

ACCURACY; ANALYTICAL ERROR; ARTICLE; BACTERIAL INFECTION; CLINICAL RESEARCH; CONTROLLED STUDY; HUMAN; MAJOR CLINICAL STUDY; PREDICTION; PRIORITY JOURNAL; PYREXIA IDIOPATHICA; SAMPLING; SIMULATION; STATISTICAL ANALYSIS; STATISTICAL MODEL; VALIDATION PROCESS;

EID: 0038243037     PISSN: 08954356     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0895-4356(03)00047-7     Document Type: Article

References (25)

1. Harrell F.E. Jr., Lee K.L., Califf R.M., et al. Regression modelling strategies for improved prognostic prediction. Stat Med. 3:1984;143-152.
2. Van Houwelingen J.C., Le Cessie S. Predictive value of statistical models. Stat Med. 9:1990;1303-1325.
3. Harrell F.E. Jr., Lee K.L., Mark D.B. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 15:1996;361-387.
4. Efron B., Tibshirani R. An introduction to the bootstrap. Monographs on statistics and applied probability. 1993;xvi Chapman and Hall, New York. 436.
5. Steyerberg E.W., Harrell F.E. Jr., Borsboom G.J., et al. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 54:2001;774-781.
6. Altman D.G., Andersen P.K. Bootstrap investigation of the stability of a Cox regression model. Stat Med. 8:1989;771-783.
7. Chatfield C. Model uncertainty, data mining and statistical inference. J Royal Stat Soc A. 158:1995;419-466.
8. Altman D.G., Royston P. What do we mean by validating a prognostic model? Stat Med. 19:2000;453-473.
9. Justice A.C., Covinsky K.E., Berlin J.A. Assessing the generalizability of prognostic information. Ann Intern Med. 130:1999;515-524.
10. Bleeker SE, Moll HA, Steyerberg EW, et al. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol, in press.
11. Stone M. Cross-validatory choice and assessment of statistical predictions. J Royal Stat Soc B. 36:1974;111-147.
12. Nagelkerke N. A note on a general definition of the coefficient of determination. Biometrika. 78:1991;691-692.
13. Miller M.E., Hui S.L., Tierney W.M. Validation techniques for logistic regression models. Stat Med. 10:1991;1213-1226.
14. Copas J.B. Regression, prediction and shrinkage. J Royal Stat Soc B. 45:1983;311-354.
15. Steyerberg E.W., Eijkemans M.J., Harrell F.E. Jr., et al. Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med. 19:2000;1059-1079.
16. Efron B., Tibshirani R. Improvements on cross-validation: the .632+ bootstrap method. J Am Stat Assoc. 92:1997;548-560.
17. del Barrio E., Cuesta-Albertos J.A., Matran C. Asymptotic stability of the bootstrap sampled mean. Stoch Proc Appl. 97:2002;289-306.
18. Buckland S.T., Burnham K.P., Augustin N.H. Model selection: an integral part of inference. Biometrics. 53:1997;603-618.
19. Steyerberg E.W., Eijkemans M.J., Habbema J.D. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol. 52:1999;935-942.
20. Ye J. On measuring and correcting the effects of data mining and model selection. J Am Stat Assoc. 93:1998;120-131.
21. Laupacis A., Sekar N., Stiell I.G. Clinical prediction rules: a review and suggested modifications of methodological standards. JAMA. 277:1997;488-494.
22. Steyerberg E.W., Eijkemans M.J., Van Houwelingen J.C., et al. Prognostic models based on literature and individual patient data in logistic regression analysis. Stat Med. 19:2000;141-160.
23. Spiegelhalter D.J. Probabilistic prediction in patient management and clinical trials. Stat Med. 5:1986;421-433.
24. Campbell G. Advances in statistical methodology for the evaluation of diagnostic and laboratory tests. Stat Med. 13:1994;499-508.
25. Van Houwelingen H.C. Validation, calibration, revision and combination of prognostic survival models. Stat Med. 19:2000;3401-3415.