External validation is necessary in prediction research: A clinical example

Journal of Clinical Epidemiology

Volumn 56, Issue 9, 2003, Pages 826-832

  Bleeker, S E ^a,b,c   Moll, H A ^a   Steyerberg, E W ^d   Donders, A R T ^b,e   Derksen Lubsen, G ^c   Grobbee, D E ^b   Moons, K G M ^b  

^a SOPHIA CHILDREN S HOSPITAL   (Netherlands)

^b UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

^c JULIANA CHILDREN S HOSPITAL   (Netherlands)

^d ERASMUS MEDICAL CENTER   (Netherlands)

^e UTRECHT UNIVERSITY   (Netherlands)

Author keywords

Bootstrap; External validation; Internal validation; Logistic Regression; Prediction models

Indexed keywords

ARTICLE; BACTERIAL INFECTION; CHILD; CLINICAL PRACTICE; CONFIDENCE INTERVAL; CONTROLLED STUDY; DATA ANALYSIS; EPIDEMIOLOGICAL DATA; HUMAN; INFANT; LOGISTIC REGRESSION ANALYSIS; MAJOR CLINICAL STUDY; MEDICAL RESEARCH; METHODOLOGY; POPULATION MODEL; POPULATION RESEARCH; PREDICTION; PRIORITY JOURNAL; PYREXIA IDIOPATHICA; SAMPLING; TECHNIQUE; THEORETICAL MODEL; VALIDATION PROCESS;

EID: 0141514712     PISSN: 08954356     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0895-4356(03)00207-5     Document Type: Article

References (43)

1. Harrell F.E., Lee K.L., Matchar D.B., Reichert T.A. Regression models for prognostic prediction: advantages, problems, and suggested solutions. Cancer Treat Rep. 69:1985;1071-1077.
2. Harrell F.E. Jr., Lee K.L., Mark D.B. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 15:1996;361-387.
3. Efron B. Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Stat Assoc. 78:1983;316-331.
4. van Houwelingen J.C., Le Cessie S. Predictive value of statistical models. Stat Med. 9:1990;1303-1325.
5. Justice A.C., Covinsky K.E., Berlin J.A. Assessing the generalizability of prognostic information. Ann Intern Med. 130:1999;515-524.
6. Wasson J.H., Sox H.C., Neff R.K., Goldman L. Clinical prediction rules. Applications and methodological standards. N Engl J Med. 313:1985;793-799.
7. Moons K.G., van Es G.A., Michel B.C., Buller H.R., Habbema J.D., Grobbee D.E. Redundancy of single diagnostic test evaluation. Epidemiology. 10:1999;276-281.
8. Laupacis A., Sekar N., Stiell I.G. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA. 277:1997;488-494.
9. McGinn T.G., Guyatt G.H., Wyer P.C., Naylor C.D., Stiell I.G., Richardson W.S. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA. 284:2000;79-84.
10. Steyerberg E.W., Eijkemans M.J., Habbema J.D. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol. 52:1999;935-942.
11. Steyerberg E.W., Eijkemans M.J., Harrell F.E. Jr., Habbema J.D. Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med. 19:2000;1059-1079.
12. Efron B., Tibshirani R. An introduction to the bootstrap. Monographs on statistics and applied probability. 1993;Chapman & Hall, New York.
13. Efron B., Tibshirani R. Improvements on cross-validation: the.632+ bootstrap method. J Am Stat Assoc. 92:1997;548-560.
14. Picard R.R., Berk K.N. Data splitting. Am Stat. 44:1990;140-147.
15. Sauerbrei W. The use of resampling methods to simplify regression models in medical statistics. Journal of the Royal Statistical Society Series C: Applied Statistics. 48:1999;313-329.
16. Steyerberg E.W., Harrell F.E., Borsboom G.J., Eijkemans M.J., Vergouwe Y., Habbema J.D. Internal validation of predictive models. Efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 54:2001;774-781.
17. Knottnerus J.A. Prediction rules: statistical reproducibility and clinical similarity. Med Decis Making. 12:1992;286-287.
18. McCarthy P.L. Fever. Pediatr Rev. 19:1998;401-408.
19. Baraff L.J., Lee S.I. Fever without source: management of children 3 to 36 months of age. Pediatr Infect Dis J. 11:1992;146-151.
20. Steensel-Moll H.A. van, Jongkind C.J., Aarsen R.S.R., Goede-Bolder A. de, Dekker A., Suijlekom-Smit L.W.A. van, et al. A problem-oriented patient classification system for general pediatrics II. Tijdschr Kindergeneesk. 64:1996;99-104.
21. Kuppermann N., Fleisher G.R., Jaffe D.M. Predictors of occult pneumococcal bacteremia in young febrile children. Ann Emerg Med. 31:1998;679-687.
22. Baker M.D., Bell L.M., Avner J.R. Outpatient management without antibiotics of fever in selected infants [discussion: N Engl J Med 1994;330:939-40]. N Engl J Med. 329:1993;1437-1441.
23. Baraff L.J., Oslund S.A., Schriger D.L., Stephen M.L. Probability of bacterial infections in febrile infants less than three months of age: a meta-analysis. Pediatr Infect Dis J. 11:1992;257-264.
24. Bleeker S.E., Moons K.G.M., Derksen-Lubsen G., Grobbee D.E., Moll H.A. Predicting serious bacterial infection in young children with fever without apparent source. Acta Paediatr. 90:2001;1226-1232.
25. Oostenbrink R., Moons K.G.M., Donders A.R.T., Grobbee D.E., Moll H.A. Prediction of bacterial meningitis in children with meningeal signs: reduction of lumbar punctures. Acta Paediatr. 90:2001;611-617.
26. Oostenbrink R., Moons K.G.M., Theunissen C.C., Derksen-Lubsen G., Grobbee D.E., Moll H.A. Signs of meningeal irritation at the emergency department: how often bacterial meningitis? Pediatr Emerg Care. 17:2001;161-164.
27. Weller S.C., Mann N.C. Assessing rater performance without a "gold standard" using consensus theory. Med Decis Making. 17:1997;71-79.
28. McCarthy P.L., Lembo R.M., Baron M.A., Fink H.D., Cicchetti D.V. Predictive value of abnormal physical examination findings in ill-appearing and well-appearing febrile children. Pediatrics. 76:1985;167-171.
29. McCarthy P.L., Lembo R.M., Fink H.D., Baron M.A., Cicchetti D.V. Observation, history, and physical examination in diagnosis of serious illnesses in febrile children less than or equal to 24 months. J Pediatr. 110:1987;26-30.
30. Kramer M.S., Lane D.A., Mills E.L. Should blood cultures be obtained in the evaluation of young febrile children without evident focus of bacterial infection? A decision analysis of diagnostic management strategies. Pediatrics. 84:1989;18-27.
31. Baraff L.J., Bass J.W., Fleisher G.R., Klein J.O., McCracken G.H., Powell K.R., Schriger D.L. Practice guideline for the management of infants and children 0 to 36 months of age with fever without source. Pediatrics. 92:1993;1-12.
32. Dagan R., Powell K., Hall C., Menegus M. Identification of infants unlikely to have serious bacterial infection although hospitalized for suspected sepsis. J Pediatr. 107:1985;855-860.
33. Isaacman D.J., Shults J., Gross T.K., Davis P.H., Harper M. Predictors of bacteremia in febrile children 3 to 36 months of age. Pediatrics. 106:2000;977-982.
34. Hewson P.H., Humphries S.M., Roberton D.M., McNamara J.M., Robinson M.J. Markers of serious illness in infants under 6 months old presenting to a children's hospital. Arch Dis Child. 65:1990;750-756.
35. Berger R.M., Berger M.Y., van Steensel-Moll H.A., Dzoljic-Danilovic G., Derksen-Lubsen G. A predictive model to estimate the risk of serious bacterial infections in febrile infants. Eur J Pediatr. 155:1996;468-473.
36. Teach S.J., Fleisher G.R. Duration of fever and its relationship to bacteremia in febrile outpatients three to 36 months old. The Occult Bacteremia Study Group. Pediatr Emerg Care. 13:1997;317-319.
37. Greenland S., Finkle W.D. A critical look at methods for handling missing covariates in epidemiologic regression analyses. Am J Epidemiol. 142:1995;1255-1264.
38. Little R.A. Regression with missing X's: a review. J Am Stat Assoc. 87:1992;1227-1237.
39. Nagelkerke N. A note on the general definition of the coefficient of determination. Biometrika. 78:1991;691-692.
40. van Houwelingen J.C. Shrinkage and penalized likelihood as methods to improve predictive accuracy. Stat Neerl. 55:2001;17-34.
41. Steyerberg E.W., Eijkemans M.J.C., Habbema J.D.F. Application of shrinkage techniques in logistic regression analysis: a case study. Stat Neerl. 55:2001;76-88.
42. Hilden J., Habbema J.D., Bjerregaard B. The measurement of performance in probabilistic diagnosis II. Trustworthiness of the exact values of the diagnostic probabilities . Methods Inf Med. 17:1978;227-237.
43. Shekelle P.G., Eccles M.P., Grimshaw J.M., Woolf S.H. When should clinical guidelines be updated? BMJ. 323:2001;155-157.