External validity of predictive models: A comparison of logistic regression, classification trees, and neural networks

Journal of Clinical Epidemiology

Volumn 56, Issue 8, 2003, Pages 721-729

  Terrin, Norma ^a,c   Schmid, Christopher H ^a   Griffith, John L ^a   D'Agostino Sr , Ralph B ^b   Selker, Harry P ^a  

^a TUFTS UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b Boston University   (United States)

^c TUFTS MEDICAL CENTER   (United States)

Author keywords

Diagnostic model; Nonlinearity; Prognostic model; Reproducibility; Transportability; Validation

Indexed keywords

ARTICLE; ARTIFICIAL NEURAL NETWORK; CLASSIFICATION; COMPARATIVE STUDY; DATA ANALYSIS; LOGISTIC REGRESSION ANALYSIS; NONLINEAR REGRESSION ANALYSIS; PREDICTION; PRIORITY JOURNAL; RECEIVER OPERATING CHARACTERISTIC; SIMULATION; STATISTICAL MODEL; VALIDATION PROCESS;

EID: 0041382897     PISSN: 08954356     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0895-4356(03)00120-3     Document Type: Article

References (43)

1. Walter L.C., Brand R.J., Counsell S.R., et al. Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization. JAMA. 285:2001;2987-2994.
2. Selker H.P., Griffith J.L., D'Agostino R.B. A tool for judging coronary care unit admission appropriateness, valid for both real-time and retrospective use: a time-insensitive predictive instrument (TIPI) for acute cardiac ischemia: a multicenter study. Med Care. 29:1991;610-627.
3. Justice A.C., Covinsky K.E., Berlin J.A. Assessing the generalizability of prognostic information. Ann Intern Med. 130:1999;515-524.
4. Harrell F.E., Lee K.L., Mark D.B. Tutorial in biostatistics multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 15:1996;361-387.
5. Steyerberg E.W., Harrell F.E. Jr., Borsboom G.J.J.M., et al. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 54:2001;774-781.
6. Charlson M.E., Ales K.L., Simon R., et al. Why predictive indexes perform less well in validation studies. Arch Intern Med. 147:1987;2155-2161.
7. Allison J.J., Centor R.M. Why models predicting bacteremia in general medical patients do not work. J Gen Intern Med. 11:1996;132-134.
8. Dukes C.E., Bussey H.J. The spread of rectal cancer and its effect on prognosis. Br J Cancer. 12:1958;309-320.
9. Jass J.R., Love S.B., Northover J.M. A new prognostic classification of rectal cancer. Lancet. 1:1987;1303-1306.
10. Selker H.P., Beshansky J.R., Griffith J.L., et al. Use of the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) to assist with triage of patients with chest pain or other symptoms suggestive of acute cardiac ischemia: a multicenter, controlled clinical trial. Ann Intern Med. 129:1998;845-855.
11. Ottenbacher K.J., Smith P.M., Illig S.B., et al. Comparison of logistic regression and neural networks to predict rehospitalization in patients with stroke. J Clin Epidemiol. 54:2001;1159-1165.
12. Ennis M., Hinton G., Naylor D., et al. A comparison of statistical learning methods on the Gusto database. Stat Med. 17:1998;2501-2508.
13. Long W., Griffith J.L., Selker H.P., et al. A comparison of logistic regression to decision-tree induction in a medical domain. Comput Biomed Res. 26:1993;74-97.
14. Selker H.P., Griffith J., Patil S., et al. A comparison of performance of mathematical predictive methods for medical diagnosis: Identifying acute cardiac ischemia among emergency department patients. J Invest Med. 43:1995;468-476.
15. LaPuerta P., L'Italien G., Paul S., et al. Neural network assessment of preoperative cardiac risk in vascular surgery patients. Med Decis Making. 18:1998;70-75.
16. Silver D.L., Hurwitz G.A. The predictive and explanatory power of inductive decision trees: a comparison with artificial neural network learning as applied to the noninvasive diagnosis of coronary artery disease. J Invest Med. 45:1997;99-108.
17. Tu J., Weinstein M., McNeil B., et al. Predicting mortality after coronary artery bypass surgery: what do artificial neural networks learn? Med Decis Making. 18:1998;229-235.
18. Arana E., Delicado P., Marti-Bonmati L. Validation procedures in radiologic diagnostic models: neural network and logistic regression. Invest Radiol. 34:1999;636-642.
19. Mat-Sakim H.A., Naguib R.N., Lakshmi M.S., et al. Analysis of image cytometry data of fine needle aspirated cells of breast cancer patients: a comparison between logistic regression and artificial neural networks. Anticancer Res. 18:1998;2723-2726.
20. Duh M.S., Walker A.M., Pagano M., et al. Prediction and cross-validation of neural networks versus logistic regression: using hepatic disorders as an example. Am J Epidemiol. 147:1998;407-413.
21. Rae S.A., Wang W.J., Partridge D. Artificial neural networks: a potential role in osteoporosis. J R Soc Med. 92:1999;119-122.
22. D'Agostino R.B. Sr., Grundy S., Sullivan L.M., et al. CHD Risk Prediction Group. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA. 286:2001;180-187.
23. Goldman L., Cook E.F., Johnson P.A., et al. Prediction of the need for intensive care in patients who come to emergency departments with acute chest pain. New Engl J Med. 334:1996;1498-1504.
24. Wigton R.S., Conor J.L., Centor R.M. Transportability of a decision rule for the diagnosis of streptococcal pharyngitis. Arch Intern Med. 146:1986;81-83.
25. Detsky A.S., Abrams H.B., McLaughlin J.R., et al. Predicting cardiac complications in patients undergoing non-cardiac surgery. J Gen Intern Med. 1:1986;211-219.
26. Justice A.C., Aiken L.H., Smith H.L., et al. The role of functional status in predicting inpatient mortality with AIDS: a comparison with current predictors. J Clin Epidemiol. 49:1996;193-201.
27. Stone V.E., Seage G.R. III, Hertz T., et al. The relation between hospital experience and mortality for patients with AIDS. JAMA. 268:1992;2655-2661.
28. Mocroft A.J., Johnson M.A., Sabin C.A., et al. Staging system for clinical AIDS patients. Royal Free/Chelsea and Westminster Hospitals Collaborative Group. Lancet. 346:1995;12-17.
29. Cozzi Lepri A., Pezzotti P., Phillips A.N., et al. Clinical staging system for AIDS patients. [letter] Lancet. 346:1995;1103.
30. Deyo R.A., Cherkin D.C., Ciol M.A. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 45:1992;613-619.
31. Clark L.A., Pregibon D. Tree-based models. Chambers J.M., Hastie T.J. Statistical models in S. 1993;377-420 Chapman & Hall/CRC, Boca Raton.
32. Ripley B.D. Pattern recognition and neural networks. 1996;Cambridge University Press, Cambridge.
33. Finoff W., Hergert F., Zimmerman H.G. Improving model selection by nonconvergent methods. Neural Networks. 6:1993;771-783.
34. Schmid C.H., Terrin N., Griffith J.L., et al. Predictive performance of missing data methods for logistic regression, classification trees, and neural networks. J Stat Comput Sim. 71:2001;115-140.
35. Schmid C.H., Griffith J.L. Multivariate classification rules: calibration and discrimination. Encyclopedia of biostatistics. 1998;2844-2850 John Wiley and Sons, New York.
36. Hosmer D.W., Lemeshow S. Applied logistic regression. 1989;Wiley, New York.
37. Altman D.G., Royston P. What do we mean by validating a prognostic model? Stat Med. 19:2000;453-473.
38. Copas J.B. Binary regression models for contaminated data. J R Stat Soc Ser B. 50:1988;225-265.
39. Copas J.B. Plotting p against x. Appl Stat. 32:1983;25-31.
40. Spiegelhalter D.J. Probabilistic prediction in patient management and clinical trials. Stat Med. 5:1986;421-433.
41. Harrell F.E., Lee K.L. Regression modeling strategies for improved prognostic prediction. Stat Med. 3:1984;143-152.
42. Steyerberg E.W., Eijkemans M.J.C., Harrell F.E. Jr., et al. Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med. 19:2000;1059-1079.
43. Steyerberg E.W., Eijkemans M.J.C., Harrell F.E. Jr., et al. Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets. Med Decis Making. 21:2001;45-56.