Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models

Statistical Methods in Medical Research

Volumn 26, Issue 2, 2017, Pages 796-808

  Austin, Peter C ^a,b,c   Steyerberg, Ewout W ^d  

^a INSTITUTE FOR CLINICAL EVALUATIVE SCIENCES   (Canada)

^b UNIVERSITY OF TORONTO   (Canada)

^c Canada   (Canada)

^d ERASMUS MEDICAL CENTER   (Netherlands)

Author keywords

bootstrap; c statistic; clinical prediction models; data splitting; discrimination; logistic regression; model validation; receiver operating characteristic curve

Indexed keywords

AGED; ARTICLE; BOOTSTRAPPING; C-STATISTIC; CONGESTIVE HEART FAILURE; CONTROLLED STUDY; EVENTS PER VARIABLE; HUMAN; LOGISTIC REGRESSION ANALYSIS; MAJOR CLINICAL STUDY; MORTALITY; MORTALITY RATE; PREDICTION; STATISTICAL ANALYSIS; STATISTICAL PARAMETERS; VALIDITY; BIOSTATISTICS; CARDIOVASCULAR DISEASES; PROCEDURES; PROGNOSIS; REPRODUCIBILITY; RISK FACTOR; SAMPLE SIZE; STATISTICAL MODEL;

BIOSTATISTICS; CARDIOVASCULAR DISEASES; DATA INTERPRETATION, STATISTICAL; HUMANS; LOGISTIC MODELS; PROGNOSIS; REPRODUCIBILITY OF RESULTS; RISK FACTORS; SAMPLE SIZE;

EID: 85018749650     PISSN: 09622802     EISSN: 14770334     Source Type: Journal    
DOI: 10.1177/0962280214558972     Document Type: Article

References (24)

1. Justice AC, Covinsky KE, Berlin JA,. Assessing the generalizability of prognostic information. Ann Intern Med 1999; 130: 515-524.
2. Snee RD,. Validation of regression models: methods and examples. Technometrics 1977; 19: 415-428.
3. Hastie T, Tibshirani R, Friedman J,. The elements of statistical learning. Data mining, Inference, and prediction, New York, NY: Springer-Verlag, 2001.
4. Picard RR, Berk KN,. Data splitting. Am Statist 1990; 44: 140-147.
5. Harrell FE Jr,. Regression modeling strategies, New York, NY: Springer-Verlag, 2001.
6. Steyerberg EW,. Clinical prediction models, New York: Springer-Verlag, 2009.
7. Steyerberg EW, Harrell FE Jr, Borsboom GJ, et al. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 2001; 54: 774-781.
8. Peduzzi P, Concato J, Kemper E, et al. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996; 49: 1373-1379.
9. Lachenbruch PA, Goldstein M,. Discriminant analysis. Biometrics 1979; 35: 69-84.
10. Efron B, Tibshirani RJ,. An introduction to the bootstrap, New York, NY: Chapman & Hall, 1993.
11. Tu JV, Donovan LR, Lee DS, et al. Effectiveness of public report cards for improving the quality of cardiac care: the EFFECT study: a randomized trial. J Am Med Assoc 2009; 302: 2330-2337.
12. Lee DS, Austin PC, Rouleau JL, et al. Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model. J Am Med Assoc 2003; 290: 2581-2587.
13. Austin PC,. A comparison of regression trees, logistic regression, generalized additive models, and multivariate adaptive regression splines for predicting AMI mortality. Stat Med 2007; 26: 2937-2957.
14. Austin PC, Lee DS, Steyerberg EW, et al. Regression trees for predicting mortality in patients with cardiovascular disease: what improvement is achieved by using ensemble-based methods? Biometric J 2012; 54: 657-673.
15. Berk KN,. Validating regression procedures with new data. Technometrics 1984; 26: 331-338.
16. Airola A, Pahikkala T, Waegeman W, et al. An experimental comparison of cross-validation techniques for estimating the area under the ROC curve. Comput Stat Data Anal 2011; 55: 1828-1844.
17. Lachenbruch PA, Mickey MR,. Estimation of error rates in discriminant analysis. Technometrics 1968; 10: 1-11.
18. Smith GC, Seaman SR, Wood AM, et al. Correcting for optimistic prediction in small data sets. Am J Epidemiol 2014; 180: 318-324.
19. Breiman L, Friedman JH, Olshen RA, et al. Classification and regression trees, Boca Raton: Chapman & Hall/CRC, 1998.
20. Breiman L,. Random forests. Mach Learn 2001; 45: 5-32.
21. Buhlmann P, Hathorn T,. Boosting algorithms: regularization, prediction and model fitting. Stat Sci 2007; 22: 477-505.
22. Freund Y, Schapire R,. Experiments with a new boosting algorithm. In: Machine learning: proceedings of the thirteenth international conference, San Francisco, CA: Morgan Kauffman, 1996, pp. 148-156.
23. Austin PC, Steyerberg EW,. Interpreting the concordance statistic of a logistic regression model: relation to the variance and odds ratio of a continuous explanatory variable. BMC Med Res Methodol 2012; 12: 82-82.
24. Austin PC, Steyerberg EW,. Predictive accuracy of risk factors and markers: a simulation study of the effect of novel markers on different performance measures for logistic regression models. Stat Med 2013; 32: 661-672.