Misuse of DeLong test to compare AUCs for nested models

Statistics in Medicine

Volumn 31, Issue 23, 2012, Pages 2577-2587

  Demler, Olga V ^a   Pencina, Michael J ^a   D'Agostino Sr , Ralph B ^b  

^a BOSTON UNIVERSITY SCHOOL OF PUBLIC HEALTH   (United States)

^b Boston University   (United States)

Author keywords

AUC; DeLong test; Discrimination; Logistic regression; Risk prediction; U statistics

Indexed keywords

ANALYTICAL ERROR; AREA UNDER THE CURVE; ARTICLE; CONFIDENCE INTERVAL; DELONG TEST; EFFECT SIZE; INTERMETHOD COMPARISON; MATHEMATICAL MODEL; MATHEMATICAL VARIABLE; NORMAL DISTRIBUTION; POWER ANALYSIS; RECEIVER OPERATING CHARACTERISTIC; STATISTICAL ANALYSIS; STATISTICAL SIGNIFICANCE;

AREA UNDER CURVE; COMPUTER SIMULATION; CORONARY DISEASE; DATA INTERPRETATION, STATISTICAL; HUMANS; MODELS, STATISTICAL; RISK ASSESSMENT; ROC CURVE;

EID: 84866436976     PISSN: 02776715     EISSN: 10970258     Source Type: Journal    
DOI: 10.1002/sim.5328     Document Type: Article

References (36)

1. Harrell FE, Jr. Regression Modeling Strategies: with Applications to Linear Models, Logistic Regression and Survival Analysis. Springer-Verlag: New York, 2001.
2. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating (Statistics for Biology and Health). Springer Science+Business Media: New York, 2009.
3. Pepe MS. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford University Press: New York, 2004.
4. Stone B. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society. Series B (Methodological) 1974; 36(2):111-147.
5. Ridker PM, Rifai N, Rose L, Buring JE, Cook NR. Comparison of C-reactive protein and low-density lipoprotein cholesterol levels in the prediction of first cardiovascular events. NEJM 2003; 348:1059-1061.
6. Polonsky TS, McClelland RL, Jorgensen NW, Bild DE, Burke GL, Guerci AD, Greenland P. Coronary artery calcium score and risk classification for coronary heart disease prediction. JAMA 2010; 303(16):1610-1616.
7. Meigs JB, Shrader P, Sullivan LM, McAteer JB, Fox CS, Dupuis J, Manning AK, Florez JC, Wilson PWF, D'Agostino RB Sr., Cupples A. Genotype score in addition to common risk factors for prediction of type II diabetes. New England Journal of Medicine 2008; 359(21):2208-2219.
8. D'Agostino RB, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, Kannel WB. General cardiovascular risk profile for use in primary care: the Framingham heart study. Circulation 2008; 117:743-753.
9. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing areas under two or more correlated reciever operating characteristics curves: a nonparamentric approach. Biometrics 1988; 44(3):837-845.
10. SAS/STAT software, Version 9.1(TS1M3) of the SAS System. Copyright © 2002-2003 SAS Institute Inc. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc., Cary, NC, USA.
11. Wang TJ, Gona P, Larson MG, Tofler GH, Levy D, Newton-Cheh C, Jacques PF, Rifai N, Selhub J, Robins SJ, Benjamin EJ, D'Agostino RB, Vasan RS. Multiple biomarkers for the prediction of first major cardiovascular events and death. The New England Journal of Medicine 2006; 355(25):2631-2639.
12. Ridker PM, Buring JE, Rifai N, Cook NR. Development and validation of improved algorithms for the assessment of global cardiovascular risk in women. JAMA 2007; 297(6):611-619.
13. Cook NR. Use and misuse of the receiver operating characteristics curve in risk prediction. Circulation 2007; 115:928-935.
14. Tzoulaki I, Liberopoulos G, Ioannisis JPA. Assessment off claims of improved prediction beyond the Framnigham risk score. JAMA 2009; 302:2345-2352.
15. Demler OV, Pencina MJ, D'Agostino R. Equivalence of improvement in area under ROC curve and linear discriminant analysis coefficient under assumption of normality. Statistics in Medicine 2011; 30(12):1410-1418.
16. Rao CR. Linear Statistical Inference and its Applications. Wiley: New York, 1973.
17. Su JQ, Liu JS. Linear combinations of multiple diagnostic markers. JASA 1993; 88(424):1350-1355.
18. Mardia KV, Kent JT, Bibby JM. Multivariate analysis. Academic Press: San Diego, 1979.
19. Executive Summary of the Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III). Journal of the American Medical Association 2001; 285:2486-2497.
20. Wilson P, D'Agostino R, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation 1998; 97:1837-1847.
21. Anderson KM, Odell PM, Wilson PWF, Kannel WB. Cardiovascular disease risk profiles. American Heart Journal 1991; 121:293-298.
22. Steyerberg EW, Harrell FE, Borsboom GJJM, Eijkemans MJC, Vergouwe Y, Habbema JDF. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. Journal of Clinical Epidemiology 2001; 54:774-781.
23. Efron B, Tibshirani R. An Introduction to Bootstrap. Chapman and Hall/CRC: Boca Raton, 1993.
24. Tibshirani R, Hall P, Wilson SR. Bootstrap Hypothesis Testing. Biometrics 1992; 48(3):969-970.
25. Hall P, Wilson SR. Two guidelines for bootstrap hypothesis testing. Biometrics 1991; 47:757-762.
26. Vickers AJ, Cronin AM, Begg CB. One statistical test is sufficient for assessing new predictive markers. BMC Medical Research Methodology 2011; 11(13):1-7.
27. Lee AJ. U-Statistics: Theory and Practice. Marcel Dekkel: New York and Basel, 1990.
28. Lehmann EL. Consistency and unbiasedness of certain nonparametric tests. The Annals of Mathematical Statistics 1951; 22(2):165-179.
29. Mises RV. On the asymptotic distribution of differentiable statistical functions. Annals of Mathematical Statistics 1947; 18:309-348.
30. Hoeffding W. A class of statistics with asymptotically normal distributions. Annals of Mathematical Statistics 1948; 19(3):293-325.
31. De Wet T, Randles RH. On the effect of substituting parameter estimators in limiting χ2 U and V statistics. The Annals of Statistics 1987; 15(1):398-441.
32. Iverson HK, Randles RH. The effects on convergence of substituting parameter estimates into U-statistics and other families of statistic. Probability Theory and Related Fields 1989; 81:453-471.
33. Randles RH. On the asymptotic normality of statistics with estimated parameters. The Annals of Statistics 1982; 10(2):462-474.
34. Antolini L, Namb B-H, D'Agostino RB. Inference on correlated discrimination measures in survival analysis: a nonparametric approach. Communications in Statistics Theory and Methods 2004; 33(9):2117-2135.
35. Lee M-LT, Rosner BA. The average area under correlated receiver operating characteristic curves: a nonparametric approach based on generalized two-sample Wilcoxon statistics. Applied Statistics 2001; 50(3):337-344.
36. Obuchowski NA, Lieber ML. Confidence intervals for the receiver operating characteristic area in studies with small samples. Academic Radiology 1998; 5:561-571.