Novel metrics for evaluating improvement in discrimination: Net reclassification and integrated discrimination improvement for normal variables and nested models

Statistics in Medicine

Volumn 31, Issue 2, 2012, Pages 101-113

  Pencina, Michael J ^a,b   D'Agostino, Ralph B ^c   Demler, Olga V ^a  

^a BOSTON UNIVERSITY   (United States)

^b HARVARD CLINICAL RESEARCH INSTITUTE   (United States)

^c Boston University   (United States)

Author keywords

AUC; Biomarker; C statistic; Model performance; Risk prediction; ROC

Indexed keywords

ALGORITHM; AREA UNDER THE CURVE; ARTICLE; CARDIOVASCULAR DISEASE; CONTROLLED STUDY; DISCRIMINANT ANALYSIS; DISCRIMINATION; EFFECT SIZE; EVALUATION; GENETIC MARKER; HIGH RISK PATIENT; HUMAN; INTEGRATED DISCRIMINATION IMPROVEMENT; MATHEMATICAL PHENOMENA; MATHEMATICAL VARIABLE; MEASUREMENT; NET RECLASSIFICATION IMPROVEMENT; PHENOTYPE; PRIMARY PREVENTION; RISK ASSESSMENT; THEORETICAL SAMPLE;

CARDIOVASCULAR DISEASES; DISCRIMINANT ANALYSIS; EPIDEMIOLOGIC RESEARCH DESIGN; HUMANS; MALE; MODELS, STATISTICAL; RISK ASSESSMENT; ROC CURVE;

EID: 84855232027     PISSN: 02776715     EISSN: 10970258     Source Type: Journal    
DOI: 10.1002/sim.4348     Document Type: Article

References (37)

1. Fisher RA. The use of multiple measurements in taxonomic problems. Annals of Eugenics 1936; 7:179-188.
2. Walker SH, Duncan DB. Estimation of the probability of an event as a function of several independent variables. Biometrika 1967; 54:167-179.
3. Cox DR. Regression models and life tables. Journal of the Royal Statistical Society. Series B (Methodological) 1972; 34:187-220.
4. Boekhold SM, Kastelein JJM. C-reactive protein and cardiovascular risk: more fuel to the fire. Lancet 2010; 375:95-96.
5. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143:29-36.
6. Harrell FE, Lee KL, Mark DB. Tutorial in biostatistics: multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine 1996; 15:361-387.
7. Pencina MJ, D'Agostino RB. Overall c as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation. Statistics in Medicine 2004; 23:2109-2123.
8. Chambless LE, Diao G. Estimation of time-dependent area under the ROC curve for long-term risk prediction. Statistics in Medicine 2006; 25:3474-3486.
9. Chambless LE, Cummiskey CP, Cui G. Several methods to assess improvement in risk prediction models: extension to survival analysis. Statistics in Medicine 2011; 30:22-38.
10. Heagerty PJ, Zheng Y. Survival model predictive accuracy and ROC curves. Biometrics 2005; 61:92-105.
11. Uno H, Cai T, Pencina MJ, D'Agostino RB, Wei LJ. On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statistics in Medicine 2011; 30:1105-1117.
12. Hand DJ. Evaluating diagnostic tests: the area under the ROC curve and the balance of errors. Statistics in Medicine 2010; 29:1502-1510.
13. Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. American Journal of Epidemiology 2004; 159:882-890.
14. Ware JH. The limitations of risk factors as prognostic tools. The New England Journal of Medicine 2006; 355:2615-2617.
15. Cook NR. Use and misuse of the receiver operating characteristics curve in risk prediction. Circulation 2007; 115:928-935.
16. Pencina MJ, D'Agostino RB Sr., D'Agostino RB Jr., Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Statistics in Medicine 2008; 27:157-172.
17. Tzoulaki I, Liberopoulos G, Ioannidis JPA. Assessment of claims of improved prediction beyond the Framingham risk score. Journal of the American Medical Association 2009; 302:2345-2352.
18. Janes H, Pepe M, Gu W. Assessing the value of risk predictions by using risk stratification tables. Annals of Internal Medicine 2008; 149:751-760.
19. Tjur T. Coefficients of determination in logistic regression models-a new proposal: the coefficient of discrimination. American Statistician 2009; 63:366-372.
20. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making 2006; 26:565-574.
21. Baker SG, Cook NR, Vickers A, Kramer BS. Using relative utility curves to evaluate risk prediction. Journal of the Royal Statistical Society. Series A, (Statistics in Society) 2009; 172:729-748.
22. Gail MH. Value of adding single-nucleotide polymorphism genotypes to a breast cancer risk model. Journal of the National Cancer Institute 2009; 101:959-963.
23. Yates JF. External correspondence: decomposition of the mean probability score. Organizational Behavior and Human Performance 1982; 30:132-156.
24. D'Agostino RB, Griffith JL, Schmidt CH, Terrin N. Measures for evaluating model performance. Proceedings of the Biometrics Section, 1997. American Statistical Association, Biometrics Section: Alexandria VA; 253-258.
25. 2 statistics for logistic regression. Statistics in Medicine 2006; 25:1383-1395.
26. Pencina MJ, D'Agostino RB Sr., Steyerberg E. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Statistics in Medicine 2011; 30:11-21.
27. Mahalanobis PC. On the generalized distance in statistics. Proceedings of the National Institute of Sciences of India 1936; 2:49-55.
28. Su JQ, Liu JS. Linear combinations of multiple diagnostic markers. Journal of the American Statistical Association 1993; 88:1350-1355.
29. Demler OV, Pencina MJ, D'Agostino RB Sr. Equivalence of improvement in area under ROC curve and linear discriminant analysis coefficient under assumption of normality. Statistics in Medicine 2011; 30:1410-1418.
30. Pepe MS, Feng Z, Gu JW. Commentary on 'Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond'. Statistics in Medicine 2008; 27:173-181.
31. Pencina MJ, D'Agostino RB Sr., D'Agostino RB Jr., Vasan RS. Comments on integrated discrimination and net reclassification improvements-practical advice. Statistics in Medicine 2008; 27:207-212.
32. Youden WJ. Index for rating diagnostic tests. Cancer 1950; 3:32-35.
33. D'Agostino RB, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, Kannel WB. General cardiovascular risk profile for use in primary care. Circulation 2008; 117:743-753.
34. Box BEP, Cox DR. An analysis of transformations. Journal of the Royal Statistical Society. Series B (Methodological) 1964; 26:211-252.
35. Morrison DF. Multivariate Statistical Methods, Second Edition. McGraw-Hill: New York, NY, 1976.
36. Cohen J. Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates: Hillsdale, NJ, 1988.
37. Mardia KV, Kent JT, Bibby JM. Multivariate Analysis. Academic Press: San Diego, CA, 1979.