Performance of reclassification statistics in comparing risk prediction models

Biometrical Journal

Volumn 53, Issue 2, 2011, Pages 237-258

  Cook, Nancy R ^a   Paynter, Nina P ^a  

^a BRIGHAM AND WOMEN S HOSPITAL   (United States)

Author keywords

Calibration; Discrimination; Model accuracy; Prediction; Reclassification

Indexed keywords

CALIBRATION; ERROR STATISTICS; RISK ASSESSMENT;

CLINICAL USE; CLINICAL UTILITY; DISCRIMINATION; MODEL FIT; MODELING ACCURACY; PERFORMANCE; POWER; RECLASSIFICATION; RISK PREDICTION MODELS; TYPE-I ERROR;

FORECASTING;

EID: 79952228087     PISSN: 03233847     EISSN: 15214036     Source Type: Journal    
DOI: 10.1002/bimj.201000078     Document Type: Article

References (41)

1. Adult Treatment Panel III. (2001). Executive summary of the third report of the National Cholesterol Education Program (NCEP) expert panel on detection, evaluation, and treatment of high blood cholesterol in adults (Adult Treatment Panel III). Journal of the American Medical Association 285, 2486-2497.
2. 2: a useful measure of model performance when predicting a dichotomous outcome. Statistics in Medicine 18, 375-384.
3. Baker, S. G., Cook, N. R., Vickers, A. and Kramer, B. S. (2009). Using relative utility curves to evaluate risk prediction. Journal of the Royal Statistics Society, Series A 172, 729-748.
4. Cook, N. R. (2007). Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 115, 928-935.
5. Cook, N. R. (2008). Comments on 'Evaluating the added predictive ability of a new biomarker: from area under the ROC curve to reclassification and beyond'. Statistics in Medicine 27, 191-195.
6. Cook, N. R. (2010). Comment: measures to summarize and compare the predictive capacity of markers. International Journal of Biostatistics 6, Article 22.
7. Cook, N. R., Buring, J. E. and Ridker, P. M. (2006). The effect of including C-reactive protein in cardiovascular risk prediction models for women. Annals of Internal Medicine 145, 21-29.
8. Cook, N. R. and Ridker, P. M. (2009). Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Annals of Internal Medicine 150, 795-802.
9. D'Agostino, R. B., Griffith, J. L., Schmidt, C. H. and Terrin, N. (1997). Measures for evaluating model performance. In: Proceedings of the Biometrics Section, American Statistical Association, Biometrics Section, Alexandria, VA, 253-258.
10. DeLong, E. R., DeLong, D. M. and Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44, 837-845.
11. Diamond, G. A. (1992). What price perfection? Calibration and discrimination of clinical prediction models. Journal of Clinical Epidemiology 45, 85-89.
12. Gail, M. H. and Pfeiffer, R. M. (2005). On criteria for evaluating models of absolute risk. Biostatistics 6, 227-239.
13. Gu, W. and Pepe, M. (2009). Measures to summarize and compare the predictive capacity of markers. The International Journal of Biostatistics 5, Article 27.
14. Hanley, J. A. and McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29-36.
15. Harrell, F. E., Jr. (2001). Regression Modeling Strategies. Springer, New York.
16. Hlatky, M. A., Greenland, P., Arnett, D. K., Ballantyne, C. M., Criqui, M. H., Elkind, M. S. V., Go, A. S., Harrell, F. E., Jr., Hong, Y., Howard, B. V., Howard, V. J., Hsue, P. Y., Kramer, C. M., McConnell, J. P., Normand, S. L. T., O'Donnell, C. J., Smith, S. C., Jr., Wilson, P. W. F., on behalf of the American Heart Association Expert Panel on Subclinical Atherosclerotic Diseases and Emerging Risk Factors and the Stroke Council. (2009). Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation 119, 2408-2416.
17. Hosmer, D. W. and Hjort, N. L. (2002). Goodness-of-fit processes for logistic regression: simulation results. Statistics in Medicine 21, 2723-2738.
18. Hosmer, D. W., Hosmer, T., Le Cessie, S. and Lemeshow, S. (1997). A comparison of goodness-of-fit tests for the logistic regression model. Statistics in Medicine 16, 965-980.
19. Hosmer, D. W. and Lemeshow, S. (1980). A goodness-of-fit test for the multiple logistic regression model. Communications in Statistics: Theory and Methods A10, 1043-1069.
20. Hosmer, D. W. and Lemeshow, S. (2000). Applied Logistic Regression. Wiley, New York.
21. 2 statistics for logistic regression. Statistics in Medicine 25, 1383-1395.
22. Huang, Y. and Pepe, M. S. (2009). A parametric ROC model-based approach for evaluating the predictiveness of continuous markers in case-control studies. Biometrics 65, 1133-1144.
23. Janes, H., Pepe, M. S. and Gu, W. (2008). Assessing the value of risk predictions by using risk stratification tables. Annals of Internal Medicine 149, 751-760.
24. Le Cessie, S. and van Houwelingen, J. C. (1991). A goodness-of-fit test for binary regression based on smoothing methods. Biometrics 47, 1267-1282.
25. McClish, D. K. (1989). Analyzing a portion of the ROC curve. Medical Decision Making 9, 190-195.
26. Mealiffe, M. E., Stokowski, R. P., Rhees, B. K., Prentice, R. L., Pettinger, M. and Hinds, D. A. (2010). Assessment of clinical validity of a breast cancer risk model combining genetic and clinical information. Journal of the National Cancer Institute 102, 1618-1627.
27. Mihaescu, R., van Zitteren, M., van Hoek, M., Sijbrands, E. J. G., Uitterlinden, A. G., Witteman, J. C. M., Hofman, A., Hunink, M. G., van Duijn, C. M. and Janssens, A. C. J. W. (2010). Improvement of risk prediction by genomic profiling: reclassification measures versus the area under the receiver operating characteristic curve. American Journal of Epidemiology 172, 353-361.
28. Moons, K. G. M. and Harrell, F. E. (2003). Sensitivity and specificity should be de-emphasized in diagnostic accuracy studies. Academic Radiology 10, 670-672.
29. Pencina, M. J., D'Agostino, R. B. S., D'Agostino, R. B. J. and Vasan, R. S. (2008). Evaluating the added predictive ability of a new biomarker: from area under the ROC curve to reclassification and beyond. Statistics in Medicine 27, 157-172.
30. Pencina, M. J., D'Agostino, R. B., Sr., D'Agostino, R. B., Jr. and Vasan, R. S. (2009). Response to 'Net reclassification improvement and decision theory' by Vickers et al. Statistics in Medicine 28, 526-528.
31. Pepe, M. S., Feng, Z. and Gu, J. W. (2008a). Comments on 'Evaluating the added predictive ability of a new biomarker: from area under the ROC curve to reclassification and beyond'. Statistics in Medicine 27, 173-181.
32. Pepe, M. S., Feng, Z., Huang, Y., Longton, G., Prentice, R., Thompson, I. M. and Zheng, Y. (2008b). Integrating the predictiveness of a marker with its performance as a classifier. American Journal of Epidemiology 167, 362-368.
33. Pepe, M. S., Janes, H., Longton, G., Leisenring, W. and Newcomb, P. (2004). Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. American Journal of Epidemiology 159, 882-890.
34. Pigeon, J. G. and Heyse, J. F. (1999). An improved goodness of fit statistic for probability prediction models. Biometrical Journal 41, 71-82.
35. Ridker, P. M., Cook, N. R., Lee, I. M., Gordon, D., Gaziano, M. J., Manson, J. E., Hennekens, C. H. and Buring, J. E. (2005). A randomized trial of low-dose aspirin in the primary prevention of cardiovascular disease in women. New England Journal of Medicine 352, 1293-1304.
36. Rosner, B. and Glynn, R. J. (2009). Power and sample size estimation for the Wilcoxon rank sum test with application to comparisons of C statistics from alternative prediction models. Biometrics 65, 188-197.
37. Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J. and Kattan, M. W. (2009). Assessing the performance of prediction models: a framework for some traditional and novel measures. Epidemiology 21, 128-138.
38. Tjur, T. (2009). Coefficients of determination in logistic regression models - A new proposal: the coefficient of discrimination. American Statistician 63, 366-372.
39. Tsiatis, A. A. (1980). A note on the goodness-of-fit test for the logistic regression model. Biometrika 67, 250-251.
40. Vickers, A. J. and Elkin, E. B. (2006). Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making 26, 565-574.
41. Whittemore, A. S. (2010). Evaluating health risk models. Statistics in Medicine 29, 2438-2452.