The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets

Statistics in Biosciences

Volumn 7, Issue 2, 2015, Pages 282-295

  Pepe, Margaret S ^a   Fan, Jing ^a   Feng, Ziding ^b   Gerds, Thomas ^c   Hilden, Jorgen ^c  

^a USA   (United States)

^b UNIVERSITY OF TEXAS MD ANDERSON CANCER CENTER   (United States)

^c UNIVERSITY OF COPENHAGEN   (Denmark)

Author keywords

Biomarkers; Classification; Diagnostic test; Receiver operating characteristic; Risk prediction

Indexed keywords

AREA UNDER THE CURVE; ARTICLE; BIOMEDICAL ENGINEERING; LOGISTIC REGRESSION ANALYSIS; MONTE CARLO METHOD; NET RECLASSIFICATION INDEX; PERFORMANCE; PREDICTION; PRIORITY JOURNAL; RECEIVER OPERATING CHARACTERISTIC; SCORING SYSTEM; SIMULATION;

EID: 85027944859     PISSN: 18671764     EISSN: 18671772     Source Type: Journal    
DOI: 10.1007/s12561-014-9118-0     Document Type: Article

References (24)

1. Baker SG, Cook NR, Vickers A, Kramer BS (2009) Using relative utility curves to evaluate risk prediction. J R Stat Soc Ser A Stat Soc 172(4):729–748
2. Baker SG, Van Calster B, Steyerberg EW (2012) Evaluating a new marker for risk prediction using the test tradeoff: an update. Int J Biostat 8(1):1–37
3. Gneiting T, Raftery AE (2007) Strictly proper scoring rules, prediction, and estimation. J Am Stat Assoc 102:359–378
4. Hastie T, Tibshirani R, Friedman JH (2001) The elements of statistical learning: data mining, inference, and prediction. Springer, New York
5. Hilden J (2014) Commentary: On NRI, IDI, and “good-looking” statistics with nothing underneath. Epidemiology 25(2):265–267
6. Hilden J, Gerds TA (2013) A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index. Stat Med. doi:10.1002/sim.5804
7. Kerr KF, McClelland RL, Brown ER, Lumley T (2011) Evaluating the incremental value of new biomarkers with integrated discrimination improvement. Am J Epidemiol 174(3):364–374
8. Kerr KF, Wang Z, Janes H, McClelland R, Psaty BM, Pepe MS (2014) Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology 25(1):114–121
9. Li J, Jiang B, Fine JP (2013) Multicategory reclassification statistics for assessing improvements in diagnostic accuracy. Biostatistics 14(2):382–394
10. McIntosh MW, Pepe MS (2002) Combining several screening tests: optimality of the risk score. Biometrics 58(3):657–664
11. Pencina M, D’Agostino R, D’Agostino R, Vasan R (2008) Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 27(2):157–172
12. Pencina MJ, D’Agostino RB, Steyerberg EW (2011) Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med 30(1):11–21
13. Pencina MJ, D’Agostino RB, Demler OV (2012) Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvement for normal variables and nested models. Stat Med 31(2):101–113
14. Pepe M, Janes H (2013) Methods for evaluating prediction performance of biomarkers and tests. In: Lee ML, Gail M, Pfeiffer R, Satten G, Cai T, Gandy A (eds) Risk assessment and evaluation of predictions. Springer, Berlin, pp 107–142
15. Pepe M, Kerr K, Longton G, Wang Z (2013a) Testing for improvement in prediction model performance. Stat Med 32(9):1467–1482
16. Pepe MS, Janes H, Kerr KF, Psaty BM (2013b) Net reclassification index: a misleading measure of prediction improvement. University of Washington Department of Biostatistics Working Paper #394. http://biostats.bepress.com/uwbiostat/paper394
17. Pfeiffer R, Gail M (2011) Two criteria for evaluating risk prediction models. Biometrics 67(3):1057–1065
18. Steyerberg EW (2010) Clinical prediction models: a practical approach to development, validation, and updating. Springer, New York
19. Thompson IM, Ankerst DP, Chi C, Lucia MS, Goodman PJ, Crowley JJ, Parnes HL, Coltman CA Jr (2005) Operating characteristics of prostate-specific antigen in men with an initial PSA level of 3.0 ng/ml or lower. JAMA 294(1):66–70
20. Tzoulaki I, Liberopoulos G, Ioannidis JP (2009) Assessment of claims of improved prediction beyond the Framingham risk score. JAMA 302(21):2345–2352
21. Vickers AJ, Cronin AM (2010) Traditional statistical methods for evaluating prediction models are uninformative as to clinical value: towards a decision analytic framework. In: Seminars in oncology, vol 37, p 31
22. Vickers A, Elkin E (2006) Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 26(6):565
23. Vickers AJ, Pepe MS (2014) Does the net reclassification index help us evaluate models and markers? Ann Intern Med 160(2):136–137
24. Vickers AJ, Cronin AM, Begg CB (2011) One statistical test is sufficient for assessing new predictive markers. BMC Med Res Methodol 11(1):13