Net risk reclassification P values: Valid or misleading?

Journal of the National Cancer Institute

Volumn 106, Issue 4, 2014, Pages

  Pepe, Margaret S ^a   Janes, Holly ^a   Li, Christopher I ^a  

^a Fred Hutchinson Cancer Research Center   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BIOLOGICAL MARKER; TUMOR MARKER;

ARTICLE; CANCER STATISTICS; HUMAN; NET RISK RECLASSIFICATION; PREDICTIVE VALUE; PRIORITY JOURNAL; RECEIVER OPERATING CHARACTERISTIC; RISK ASSESSMENT; SIMULATION; STATISTICAL SIGNIFICANCE; VALIDITY; AREA UNDER THE CURVE; FACTUAL DATABASE; LABORATORY DIAGNOSIS; PROGNOSIS; RISK; RISK FACTOR; STATISTICAL MODEL;

AREA UNDER CURVE; DATABASES, FACTUAL; FALSE POSITIVE REACTIONS; HUMANS; MODELS, STATISTICAL; ODDS RATIO; PREDICTIVE VALUE OF TESTS; PROGNOSIS; RISK ASSESSMENT; RISK FACTORS; ROC CURVE; TUMOR MARKERS, BIOLOGICAL;

EID: 84898842206     PISSN: 00278874     EISSN: 14602105     Source Type: Journal    
DOI: 10.1093/jnci/dju041     Document Type: Article

References (34)

1. Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157-172.
2. Pencina MJ, D'Agostino Sr RB, Steyerberg EW. Extensions of net reclassi-fication improvement calculations to measure usefulness of new biomark-ers. Stat Med. 2011;30(1):11-21.
3. Hlatky MA, Greenland P, Arnett DK, et al. Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association Circulation. 2009;119(17):2408-2416.
4. Moons KG, Kengne A P, Woodward M, et al. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012;98(9):683-690.
5. Spitz MR, Amos CI, D'Amelio A Jr, Dong Q, Etzel C. Re: Discriminatory accuracy from single-nucleotide polymorphisms in models to predict breast cancer risk. J Natl Cancer Inst. 2009;101(24):1731-1732.
6. Mealiffe ME, Stokowski R P, Rhees BK, Prentice RL, Pettinger M, Hinds DA. Assessment of clinical validity of a breast cancer risk model combining genetic and clinical information. J Natl Cancer Inst. 2010;102(21):1618-1627.
7. Fox P, Hudson M, Brown C, et al. Markers of systemic inflammation predict survival in patients with advanced renal cell cancer. Br J Cancer. 2013;109(1):147-153.
8. Gentles AJ, Alizadeh AA. Utility in prognostic value added by molecular profiles for diffuse large B-cell lymphoma. Blood. 2013;121(15):3052-3054.
9. Vander Lugt MT, Braun TM, Hanash S, et al. NST2 as a marker for risk of therapy-resistant graft-versus-host disease and death. N Engl J Med. 2013;369(6):529-539
10. Hilden J, Gerds TA. A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassifica-tion index [published online ahead of print April 2, 2013]. Stat Med. 2013; doi:10.1002/sim.5804.
11. Pepe MS, Fan J, Feng Z, Gerds T, Hilden J. The Net Reclassification Index (NRI): a misleading measure of prediction improvement with mis-calibrated or over fit models. UW Biostatistics Working Paper Series. 2013 Working Paper 392. http://biostats.bepress.com/uwbiostat/paper392. Accessed February 12, 2014.
12. Gail MH. Value of adding single-nucleotide polymorphism genotypes to a breast cancer risk model. J Natl Cancer Inst. 2009;101(13):959-963.
13. Pepe MS. Problems with risk reclassification methods for evaluating prediction models. Am J Epidemiol. 2011;173(11):1327-1335.
14. Pepe, MS, Kerr, KF, Longton, G and Wang, Z. Testing for improvement in prediction model performance. Stat Med 2013;32(9):1467-1482.
15. Pencina MJ, D'Agostino RB, Pencina KM, Janssens AC, Greenland P. Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol. 2012;176(6):473-481.
16. Pepe MS. The Statistical Evaluation of Medical Tests for Classification and Prediction. 1st ed. New York: Oxford University Press; 2003.
17. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837-845.
18. Cook NR, Paynter N P. Performance of reclassification statistics in comparing risk prediction models. Biom J. 2011;53(2):237-258.
19. Pepe MS, Janes H, Kerr KF, Psaty BM. Net Reclassification Index: a misleading measure of prediction improvement. 2013 UW Biostatistics Working Paper Series. Working Paper 394. http://biostats.bepress.com/uwbiostat/paper394. Accessed February 12, 2014.
20. van Belle G. Statistical Rules of Thumb. 2nd ed. Hoboken, NJ: John Wiley & Sons; 2008
21. Pepe MS, Feng Z, Gu JW. Comments on "Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond." Stat Med 2008;27(2):173-181.
22. Vickers AJ, Cronin AM, Begg CB. One statistical test is sufficient for assessing new predictive markers. BMC Med Res Methodol. 2011;11:13.
23. Demler OV, Pencina MJ, D'Agostino RB, Sr. Misuse of DeLong test to compare AUCs for nested models. Stat Med. 2012;31(23):2577-2587.
24. Pepe MS, Janes H. Gauging the performance of SNPs, biomarkers, and clinical factors for predicting risk of breast cancer. J Natl Cancer Inst. 2008;100(14):978-979.
25. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928-935.
26. Kerr KF, Wang Z, Janes H, McClelland R, Psaty BM, Pepe MS. Net Reclassification Indices for evaluating risk prediction instruments: a critical review. Epidemiology. 2014;25(1):114-121.
27. Pickering JW, Endre ZH. New metrics for assessing diagnostic potential of candidate biomarkers. Clin J Am Soc Nephrol. 2012;7(8):1-10.
28. Gail MH. Response: Re: discriminatory accuracy from single-nucleotide polymorphisms in models to predict breast cancer risk. J Natl Cancer Inst. 2009;101(24):1732.
29. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565-574.
30. Baker SG. Putting Risk Prediction in Perspective: Relative Utility Curves. J Natl Cancer Inst. 2009;101(22):1538-1542.
31. Vickers A, Cronin A, Roobol M, et al. Reducing unnecessary biopsy during prostate cancer screening using a four-kalikrein panel: an independent replication. J Clin Oncol. 2010;28(15):2493-2498.
32. Di Napoli M, Godoy DA, Campi V, et al. C-reactive protein level measurement improves mortality prediction when added to the spontaneous intracerebral hemorrhage score. Stroke. 2011;42:1230-1236.
33. Scattoni V, Lazzeri M, Lughezzani G, et al. Head-to-head comparison of prostate health index and urinary PCA3 for predicting cancer at initial or repeat biopsy. J Urol. 2013;190(2):496-501.
34. Raji OY, Duffy SW, Agbaje OF, Baker SG, Christiani DC, Cassidy A, Field JK. Predictive accuracy of the Liverpool Lung Project risk model for stratifying patients for computed tomography screening for lung cancer: a case-control and cohort validation study. Ann Intern Med. 2012;157(4):242-250.