Testing for improvement in prediction model performance

Statistics in Medicine

Volumn 32, Issue 9, 2013, Pages 1467-1482

  Pepe, Margaret Sullivan ^a   Kerr, Kathleen F ^b   Longton, Gary ^a   Wang, Zheyu ^b  

^a Fred Hutchinson Cancer Research Center   (United States)

^b UNIVERSITY OF WASHINGTON   (United States)

Author keywords

Biomarker; Logistic regression; Receiver operating characteristic curve; Risk factors; Risk reclassification

Indexed keywords

BIOLOGICAL MARKER; CREATININE;

AREA UNDER THE CURVE; ARTICLE; CREATININE BLOOD LEVEL; HIGH RISK POPULATION; HUMAN; KIDNEY ARTERY STENOSIS; LOGISTIC REGRESSION ANALYSIS; MATHEMATICAL COMPUTING; MATHEMATICAL MODEL; MEDICAL DECISION MAKING; NULL HYPOTHESIS; PREDICTION; PREDICTIVE VALUE; RECEIVER OPERATING CHARACTERISTIC; RISK ASSESSMENT; RISK FACTOR; SIMULATION;

AREA UNDER CURVE; BIOLOGICAL MARKERS; COMPUTER SIMULATION; HUMANS; MODELS, STATISTICAL; PREDICTIVE VALUE OF TESTS; RENAL ARTERY OBSTRUCTION; RISK ASSESSMENT; ROC CURVE;

EID: 84876321464     PISSN: 02776715     EISSN: 10970258     Source Type: Journal    
DOI: 10.1002/sim.5727     Document Type: Article

References (45)

1. Gail MH, Brinton LA, Byar DP, Corle DK, Green SB, Schairer C, Mulvihill JJ. Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. Journal of the National Cancer Institute 1989; 81:1879-1886. DOI: 10.1093/jnci/81.24.1879.
2. Gail MH, Costantino JP. Validating and improving models for projecting the absolute risk of breast cancer. Journal of the National Cancer Institute 2001; 93:334-335. DOI: 10.1093/jnci/93.5.334.
3. Barlow WE, White E, Ballard-Barbash R, Vacek PM, Titus-Ernstoff L, Carney PA, Tice JA, Buist DS, Geller BM, Rosenberg R, Yankaskas BC, Kerlikowske K. Prospective breast cancer risk prediction model for women undergoing screening mammography. Journal of the National Cancer Institute 2006; 98:1204-1214. DOI: 10.1093/jnci/djj331.
4. Chen J, Pee D, Ayyagari R, Graubard B, Schairer C, Byrne C, Benichou J, Gail MH. Projecting absolute invasive breast cancer risk in white women with a model that includes mammographic density. Journal of the National Cancer Institute 2006; 98:1215-1226. DOI: 10.1093/jnci/djj332.
5. Wacholder S, Hartge P, Prentice R, Garcia-Closas M, Feigelson HS, Diver WR, Thun MJ, Cox DG, Hankinson SE, Kraft P, Rosner B, Berg CD, Brinton LA, Lissowska J, Sherman ME, Chlebowski R, Kooperberg C, Jackson RD, Buckman DW, Hui P, Pfeiffer R, Jacobs KB, Thomas GD, Hoover RN, Gail MH, Chanock SJ, Hunter DJ. Performance of common genetic variants in breast-cancer risk models. New England Journal of Medicine 2010; 362:986-993. DOI: 10.1056/NEJMoa0907727.
6. Gail MH. Probability discriminatory accuracy from single-nucleotide polymorphisms in models to predict breast cancer risk. Journal of the National Cancer Institute 2008; 100:1037-1041. DOI: 10.1093/jnci/djn180.
7. Gail MH. Value of adding single-nucleotide polymorphism genotypes to a breast cancer risk model. Journal of the National Cancer Institute 2009; 101:959-963. DOI: 10.1093/jnci/djp130.
8. Wilson PW, D'Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation 1998; 97:1837-1847. DOI: 10.1161/01.CIR.97.18.1837.
9. Tzoulaki I, Liberopoulos G, Ioannidis JP. Assessment of claims of improved prediction beyond the Framingham risk score. Journal of the American Medical Association 2009; 302:2345-2352. DOI: 10.1001/jama.2009.1757.
10. Pepe MS, Feng Z, Huang Y, Longton G, Prentice R, Thompson IM, Zheng Y. Integrating the predictiveness of a marker with its performance as a classifier. American Journal of Epidemiology 2008; 167:362-368. DOI: 10.1093/aje/kwm305.
11. Pepe MS, Gu JW, Morris DE. The potential of genes and other markers to inform about risk. Cancer Epidemiology Biomarkers and Prevention 2010; 3:655-665. DOI: 10.1158/1055-9965.EPI-09-0510.
12. Pencina MJ, D'Agostino RB, Sr, D'Agostino RB, Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Statistics in Medicine 2008; 27:157-172. DOI: 10.1002/sim.2929.
13. Pencina MJ, D'Agostino RB, Sr, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Statistics in Medicine 2010; 30:11-21. DOI: 10.1002/sim.4085.
14. Cook NR, Paynter NP. Performance of reclassification statistics in comparing risk prediction models. Biometrical Journal 2011; 53:237-258. DOI: 10.1002/bimj.201000078.
15. Cook NR, Ridker PM. Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Annals of Internal Medicine 2009; 150:795-802. DOI: 10.1059/0003-4819-150-11-200906020-00007.
16. Pepe MS. Problems with risk reclassification methods for evaluating prediction models. American Journal of Epidemiology 2011; 173:1327-1335. DOI: 10.1093/aje/kwr013.
17. Vickers AJ, Cronin AM, Begg CB. One statistical test is sufficient for assessing new predictive markers. BMC Medical Research Methodology 2011; 11:13. DOI: 10.1186/1471-2288-11-13.
18. Demler OV, Pencina MJ, D'Agostino RB, Sr. Misuse of DeLong test to compare AUCs for nested models. Statistics in Medicine 2012; 31(23):2577-2587. DOI: 10.1002/sim.5328.
19. Pepe MS. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford University Press: Oxford, 2003.
20. Janes H, Pepe MS. Adjusting for covariate effects on classification accuracy using the covariate adjusted ROC curve. Biometrika 2009; 96:371-382. DOI: 10.1093/biomet/asp002.
21. Janes H, Pepe MS. Matching in studies of classification accuracy: implications for analysis, efficiency, and assessment of incremental value. Biometrics 2008; 64:1-9. DOI: 10.1111/j.1541-0420.2007.00823.x.
22. Huang Y, Pepe MS, Feng Z. Evaluating the predictiveness of a continuous marker. Biometrics 2007; 63:1181-1188. DOI: 10.1111/j.1541-0420.2007.00814.x.
23. Stern RH. Evaluating new cardiovascular risk factors for risk stratification. Journal of Clinical Hypertension 2008; 10:485-488. DOI: 10.1111/j.1751-7176.2008.07814.x.
24. Pepe MS, Feng Z, Gu JW. Invited commentary on 'Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond'. Statistics in Medicine 2008; 27:173-181. DOI: 10.1002/sim.2991.
25. Gu J W, Pepe MS. Measures to summarize and compare the predictive capacity of markers. International Journal of Biostatistics 2009; 5(1):article 27. DOI: 10.2202/1557-4679.1188.
26. Bura E, Gastwirth JL. The binary regression quantile plot: assessing the importance of predictors in binary regression visually. Biometrical Journal 2001; 43:5-21. DOI: 10.1002/1521-4036(200102)43:1<5: AID-BIMJ5>3.0.CO;2-6.
27. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making 2006; 26:565-574. DOI: 10.1177/0272989X06295361.
28. Baker SG. Putting risk prediction in perspective: relative utility curves. Journal of the National Cancer Institute 2009; 101:1538-1542. DOI: 10.1093/jnci/djp353.
29. Baker SG, Cook NR, Vickers A, Kramer BS. Using relative utility curves to evaluate risk prediction. Journal of the Royal Statistical Society Series B 2009; 172:729-748. DOI: 10.1111/j.1467-985X.2009.00592.x.
30. Green DM, Swets JA. Signal Detection Theory and Psychophysics. Wiley: New York, 1966.
31. Neyman J, Pearson ES. On the problem of the most efficient tests of statistical hypothesis. Philosophical Transactions of the Royal Society of London, Series A 1933; 231:289-337. DOI: 10.1098/rsta.1933.0009.
32. McIntosh MS, Pepe MS. Combining several screening tests: optimality of the risk score. Biometrics 2002; 58:657-664. DOI: 10.1111/j.0006-341X.2002.00657.x.
33. Gail MH, Pfeiffer RM. On criteria for evaluating models of absolute risk. Biostatistics 2005; 6:227-239. DOI: 10.1093/biostatistics/kxi005.
34. Huang Y, Pepe MS. A parametric ROC model based approach for evaluating the predictiveness of continuous markers in case-control studies. Biometrics 2009; 65:1133-1144. DOI: 10.1111/j.1541-0420.2009.01201.x.
35. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44:837-845. DOI: 10.1111/j.0006-341X.2002.00657.x.
36. Pepe M, Longton G, Janes H. Estimation and comparison of receiver operating characteristic curves. Stata Journal 2009; 9:1-16. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2774909/pdf/nihms90148.pdf.
37. Kerr KF, McClelland RL, Brown ER, Lumley T. Evaluating the incremental value of new biomarkers with integrated discrimination improvement. American Journal of Epidemiology 2011; 174:364-374. DOI: 10.1093/aje/kwr086.
38. Harrell F. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer Verlag: New York, 2001.
39. Steyerberg E. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer: New York, 2010.
40. Janssens AC, Deng Y, Borsboom GJ, Eijkemans MJ, Habbema JD, Steyerberg EW. A new logistic regression approach for the evaluation of diagnostic test results. Medical Decision Making 2005; 25:168-177. DOI: 10.1177/0272989X05275154.
41. Hosmer DW, Hosmer T, Le Cessie S, Lemeshow SA. Comparison of goodness-of-fit tests for the logistic regression model. Statistics in Medicine 1997; 16:965-980. DOI: 10.1002/(SICI)1097-0258(19970515)16:9<965: AID-SIM509>3.0.CO;2-O.
42. Hosmer DW, Lemeshow S. Applied Logistic Regression. Wiley: New York, 2000.
43. van Houwelingen HC. Validation, calibration, revision and combination of prognostic survival models. Statistics in Medicine 2000; 19:3401-3415. DOI: 10.1002/1097-0258(20001230)19:24<3401: AID-SIM554>3.0.CO;2-2.
44. Pfeiffer RM, Gail MH. Two criteria for evaluating risk prediction models. Biometrics 2011; 67:1057-1065. DOI: 10.1111/j.1541-0420.2010.01523.x.
45. Rapsomaniki E, White IR, Wood AM, Thompson SG, Emerging Risk Factors Collaboration. A framework for quantifying net benefits of alternative prognostic models. Statistics in Medicine 2012; 31:114-130. DOI: 10.1002/sim.4362.