Predictive accuracy of novel risk factors and markers: A simulation study of the sensitivity of different performance measures for the Cox proportional hazards regression model

Statistical Methods in Medical Research

Volumn 26, Issue 3, 2017, Pages 1053-1077

  Austin, Peter C ^a,b,c   Pencinca, Michael J ^d,e   Steyerberg, Ewout W ^f  

^a INSTITUTE FOR CLINICAL EVALUATIVE SCIENCES   (Canada)

^b UNIVERSITY OF TORONTO   (Canada)

^c Canada   (Canada)

^d DUKE CLINICAL RESEARCH INSTITUTE   (United States)

^e DUKE UNIVERSITY SCHOOL OF MEDICINE   (United States)

^f ERASMUS MEDICAL CENTER   (Netherlands)

Author keywords

Cox proportional hazards model; discrimination; model performance; Monte Carlo simulations; predictive accuracy; predictive models; risk factors; Survival analysis

Indexed keywords

BIOLOGICAL MARKER;

ACCURACY; AGE; ARTICLE; CASE STUDY; COHORT ANALYSIS; COMPUTER SIMULATION; COX PROPORTIONAL HAZARDS REGRESSION MODEL; DATA ANALYSIS SOFTWARE; HEART FAILURE; HOSPITAL PATIENT; INTERMETHOD COMPARISON; MONTE CARLO METHOD; MORTALITY; OUTCOME ASSESSMENT; PREDICTIVE VALUE; PREVALENCE; PROPORTIONAL HAZARDS MODEL; RISK ASSESSMENT; RISK FACTOR; SCORING SYSTEM; SENSITIVITY ANALYSIS; SURVIVAL RATE; HUMAN;

AGE FACTORS; BIOMARKERS; HEART FAILURE; HUMANS; MONTE CARLO METHOD; PROPORTIONAL HAZARDS MODELS; RISK FACTORS;

EID: 85020744103     PISSN: 09622802     EISSN: 14770334     Source Type: Journal    
DOI: 10.1177/0962280214567141     Document Type: Article

References (45)

1. Austin PC, Manca A, Zwarenstein M, et al. A substantial and confusing variation exists in handling of baseline covariates in randomized controlled trials: a review of trials published in leading medical journals. J Clin Epidemiol 2010; 63: 142-153.
2. Chambless LE, Cummiskey CP, Cui G,. Several methods to assess improvement in risk prediction models: extension to survival analysis. Stat Med 2011; 30: 22-38.
3. Austin PC, Steyerberg EW,. Predictive accuracy of risk factors and markers: a simulation study of the effect of novel markers on different performance measures for logistic regression models. Stat Med 2013; 32: 661-672.
4. Choodari-Oskooei B, Royston P, Parmar MK,. A simulation study of predictive ability measures in a survival model II: explained randomness and predictive accuracy. Stat Med 2012; 31: 2644-2659.
5. Choodari-Oskooei B, Royston P, Parmar MK,. A simulation study of predictive ability measures in a survival model I: explained variation measures. Stat Med 2012; 31: 2627-2643.
6. Hielscher T, Zucknick M, Werft W, et al. On the prognostic value of survival models with application to gene expression signatures. Stat Med 2010; 29: 818-829.
7. Schemper M, Stare J,. Explained variation in survival analysis. Stat Med 1996; 15: 1999-2012.
8. Pencina MJ, D'Agostino RB Sr, Song L,. Quantifying discrimination of Framingham risk functions with different survival C statistics. Stat Med 2012; 31 (15): 1543-1553.
9. Uno H, Cai T, Pencina MJ, et al. On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Stat Med 2011; 30: 1105-1117.
10. Harrell FE Jr,. Regression modeling strategies, New York, NY: Springer-Verlag, 2001.
11. Steyerberg EW,. Clinical prediction models, New York: Springer-Verlag, 2009.
12. Pencina MJ, D'Agostino RB,. Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation. Stat Med 2004; 23: 2109-2123.
13. Chambless LE, Diao G,. Estimation of time-dependent area under the ROC curve for long-term risk prediction. Stat Med 2006; 25: 3474-3486.
14. Harrell FE Jr, Lee KL, Mark DB,. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996; 15: 361-387.
15. Gonen M, Heller G,. Concordance probability and discriminatory power in proportional hazards regression. Biometrika 2005; 92: 965-970.
16. Royston P, Sauerbrei W,. A new measure of prognostic separation in survival data. Stat Med 2004; 23: 723-748.
17. Kent J, O'Quigley J,. Measures of dependence for censored survival data. Biometrika 1988; 75: 525-534.
18. Korn EL, Simon R,. Measures of explained variation for survival data. Stat Med 1990; 9: 487-503.
19. O'Quigley J, Flandre P,. Predictive capability of proportional hazards regression. Proc Natl Acad Sci USA 1994; 91: 2310-2314.
20. Akazawa K,. Measures of explained variation for a regression model used in survival analysis. J Med Syst 1997; 21: 229-238.
21. O'Quigley J, Xu R,. Handbook of statistics in clinical oncology, New York: Marcel Dekker, 2001, pp. 397-410.
22. Royston P,. Explained variation for survival models. Stata J 2006; 6: 1-14.
23. Nagelkerke NJD,. A note on a general definition of the coefficient of determination. Biometrika 1991; 78: 691-692.
24. Xu R, O'Quigley J,. A measure of dependence for proportional hazards models. J Nonparametr Stat 1999; 12: 83-107.
25. O'Quigley J, Xu R, Stare J,. Explained randomness in proportional hazards models. Stat Med 2005; 24: 479-489.
26. Schemper M,. The explained variation in proportional hazards regression. Biometrika 1990; 77: 216-218.
27. Graf E, Schmoor C, Sauerbrei W, et al. Assessment and comparison of prognostic classification schemes for survival data. Stat Med 1999; 18: 2529-2545.
28. Schemper M, Henderson R,. Predictive accuracy and explained variation in Cox regression. Biometrics 2000; 56: 249-255.
29. Schemper M, Kaider A,. A new approach to estimate correlation coefficient in the presence of censoring and proportional hazards. Comput Stat Data Anal 1997; 23: 467-476.
30. Harrell FE,. The phglm procedure. SUGI supplemental library users guide, Cary, NC: SAS Institute, 1986, pp. 437-466.
31. Allison PD,. Survival analysis using SAS®: a practical guide, Cary, NC: SAS Institute, 2010.
32. Cragg JG, Uhler R,. The demand for automobiles. Can J Econ 1970; 3: 386-406.
33. Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27: 157-172.
34. Lusa L, Miceli R, Mariani L,. Estimation of predictive accuracy in survival analysis using R and S-PLUS. Comput Methods Programs Biomed 2007; 87: 132-137.
35. Bender R, Augustin T, Blettner M,. Generating survival times to simulate Cox proportional hazards models. Stat Med 2005; 24: 1713-1723.
36. Tu JV, Donovan LR, Lee DS, et al. Effectiveness of public report cards for improving the quality of cardiac care: the EFFECT study: a randomized trial. J Am Med Assoc 2009; 302: 2330-2337.
37. Tu JV, Donovan LR, Lee DS, et al. Quality of cardiac care in Ontario, Toronto, Ontario: Institute for Clinical Evaluative Sciences, 2004.
38. Lee DS, Austin PC, Rouleau JL, et al. Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model. J Am Med Assoc 2003; 290: 2581-2587.
39. Janssens AC, Moonesinghe R, Yang Q, et al. The impact of genotype frequencies on the clinical validity of genomic profiling for predicting common chronic diseases. Genet Med 2007; 9: 528-535.
40. Tzoulaki I, Liberopoulos G, Ioannidis JP,. Assessment of claims of improved prediction beyond the Framingham risk score. J Am Med Assoc 2009; 302: 2345-2352.
41. Austin PC, Steyerberg EW,. Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models. Stat Methods Med Res 2017; 26: 796-808.
42. Steyerberg EW, Harrell FE Jr, Borsboom GJ, et al. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 2001; 54: 774-781.
43. Lobo JM, Jiménez-Valverde A, Real R,. AUC: a misleading measure of the performance of predictive distribution models. Global Ecol Biogeogr 2007; 17: 145-151.
44. Hand DJ,. Measuring classifier performance: a coherent alternative to the area under the ROC curve. Mach Learn 2009; 77: 103-123.
45. Austin PC, Steyerberg EW,. Interpreting the concordance statistic of a logistic regression model: relation to the variance and odds ratio of a continuous explanatory variable. BMC Med Res Methodol 2012; 12: 82-82. DOI: 10.1186/1471-2288-12-82.