Extending the c-statistic to nominal polytomous outcomes: The Polytomous Discrimination Index

Statistics in Medicine

Volumn 31, Issue 23, 2012, Pages 2610-2626

  Van Calster, Ben ^a,b   Van Belle, Vanya ^a,c   Vergouwe, Yvonne ^b   Timmerman, Dirk ^a   Van Huffel, Sabine ^a,c   Steyerberg, Ewout W ^b  

^a UNIVERSITY OF LEUVEN   (Belgium)

^b ERASMUS MEDICAL CENTER   (Netherlands)

^c Katholieke Universiteit Leuven   (Belgium)

Author keywords

c Statistic; Discrimination; Model performance; Polytomous discrimination index; Polytomous risk prediction

Indexed keywords

ARTICLE; CONCORDANCE STATISTICS; DISCRIMINANT ANALYSIS; INTERMETHOD COMPARISON; MULTIVARIATE LOGISTIC REGRESSION ANALYSIS; POLYTOMOUS DISCRIMINATION INDEX; PREDICTIVE VALUE; PROBABILITY; STATISTICAL CONCEPTS;

AREA UNDER CURVE; DATA INTERPRETATION, STATISTICAL; FEMALE; HUMANS; MALE; NUMERICAL ANALYSIS, COMPUTER-ASSISTED; OVARIAN NEOPLASMS; PROGNOSIS; ROC CURVE; TESTICULAR NEOPLASMS;

EID: 84866464239     PISSN: 02776715     EISSN: 10970258     Source Type: Journal    
DOI: 10.1002/sim.5321     Document Type: Article

References (43)

1. D'Agostino Sr RB, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, Kannel WB.General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation 2008; 117:743-753. DOI: 10.1161/CIRCULATIONAHA.107.699579.
2. Steyerberg EW, Mushkudiani N, Perel P, Butcher I, Lu J, McHugh GS, Murray GD, Marmarou A, Roberts I, Habbema JDF, Maas AIR.Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Medicine 2008; 5:e165. DOI: 10.1371/journal.pmed.0050165.
3. Timmerman D, Testa AC, Bourne T, Ferrazzi E, Ameye L, Konstantinovic ML, Van Calster B, Collins WP, Vergote I, Van Huffel S, Valentin L.A logistic regression model to distinguish between the benign and malignant adnexal mass before surgery: a multicenter study by the International Ovarian Tumor Analysis (IOTA) group. Journal of Clinical Oncology 2005; 23:8794-8801. DOI: 10.1200/JCO.2005.01.7632.
4. Harrell Jr FE, Califf RM, Pryor DB, Lee KL, Rosati RA.Evaluating the yield of medical tests. Journal of the American Medical Association 1982; 247:2543-2546. DOI: 10.1001/jama.247.18.2543.
5. Harrell Jr FE.Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer: New York, 2001.
6. Mandic S, Go C, Aggarwal I, Myers J, Froelicher VF.Relationship of predictive modeling to receiver operating characteristics. Journal of Cardiopulmonary Rehabilitation & Prevention 2008; 28:415-419. DOI: 10.1097/HCR.0b013e31818c3c78.
7. Newcombe RG.Confidence intervals for an effect size measure based on the Mann-Whitney statistic. Part 1: general issues and tail-area-based methods. Statistics in Medicine 2006; 25:543-557. DOI: 10.1002/sim.2323.
8. Newcombe RG.Confidence intervals for an effect size measure based on the Mann-Whitney statistic. Part 2: asymptotic methods and evaluation. Statistics in Medicine 2006; 25:559-573. DOI: 10.1002/sim.2324.
9. Swets JA.Measuring the accuracy of diagnostic systems. Science 1988; 240:1285-1293. DOI: 10.1126/science.3287615.
10. Hanley JA, McNeil BJ.The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143:29-36.
11. Bamber D.The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology 1975; 12:387-415. DOI: 10.1016/0022-2496(75)90001-2.
12. Harrell Jr FE, Margolis PA, Gove S, Mason KE, Mulholland EK, Lehmann D, Muhe L, Gatchalian S, Eichenwald HF.Development of a clinical prediction model for an ordinal outcome: the World Health Organization multicentre study of clinical signs and etiological agents of pneumonia, sepsis and meningitis in young infants. WHO/ARI Young Infant Multicentre Study Group. Statistics in Medicine 1998; 17:909-944. DOI: 10.1002/(SICI)1097-0258(19980430)17:8<909::AID-SIM753>3.3.CO;2-F.
13. Hilden J.The area under the ROC curve and its competitors. Medical Decision Making 1991; 11:95-101. DOI: 10.1177/0272989X9101100204.
14. Mossman D.Three-way ROCs. Medical Decision Making 1999; 19:78-89. DOI: 10.1177/0272989X9901900110.
15. Scurfield BK.Multiple-event forced-choice tasks in the theory of signal detectability. Journal of Mathematical Psychology 1996; 40:253-269. DOI: 10.1006/jmps.1996.0024.
16. Pepe MS, Feng Z, Huang Y, Longton G, Prentice R, Thompson IM, Zheng Y.Integrating the predictiveness of a marker with its performance as a classifier. American Journal of Epidemiology 2008; 167:362-368. DOI: 10.1093/aje/kwm305.
17. Pencina MJ, D'Agostino RB, D'Agostino Jr RB, Vasan RS.Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Statistics in Medicine 2008; 27:157-172. DOI: 10.1002/sim.2929.
18. Cook NR.Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007; 115:928-935. DOI: 10.1161/CIRCULATIONAHA.106.672402.
19. Janes H, Pepe MS, Gu W.Assessing the value of risk predictions by using risk stratification tables. Annals of Internal Medicine 2008; 149:751-760.
20. Altman DG, Vergouwe Y, Royston P, Moons KGM.Prognosis and prognostic, research: validating a prognostic model. British Medical Journal 2009; 338:1432-1435. DOI: 10.1136/bmj.b605.
21. Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Elkind MSV, Go AS, Harrell Jr FE, Hong Y, Howard BV, Howard VJ, Hsue PY, Kramer CM, McConnell JP, Normand SLT, O'Donnell CJ, Smith Jr SC, Wilson PWF, American Heart Association Expert Panel on Subclinical Atherosclerotic Diseases and Emerging Risk Factors and the Stroke Council.Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation 2009; 119:2408-2416. DOI: 10.1161/CIRCULATIONAHA.109.192278.
22. Pencina MJ, D'Agostino Sr RB, Steyerberg EW.Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Statistics in Medicine 2011; 30:11-21. DOI: 10.1002/sim.4085.
23. Van Calster B, Van Huffel S.Integrated discrimination improvement and probability-sensitive AUC variants. Statistics in Medicine 2010; 29:318-319. DOI: 10.1002/sim.3761.
24. Moons KGM, Stijnen T, Michel BC, Büller HR, Van Es GA, Grobbee DE, Habbema JDF.Application of treatment thresholds to diagnostic-test evaluation: an alternative to the comparison of areas under receiver operating characteristic curves. Medical Decision Making 1997; 17:447-454. DOI: 10.1177/0272989X9701700410.
25. Adams NM, Hand DJ.An improved measure for comparing diagnostic tests. Computers in Biology and Medicine 2000; 30:89-96. DOI: 10.1016/S0010-4825(99)00025-6.
26. Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JDF.Validity of prognostic models: when is a model clinically useful?Seminars in Urologic Oncology 2002; 20:96-107. DOI: 10.1053/suro.2002.32521.
27. Vickers AJ, Elkin EB.Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making 2006; 26:565-574. DOI: 10.1177/0272989X06295361.
28. Steyerberg EW, Pencina MJ, Lingsma HF, Kattan MW, Vickers AJ, Van Calster B.Assessing the incremental value of diagnostic and prognostic markers: a review and illustration. European Journal of Clinical Investigation 2012; 42:216-228.
29. Steyerberg EW, Van Calster B, Pencina MJ.Performance measures for prediction models and markers: evaluation of predictions and classifications. Revista Española de Cardiologia 2011; 64:788-794. DOI: 10.1016/j.rec.2011.05.004.
30. Vickers AJ, Cronin AM, Begg CB.One statistical test is sufficient for assessing new predictive markers. BMC Medical Research Methodology 2011; 11:13. DOI: 10.1186/1471-2288-11-13.
31. Baker SG, Cook NR, Vickers A, Kramer BS.Using relative utility curves to evaluate risk prediction. Journal of the Royal Statistical Society: Series A 2009; 172:729-748. DOI: 10.1111/j.1467-985X.2009.00592.x.
32. Steyerberg EW.Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer: New York, 2009. 10.1007/978-0-387-77244-8.
33. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW.Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21:128-138. DOI: 10.1097/EDE.0b013e3181c30fb2.
34. Hand DJ, Till RJ.A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning 2001; 45:171-186. DOI: 10.1023/A:1010920819831.
35. Obuchowski NA, Goske MJ, Applegate KE.Assessing physicians' accuracy in diagnosing paediatric patients with acute abdominal pain: measuring accuracy for multiple diseases. Statistics in Medicine 2001; 20:3261-3278. DOI: 10.1002/sim.944.
36. Ferri C, Hernandez-Orallo J, Salido MA.Volume under the ROC surface for multi-class problems. Proceedings of the 14th European Conference on Machine Learning (ECML2003), Cavtat-Dubrovnik, Croatia, 2003; 108-120. DOI: 10.1007/978-3-540-39857-8_12.
37. Provost F, Domingos P.Tree induction for probability-based ranking. Machine Learning 2003; 52:199-215. DOI: 10.1023/A:1024099825458.
38. Dreiseitl S, Ohno-Machado L, Binder M.Comparing three-class diagnostic tests by three-way ROC analysis. Medical Decision Making 2000; 20:323-331. DOI: 10.1177/0272989X0002000309.
39. Vergouwe Y, Steyerberg EW, Foster RS, Sleijfer DT, Fosså SD, Gerl A, de Wit R, Roberts JT, Habbema JD.Predicting retroperitoneal histology in postchemotherapy testicular germ cell cancer: a model update and multicentre validation with more than 1000 patients. European Urology 2007; 51:424-432. DOI: 10.1016/j.eururo.2006.06.047.
40. Van Calster B, Valentin L, Van Holsbeke C, Testa AC, Bourne T, Van Huffel S, Timmerman D.Polytomous diagnosis of ovarian tumors as benign, borderline, primary invasive or metastatic: development and validation of standard and kernel-based risk prediction models. BMC Medical Research Methodology 2010; 10:96. DOI: 10.1186/1471-2288-10-96.
41. Steyerberg EW, Keizer HJ, Fosså SD, Sleijfer DT, Toner GC, Schraffordt Koops H, Mulders PF, Messemer JE, Ney K, Donohue JP.Prediction of residual retroperitoneal mass histology after chemotherapy for metastatic nonseminomatous germ cell tumor: multivariate analysis of individual patient data from six study groups. Journal of Clinical Oncology 1995; 13:1177-1187.
42. Hilden J.Evaluation of diagnostic tests: the schism. Internal report, Institute of Public Health, University of Copenhagen, 2004. Accessed via http://staff.pubhealth.ku.dk/~jh/ on November 18, 2010.
43. 1, 2, 6 model predicts risk of MLH1, MLH2, and MSH6 germline mutations based on cancer history. Gastroenterology 2011; 140:73-81. DOI: 10.1053/j.gastro.2010.08.021.