Discrimination ability of prediction models for ordinal outcomes: Relationships between existing measures and a new measure

Biometrical Journal

Volumn 54, Issue 5, 2012, Pages 674-685

  Van Calster, Ben ^a   Van Belle, Vanya ^a   Vergouwe, Yvonne ^b   Steyerberg, Ewout W ^b  

^a UNIVERSITY OF LEUVEN   (Belgium)

^b ERASMUS MEDICAL CENTER   (Netherlands)

Author keywords

c Index; Discrimination; Model performance; Ordinal outcomes; Prediction models

Indexed keywords

BRAIN;

C-INDEX; CHARACTERISTIC SURFACES; DISCRIMINATION; DISCRIMINATION ABILITY; MODEL-BASED OPC; MODELING PERFORMANCE; ORDINAL OUTCOME; PREDICTION MODELLING; RECEIVER OPERATING CHARACTERISTIC CURVES; RECEIVER OPERATING CHARACTERISTICS;

FORECASTING;

EID: 84865536455     PISSN: 03233847     EISSN: 15214036     Source Type: Journal    
DOI: 10.1002/bimj.201200026     Document Type: Article

References (41)

1. Adams, N. M. and Hand, D. J. (2000). An improved measure for comparing diagnostic tests. Computers in Biology and Medicine 30, 89-96.
2. Alonzo, T. A., Nakas, C. T., Yiannoutsos, C. T. and Bucher, S. (2009). A comparison of tests for restricted orderings in the three-class case. Statistics in Medicine 28, 1144-1158.
3. Baker, S. G., Cook, N. R., Vickers, A. and Kramer, B. S. (2009). Using relative utility curves to evaluate risk prediction. Journal of the Royal Statistical Society Series A 172, 729-748.
4. Bamber, D. (1975). The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology 12, 387-415.
5. Cook, N. R. (2007). Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 115, 928-935.
6. Feinstein, A. R. (1990). The inadequacy of binary models for the clinical reality of three-zone diagnostic decisions. Journal of Clinical Epidemiology 43, 109-113.
7. Flandre, P. and O'Quigley, J. (2007). Predictive strength of Jonckheere's test for trend: an application to genotypic scores in HIV infection. Statistics in Medicine 26, 4441-4454.
8. Gerds, T. A., Cai, T. and Schumacher, M. (2008). The performance of risk prediction models. Biometrical Journal 50, 457-479.
9. Gerds, T. A. and van de Wiel, M. A. (2011). Confidence scores for prediction models. Biometrical Journal 53, 259-274.
10. Hanley, J. A. and McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29-36.
11. Harrell, F. E. Jr. (2001). Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer, New York.
12. Harrell, F. E. Jr, Califf, R. M., Pryor, D. B., Lee, K. L. and Rosati, R. A. (1982). Evaluating the yield of medical tests. Journal of the American Medical Association 247, 2543-2546.
13. Harrell, F. E. Jr, Margolis, P. A., Gove, S., Mason, K. E., Mulholland, E. K., Lehmann, D., Muhe, L., Gatchalian, S. and Eichenwald, H. F. (1998). Development of a clinical prediction model for an ordinal outcome: the World Health Organization Multicentre Study of Clinical Signs and Etiological Agents of Pneumonia, Sepsis and Meningitis in Young Infants. WHO/ARI Young Infant Multicentre Study Group. Statistics in Medicine 17, 909-944.
14. Herbrich, R., Graepel, T. and Obermayer, K. (2000). Large margin rank boundaries for ordinal regression. In: Bartlett, P. J., Schölkopf, B., Schuurmans, D. and Smola, A. (Eds.), Advances in Large Margin Classifiers, The MIT Press, Cambridge, pp. 115-132.
15. Hilden, J. (1991). The area under the ROC curve and its competitors. Medical Decision Making 11, 95-101.
16. Hlatky, M. A., Greenland, P., Arnett, D. K., Ballantyne, C. M., Elkind, M. S. V., Go, A. S., Harrell, F. E. Jr, Hong, Y., Howard, B. V., Howard, V. J., Hsue, P. Y., Kramer, C. M., McConnell, J. P., Normand, S. L. T., O'Donnell, C. J., Smith, S. C. Jr, Wilson, P. W. F. and American Heart Association Expert Panel on Subclinical Atherosclerotic Diseases and Emerging Risk Factors and the Stroke Council. (2009). Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation 119, 2408-2416.
17. Janes, H., Pepe, M. S. and Gu, W. (2008). Assessing the value of risk predictions by using risk stratification tables. Annals of Internal Medicine 149, 751-760.
18. Jennett, B. and Bond, M. (1975). Assessment of outcome after severe brain damage. Lancet 305, 480-484.
19. Jonckheere, A. R. (1954). A distribution-free k-sample test against ordered alternatives. Biometrika 41, 133-145.
20. Kim, J. O. (1971). Predictive measures of ordinal association. American Journal of Sociology 76, 891-907.
21. Hukkelhoven, C. W., Steyerberg, E. W., Habbema, J. D. F., Farace, E., Marmarou, A., Murray, G. D., Marshall, L. F. and Maas, A. I. R. (2005). Predicting outcome after traumatic brain injury: development and validation of a prognostic score based on admission characteristics. Journal of Neurotrauma 22, 1025-1039.
22. Mossman, D. (1999). Three-way ROCs. Medical Decision Making 19, 78-89.
23. Nakas, C. T. and Yiannoutsos, C. T. (2004). Ordered multiple-class ROC analysis with continuous measurements. Statistics in Medicine 23, 3437-3449.
24. Newcombe, R. G. (2006a). Confidence intervals for an effect size measure based on the Mann-Whitney statistic. Part 1: general issues and tail-area-based methods. Statistics in Medicine 25, 543-557.
25. Newcombe, R. G. (2006b). Confidence intervals for an effect size measure based on the Mann-Whitney statistic. Part 2: asymptotic methods and evaluation. Statistics in Medicine 25, 559-573.
26. Obuchowski, N. A. (2005). Estimating and comparing diagnostic tests' accuracy when the gold standard is not binary. Academic Radiology 12, 1198-1204.
27. Pencina, M. J., D'Agostino, R. B. Sr and Steyerberg, E. W. (2011). Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Statistics in Medicine 30, 11-21.
28. Scurfield, B. K. (1996). Multiple-event forced-choice tasks in the theory of signal detectability. Journal of Mathematical Psychology 40, 253-269.
29. Smith, W. D., Dutton, R. C. and Smith, N. T. (1996). A measure of association for assessing prediction accuracy that is a generalizaiton of non-parametric ROC area. Statistics in Medicine 15, 1199-1215.
30. Somers, R. H. (1962). A new asymmetric measure of association for ordinal variables. American Sociological Review 27, 799-811.
31. Steyerberg, E. W. (2009). Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer, New York.
32. Steyerberg, E. W., Mushkudiani, N., Perel, P., Butcher, I., Lu, J., McHugh, G. S., Murray, G. D., Marmarou, A., Roberts, I., Habbema, J. D. F. and Maas, A. I. R. (2008). Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Medicine 5, e165.
33. Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J. and Kattan, M. W. (2010). Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21, 128-138.
34. Terpstra, J. and Magel, R. (2003). A new nonparametric test for the ordered alternative problem. Journal of Nonparametric Statistics 15, 289-301.
35. Timmerman, D., Van Calster, B., Testa, A. C., Guerriero, S., Fischerova, D., Lissoni, A. A., Van Holsbeke, C., Fruscio, R., Czekierdowski, A., Jurkovic, D., Savelli, L., Vergote, I., Bourne, T., Van Huffel, S. and Valentin, L. (2010). Ovarian cancer prediction in adnexal masses using ultrasound-based logistic regression models: a temporal and external validation study by the IOTA group. Ultrasound in Obstetrics and Gynecology 36, 226-234.
36. Van Calster, B. and Van Huffel, S. (2010). Integrated discrimination improvement and probability-sensitive AUC variants. Statistics in Medicine 29, 318-319.
37. Van Holsbeke, C., Van Calster, B., Bourne, T., Ajossa, S., Testa, A. C., Guerriero, S., Fruscio, R., Lissoni, A. A., Czekierdowski, A., Savelli, L., Van Huffel, S., Valentin, L. and Timmerman, D. (2012). External validation of diagnostic models to estimate the risk of malignancy in adnexal masses. Clinical Cancer Research 18, 815-825.
38. Vergouwe, Y., Steyerberg, E. W., Eijkemans, M. J. and Habbema, J. D. F. (2002). Validity of prognostic models: when is a model clinically useful? Seminars in Urologic Oncology 20, 96-107.
39. Vickers, A. J. and Elkin, E. B. (2006). Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making 26, 565-574.
40. Waegeman, W., De Baets, B. and Boullart, L. (2008). ROC analysis in ordinal regression learning. Pattern Recognition Letters 29, 1-9.
41. Xiong, C., van Belle, G., Miller, J. P. and Morris, J. C. (2006). Measuring and estimating diagnostic accuracy when there are three ordinal diagnostic groups. Statistics in Medicine 25, 1251-1273.