Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: A discussion and proposal for an alternative approach

European Radiology

Volumn 25, Issue 4, 2015, Pages 932-939

  Halligan, Steve ^a   Altman, Douglas G ^b   Mallett, Susan ^c  

^a UNIVERSITY COLLEGE HOSPITAL   (United Kingdom)

^b UNIVERSITY OF OXFORD   (United Kingdom)

^c UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

Area under curve; CT colonography; Data interpretation; ROC curve; Sensitivity and specificity; Statistical

Indexed keywords

AREA UNDER THE CURVE; ARTICLE; CLINICAL ARTICLE; COMPUTED TOMOGRAPHIC COLONOGRAPHY; CONTROLLED STUDY; DIAGNOSTIC TEST; DIAGNOSTIC TEST ACCURACY STUDY; FALSE NEGATIVE RESULT; FALSE POSITIVE RESULT; HUMAN; IMAGING; META ANALYSIS (TOPIC); PREVALENCE; PRIORITY JOURNAL; RANDOMIZED CONTROLLED TRIAL; RECEIVER OPERATING CHARACTERISTIC; SENSITIVITY AND SPECIFICITY; DIAGNOSTIC IMAGING; REPRODUCIBILITY; STATISTICS AND NUMERICAL DATA;

AREA UNDER CURVE; COLONOGRAPHY, COMPUTED TOMOGRAPHIC; DIAGNOSTIC IMAGING; HUMANS; REPRODUCIBILITY OF RESULTS; ROC CURVE; SENSITIVITY AND SPECIFICITY;

EID: 84925496226     PISSN: 09387994     EISSN: 14321084     Source Type: Journal    
DOI: 10.1007/s00330-014-3487-0     Document Type: Article

References (28)

1. Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:29–36
2. Boyer B, Canale S, Arfi-Rouche J, Monzani Q, Khaled W, Balleyguier C (2013) Variability and errors when applying the BIRADS mammography classification. Eur J Radiol 82:388–397
3. Obuchowski NA (2005) ROC analysis. AJR Am J Roentgenol 184:364–372
4. Dwyer AJ (1996) In pursuit of a piece of the ROC. Radiology 201:621–625
5. Fawcett T (2006) An Introduction to ROC analysis. Pattern Recogn Lett 27:861–874
6. Wagner RF, Beiden SV, Campbell G, Metz CE, Sacks WM (2002) Assessment of medical imaging and computer-assist systems: lessons from recent experience. Acad Radiol 9:1264–1277
7. Obuchowski NA (2007) New methodological tools for multiple-reader ROC studies. Radiology 243:10–12
8. Obuchowski NA, Beiden SV, Berbaum KS et al (2004) Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol 11:980–995
9. Park SH, Goo JM, Jo CH (2004) Receiver operating characteristic (ROC) curve: practical review for radiologists. Korean J Radiol 5:11–18
10. Zweig MHCG (1993) Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem 39:561–577
11. Spiegelhalter D, Pearson M, Short I (2011) Visualizing uncertainty about the future. Science 333:1393–1400
12. McClish DK (1989) Analyzing a portion of the ROC curve. Med Decis Making 9:190–195
13. Boone D, Mallett S, Zhu S et al (2013) Patients' & healthcare professionals' values regarding true- & false-positive diagnosis when colorectal cancer screening by CT colonography: discrete choice experiment. PLoS One 8:e80767
14. Schwartz LM, Woloshin S, Sox HC, Fischhoff B, Welch HG (2000) US women's attitudes to false positive mammography results and detection of ductal carcinoma in situ: cross sectional survey. BMJ 320:1635–1640
15. Baker SG (2003) The central role of receiver operating characteristic (ROC) curves in evaluating tests for the early detection of cancer. J Natl Cancer Inst 95:511–515
16. Harrington MB (1990) Some methodological questions concerning receiver operating characteristic (ROC) analysis as a method for assessing image quality in radiology. J Digit Imaging 3:211–218
17. Halligan S, Altman DG, Mallett S et al (2006) Computed tomographic colonography: assessment of radiologist performance with and without computer-aided detection. Gastroenterology 131:1690–1699
18. Wagner RF, Metz CE, Campbell G (2007) Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol 14:723–748
19. Gur D, Rockette HE, Bandos AI (2007) "Binary" and "non-binary" detection tasks: are current performance measures optimal? Acad Radiol 14:871–876
20. Lewin JM, Hendrick RE, D'Orsi CJ et al (2001) Comparison of full-field digital mammography with screen-film mammography for cancer detection: results of 4,945 paired examinations. Radiology 218:873–880
21. Gur D, Bandos AI, Rockette HE (2008) Comparing areas under receiver operating characteristic curves: potential impact of the "Last" experimentally measured operating point. Radiology 247:12–15
22. Alemayehu D, Zou KH (2012) Applications of ROC analysis in medical research: recent developments and future directions. Acad Radiol 19:1457–1464
23. Zou KH (2012) Professor Charles E. Metz leaves profound legacy in ROC methodology: an introduction to the two Metz Memorial Issues. Acad Radiol 19:1447–1448
24. Van Calster B, Vickers AJ, Pencina MJ, Baker SG, Timmerman D, Steyerberg EW (2013) Evaluation of Markers and Risk Prediction Models: Overview of Relationships between NRI and Decision-Analytic Measures. Med Decis Making 33:490–501
25. Moons KG, Stijnen T, Michel BC et al (1997) Application of treatment thresholds to diagnostic-test evaluation: an alternative to the comparison of areas under receiver operating characteristic curves. Med Decis Making 17:447–454
26. Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS (2008) Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 27:157–172, discussion 207-112
27. Halligan S, Mallett S, Altman DG et al (2011) Incremental benefit of computer-aided detection when used as a second and concurrent reader of CT colonographic data: multiobserver study. Radiology 258:469–47628
28. Mallett S, Halligan S, Thompson M, Collins GS, Altman DG (2012) Interpreting diagnostic accuracy studies for patient care. BMJ 345:e3999