Exploration of analysis methods for diagnostic imaging tests: Problems with ROC AUC and confidence scores in CT colonography

PLoS ONE

Volumn 9, Issue 10, 2014, Pages

  Mallett, Susan ^a   Halligan, Steve ^b   Collins, Gary S ^c   Altman, Doug G ^c  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

AREA UNDER THE CURVE; ARTICLE; COLON POLYP; COMPUTED TOMOGRAPHIC COLONOGRAPHY; COMPUTER ASSISTED RADIOGRAPHY; CONFIDENCE SCORE; CONTROLLED STUDY; CURVE FITTING; DATA ANALYSIS; DIAGNOSTIC ACCURACY; DIAGNOSTIC IMAGING; DIAGNOSTIC TEST ACCURACY STUDY; FALSE POSITIVE RESULT; HUMAN; IMAGE ANALYSIS; MAJOR CLINICAL STUDY; NORMAL DISTRIBUTION; RECEIVER OPERATING CHARACTERISTIC; SCORING SYSTEM; SENSITIVITY AND SPECIFICITY; STATISTICAL PARAMETERS; COMPARATIVE STUDY; COMPUTER ASSISTED DIAGNOSIS; PROCEDURES; RADIOGRAPHY;

AREA UNDER CURVE; COLONIC POLYPS; COLONOGRAPHY, COMPUTED TOMOGRAPHIC; DIAGNOSTIC IMAGING; HUMANS; RADIOGRAPHIC IMAGE INTERPRETATION, COMPUTER-ASSISTED; ROC CURVE; SENSITIVITY AND SPECIFICITY;

EID: 84908544644     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0107633     Document Type: Article

References (43)

1. Mallett S, Halligan S, Thompson M, Collins GS, Altman DG (2012) Interpreting diagnostic accuracy studies for patient care. BMJ 345: e3999.
2. Halligan S, Mallett S, Altman DG, McQuillan J, Proud M, et al. (2011) Incremental benefit of computer-aided detection when used as a second and concurrent reader of CT colonographic data: multiobserver study. Radiology 258: 469-476.
3. Moons KG, Stijnen T, Michel BC, Buller HR, Van Es GA, et al. (1997) Application of treatment thresholds to diagnostic-test evaluation: an alternative to the comparison of areas under receiver operating characteristic curves. Med Decis Making 17: 447-454.
4. Adams NM, Hand DJ (2000) An improved measure for comparing diagnostic tests. Comput Biol Med 30: 89-96.
5. Vickers AJ, Cronin AM, Elkin EB, Gonen M (2008) Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak 8: 53.
6. Staff DGfIaF (2012) Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data - Premarket Approval (PMA) and Premarket Notification [510(k)] Submissions. In: Center for Devices and Radiological Health FaDA, U.S. Government, editor.
7. Obuchowski NA, Gallas BD, Hillis SL (2012) Multi-reader ROC studies with split-plot designs: a comparison of statistical methods. Acad Radiol 19: 1508-1517.
8. Boone D, Mallett S, Zhu S, Yao GL, Bell N, et al. (2013) Patients' and healthcare professionals' values regarding true- and false-positive diagnosis during colorectal cancer screening with CT colonography: A discrete choice experiment. PLOS ONE (in press).
9. Schwartz LM, Woloshin S, Sox HC, Fischhoff B,Welch HG (2000) US women's attitudes to false positive mammography results and detection of ductal carcinoma in situ: cross sectional survey. BMJ 320: 1635-1640.
10. Hilden J (1991) The area under the ROC curve and its competitors. Med Decis Making 11: 95-101.
11. Lewin JM, Hendrick RE, D'Orsi CJ, Isaacs PK, Moss LJ, et al. (2001) Comparison of full-field digital mammography with screen-film mammography for cancer detection: results of 4,945 paired examinations. Radiology 218: 873-880.
12. Halligan S, Altman DG, Mallett S, Taylor SA, Burling D, et al. (2006) Computed tomographic colonography: assessment of radiologist performance with and without computer-aided detection. Gastroenterology 131: 1690-1699.
13. Pan X, Metz CE (1997) The "proper" binormal model: parametric receiver operating characteristic curve estimation with degenerate data. Acad Radiol 4: 380-389.
14. Metz CE (1986) ROC methodology in radiologic imaging. Invest Radiol 21: 720-733.
15. Obuchowski NA, Beiden SV, Berbaum KS, Hillis SL, Ishwaran H, et al. (2004) Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol 11: 980-995.
16. Eng J (2012) Teaching receiver operating characteristic analysis: an interactive laboratory exercise. Acad Radiol 19: 1452-1456.
17. Hussain HK, Syed I, Nghiem HV, Johnson TD, Carlos RC, et al. (2004) T2- weighted MR imaging in the assessment of cirrhotic liver. Radiology 230: 637-644.
18. Westphalen AC, Coakley FV, Qayyum A, Swanson M, Simko JP, et al. (2008) Peripheral zone prostate cancer: accuracy of different interpretative approaches with MR and MR spectroscopic imaging. Radiology 246: 177-184.
19. Baker ME, Bogoni L, Obuchowski NA, Dass C, Kendzierski RM, et al. (2007) Computer-aided detection of colorectal polyps: can it improve sensitivity of lessexperienced readers? Preliminary findings. Radiology 245: 140-149.
20. Petrick N, Haider M, Summers RM, Yeshwant SC, Brown L, et al. (2008) CT colonography with computer-aided detection as a second reader: observer performance study. Radiology 246: 148-156.
21. Obuchowski NA (1997) Nonparametric analysis of clustered ROC curve data. Biometrics 53: 567-578.
22. Kim MJ, Lim JS, Oh YT, Kim JH, Chung JJ, et al. (2004) Preoperative MRI of rectal cancer with and without rectal water filling: an intraindividual comparison. AJR Am J Roentgenol 182: 1469-1476.
23. Krupinski EA, Lubin J, Roehrig H, Johnson J, Nafziger J (2006) Using a human visual system model to optimize soft-copy mammography display: influence of veiling glare. Acad Radiol 13: 289-295.
24. Krzanowski WJ, Hand DJ (2009) ROC curves for continuous data: CRC/Chapman and Hall.
25. Harrington MB (1990) Some methodological questions concerning receiver operating characteristic (ROC) analysis as a method for assessing image quality in radiology. J Digit Imaging 3: 211-218.
26. Waymire E, Gupta VK (1981) The mathematical structure of rainfall representations: A review of the stochastic rainfall models. Water resources research 17: 1261-1272.
27. Swets JA (1986) Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance. Psychol Bull 99: 181-198.
28. Hanley JA (1988) The robustness of the "binormal" assumptions used in fitting ROC curves. Med Decis Making 8: 197-203.
29. Gur D, Rockette HE, Bandos AI (2007) "Binary" and "non-binary" detection tasks: are current performance measures optimal? Acad Radiol 14: 871-876.
30. Dorfman DD, Berbaum KS, Brandser EA (2000) A contaminated binormal model for ROC data: Part I. Some interesting examples of binormal degeneracy. Acad Radiol 7: 420-426.
31. Metz CE (1989) Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol 24: 234-245.
32. Agresti A (1999) Modelling ordered categorical data: recent advances and future challenges. Stat Med 18: 2191-2207.
33. Krupinski EA, Jiang Y (2008) Anniversary paper: evaluation of medical imaging systems. Med Phys 35: 645-659.
34. Gur D, Bandos AI, Rockette HE (2008) Comparing areas under receiver operating characteristic curves: potential impact of the "Last" experimentally measured operating point. Radiology 247: 12-15.
35. Jiang Y, Nishikawa RM, Schmidt RA, Metz CE (2006) Comparison of independent double readings and computer-aided diagnosis (CAD) for the diagnosis of breast calcifications. Acad Radiol 13: 84-94.
36. Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS (2008) Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 27: 157-172; discussion 207-112.
37. Pepe MS, Feng Z, Gu JW (2008) Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med 27: 173-181.
38. Obuchowski NA (2005) ROC analysis. AJR Am J Roentgenol 184: 364-372.
39. Centor RM (1991) Signal detectability: the use of ROC curves and their analyses. Med Decis Making 11: 102-106.
40. Beiden SV, Wagner RF, Campbell G (2000) Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis. Acad Radiol 7: 341-349.
41. Dorfman DD, Berbaum KS, Metz CE (1992) Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Invest Radiol 27: 723-731.
42. Bandos AI, Rockette HE, Gur D (2006) A permutation test for comparing ROC curves in multireader studies a multi-reader ROC, permutation test. Acad Radiol 13: 414-420.
43. Gallas BD (2006) One-shot estimate of MRMC variance: AUC. Acad Radiol 13: 353-362.