Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: Systematic review with a focus on quality of data reporting

PLoS ONE

Volumn 9, Issue 12, 2014, Pages

  Dendumrongsup, Thaworn ^a   Plumb, Andrew A ^b   Halligan, Steve ^b   Fanshawe, Thomas R ^c   Altman, Douglas G ^c   Mallett, Susan ^c  

^a PRINCE OF SONGKLA UNIVERSITY   (Thailand)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

AREA UNDER THE CURVE; AREA UNDER THE RECEIVER OPERATING CURVE; ARTICLE; COMPUTER ASSISTED TOMOGRAPHY; DIAGNOSTIC ACCURACY; DIAGNOSTIC IMAGING; DIAGNOSTIC TEST; DIAGNOSTIC TEST ACCURACY STUDY; ECHOGRAPHY; HUMAN; INTERMETHOD COMPARISON; MAMMOGRAPHY; NUCLEAR MAGNETIC RESONANCE IMAGING; OUTCOME ASSESSMENT; RECEIVER OPERATING CHARACTERISTIC; STATISTICAL ANALYSIS; SYSTEMATIC REVIEW; METHODOLOGY; PROCEDURES; REPRODUCIBILITY; SENSITIVITY AND SPECIFICITY;

DIAGNOSTIC TESTS, ROUTINE; HUMANS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; ROC CURVE; SENSITIVITY AND SPECIFICITY;

EID: 84919884411     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0116018     Document Type: Article

References (92)

1. Wagner RF, Metz CE, Campbell G (2007) Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol 14: 723-748.
2. Obuchowski NA (2003) Receiver operating characteristic curves and their use in radiology. Radiology 229: 3-8.
3. Obuchowski NA (2005) ROC analysis. AJR Am J Roentgenol 184: 364-372.
4. Obuchowski NA, Beiden SV, Berbaum KS, Hillis SL, Ishwaran H, et al. (2004) Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol 11: 980-995.
5. Gallas BD, Chan HP, D'Orsi CJ, Dodd LE, Giger ML, et al. (2012) Evaluating imaging and computeraided detection and diagnosis devices at the FDA. Acad Radiol 19: 463-477.
6. Halligan S, Altman DG, Mallett S, Taylor SA, Burling D, et al. (2006) Computed tomographic colonography: assessment of radiologist performance with and without computer-aided detection. Gastroenterology 131: 1690-1699.
7. Mallett S, Halligan S, Collins GS, Altman DG (2014) Exploration of analysis methods for diagnostic imaging tests: Problems woth ROC AUC and confidence scores in CT colonography. PLoS One (in press).
8. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6: e1000097.
9. Obuchowski NA, Rockette HE (1995) Hypothesis testing of diagnostic accuracy for multiple observers and multiple tests: an ANOVA approach with dependent observations. Comm Stat 24: 934-936.
10. Obuchowski NA (1995) Multireader receiver operating characteristic studies: a comparison of study designs. Acad Radiol 2: 709-716.
11. Dorfman DD, Berbaum KS, Metz CE (1992) Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Invest Radiol 27: 723-731.
12. Dorfman DD, Berbaum KS, Lenth RV, Chen YF, Donaghy BA (1998) Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design. Acad Radiol 5: 591-602.
13. Hillis SL, Berbaum KS (2005) Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification. Acad Radiol 12: 1534-1541.
14. Hillis SL, Berbaum KS (2004) Power estimation for the Dorfman-Berbaum-Metz method. Acad Radiol 11: 1260-1273.
15. Hillis SL, Obuchowski NA, Schartz KM, Berbaum KS (2005) A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data. Stat Med 24: 1579-1607.
16. Hillis SL (2007) A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Stat Med 26: 596-619.
17. Warren LM, Mackenzie A, Cooke J, Given-Wilson RM, Wallis MG, et al. (2012) Effect of image quality on calcification detection in digital mammography. Med Phys 39: 3202-3213.
18. Berbaum KS, Schartz KM, Caldwell RT, El-Khoury GY, Ohashi K, et al. (2012) Satisfaction of search for subtle skeletal fractures may not be induced by more serious skeletal injury. J Am Coll Radiol 9: 344-351.
19. Destounis S, Somerville P, Murphy P, Seifert P (2011) Perceived sufficiency of full-field digital mammograms with and without irreversible image data compression for comparison with next-year mammograms. J Digit Imaging 24: 66-74.
20. Jinzaki M, Matsumoto K, Kikuchi E, Sato K, Horiguchi Y, et al. (2011) Comparison of CT urography and excretory urography in the detection and localization of urothelial carcinoma of the upper urinary tract. AJR Am J Roentgenol 196: 1102-1109.
21. Krupinski EA, Silverstein LD, Hashmi SF, Graham AR, Weinstein RS, et al. (2012) Observer performance using virtual pathology slides: impact of LCD color reproduction accuracy. J Digit Imaging 25: 738-743.
22. Leong DL, Rainford L, Haygood TM, Whitman GJ, Tchou PM, et al. (2012) Verification of DICOM GSDF in complex backgrounds. J Digit Imaging 25: 662-669.
23. Nishida K, Yuen S, Kamoi K, Yamada K, Akazawa K, et al. (2011) Incremental value of T2-weighted and diffusion-weighted MRI for prediction of biochemical recurrence after radical prostatectomy in clinically localized prostate cancer. Acta Radiol 52: 120-126.
24. Obuchowski NA, Meziane M, Dachman AH, Lieber ML, Mazzone PJ (2010) What's the control in studies measuring the effect of computer-aided detection (CAD) on observer performance? Acad Radiol 17: 761-767.
25. Okamoto S, Shiga T, Hattori N, Kubo N, Takei T, et al. (2011) Semiquantitative analysis of C-11 methionine PET may distinguish brain tumor recurrence from radiation necrosis even in small lesions. Ann Nucl Med 25: 213-220.
26. Reed WM, Ryan JT, McEntee MF, Evanoff MG, Brennan PC (2011) The effect of abnormalityprevalence expectation on expert observer performance and visual search. Radiology 258: 938-943.
27. Svane G, Azavedo E, Lindman K, Urech M, Nilsson J, et al. (2011) Clinical experience of photon counting breast tomosynthesis: comparison with traditional mammography. Acta Radiol 52: 134-142.
28. Szucs-Farkas Z, Kaelin I, Flach PM, Rosskopf A, Ruder TD, et al. (2010) Detection of chest trauma with whole-body low-dose linear slit digital radiography: a multireader study. AJR Am J Roentgenol 194: W388-395.
29. Webb LJ, Samei E, Lo JY, Baker JA, Ghate SV, et al. (2011) Comparative performance of multiview stereoscopic and mammographic display modalities for breast lesion detection. Med Phys 38: 1972-1980.
30. Yakabe M, Sakai S, Yabuuchi H, Matsuo Y, Kamitani T, et al. (2010) Effect of dose reduction on the ability of digital mammography to detect simulated microcalcifications. J Digit Imaging 23: 520-526.
31. Zanca F, Hillis SL, Claus F, Van Ongeval C, Celis V, et al. (2012) Correlation of free-response and receiver-operating-characteristic area-under-the-curve estimates: results from independently conducted FROCROC studies in mammography. Med Phys 39: 5917-5929.
32. Aoki T, Oda N, Yamashita Y, Yamamoto K, Korogi Y (2011) Usefulness of computerized method for lung nodule detection in digital chest radiographs using temporal subtraction images. Acad Radiol 18: 1000-1005.
33. Aoki T, Oda N, Yamashita Y, Yamamoto K, Korogi Y (2012) Usefulness of computerized method for lung nodule detection on digital chest radiographs using similar subtraction images from different patients. Eur J Radiol 81: 1062-1067.
34. Berg WA, Madsen KS, Schilling K, Tartar M, Pisano ED, et al. (2012) Comparative effectiveness of positron emission mammography and MRI in the contralateral breast of women with newly diagnosed breast cancer. AJR Am J Roentgenol 198: 219-232.
35. Bilello M, Suri N, Krejza J, Woo JH, Bagley LJ, et al. (2010) An approach to comparing accuracies of two FLAIR MR sequences in the detection of multiple sclerosis lesions in the brain in the absence of gold standard. Acad Radiol 17: 686-695.
36. Choi HJ, Lee JH, Kang BS (2012) Remote CT reading using an ultramobile PC and web-based remote viewing over a wireless network. J Telemed Telecare 18: 26-31.
37. Cole EB, Toledano AY, Lundqvist M, Pisano ED (2012) Comparison of radiologist performance with photon-counting full-field digital mammography to conventional full-field digital mammography. Acad Radiol 19: 916-922.
38. Collettini F, Martin JC, Diekmann F, Fallenberg E, Engelken F, et al. (2012) Diagnostic performance of a Near-Infrared Breast Imaging system as adjunct to mammography versus X-ray mammography alone. Eur Radiol 22: 350-357.
39. Dachman AH, Obuchowski NA, Hoffmeister JW, Hinshaw JL, Frew MI, et al. (2010) Effect of computer-aided detection for CTcolonography in a multireader, multicase trial. Radiology 256: 827-835.
40. Dromain C, Thibault F, Diekmann F, Fallenberg EM, Jong RA, et al. (2012) Dual-energy contrastenhanced digital mammography: initial clinical results of a multireader, multicase study. Breast Cancer Res 14: R94.
41. Gennaro G, Toledano A, di Maggio C, Baldan E, Bezzon E, et al. (2010) Digital breast tomosynthesis versus digital mammography: a clinical performance study. Eur Radiol 20: 1545-1553.
42. Hupse R, Samulski M, Lobbes MB, Mann RM, Mus R, et al. (2013) Computer-aided detection of masses at mammography: interactive decision support versus prompts. Radiology 266: 123-129.
43. Kelly KM, Dean J, Lee SJ, Comulada WS (2010) Breast cancer detection: radiologists' performance using mammography with and without automated whole-breast ultrasound. Eur Radiol 20: 2557-2564.
44. Kim H, Choi D, Lee JH, Lee SJ, Jo H, et al. (2012) High-risk esophageal varices in patients treated with locoregional therapy for hepatocellular carcinoma: assessment with liver computed tomography. World J Gastroenterol 18: 4905-4911.
45. Kim S, Yoon CS, Ryu JA, Lee S, Park YS, et al. (2010) A comparison of the diagnostic performances of visceral organ-targeted versus spine-targeted protocols for the evaluation of spinal fractures using sixteen-channel multidetector row computed tomography: is additional spine-targeted computed tomography necessary to evaluate thoracolumbar spinal fractures in blunt trauma victims? J Trauma 69: 437-446.
46. Li F, Engelmann R, Pesce L, Armato SG 3rd, Macmahon H (2012) Improved detection of focal pneumonia by chest radiography with bone suppression imaging. Eur Radiol 22: 2729-2735.
47. Li F, Engelmann R, Pesce LL, Doi K, Metz CE, et al. (2011) Small lung cancers: improved detection by use of bone suppression imaging-comparison with dual-energy subtraction chest radiography. Radiology 261: 937-949.
48. Li F, Hara T, Shiraishi J, Engelmann R, MacMahon H, et al. (2011) Improved detection of subtle lung nodules by use of chest radiographs with bone suppression imaging: receiver operating characteristic analysis with and without localization. AJR Am J Roentgenol 196: W535-541.
49. Matsushima M, Naganawa S, Ikeda M, Itoh S, Ogawa H, et al. (2010) Diagnostic value of SPIOmediated breath-hold, black-blood, fluid-attenuated, inversion recovery (BH-BB-FLAIR) imaging in patients with hepatocellular carcinomas. Magn Reson Med Sci 9: 49-58.
50. McNulty JP, Ryan JT, Evanoff MG, Rainford LA (2012) Flexible image evaluation: iPad versus secondary-class monitors for review of MR spinal emergency cases, a comparative study. Acad Radiol 19: 1023-1028.
51. Medved M, Fan X, Abe H, Newstead GM, Wood AM, et al. (2011) Non-contrast enhanced MRI for evaluation of breast lesions: comparison of non-contrast enhanced high spectral and spatial resolution (HiSS) images versus contrast enhanced fat-suppressed images. Acad Radiol 18: 1467-1474.
52. Mermuys K, De Geeter F, Bacher K, Van De Moortele K, Coenegrachts K, et al. (2010) Digital tomosynthesis in the detection of urolithiasis: Diagnostic performance and dosimetry compared with digital radiography with MDCT as the reference standard. AJR Am J Roentgenol 195: 161-167.
53. Moin P, Deshpande R, Sayre J, Messer E, Gupte S, et al. (2011) An observer study for a computeraided reading protocol (CARP) in the screening environment for digital mammography. Acad Radiol 18: 1420-1429.
54. Muramatsu C, Schmidt RA, Shiraishi J, Li Q, Doi K (2010) Presentation of similar images as a reference for distinction between benign and malignant masses on mammograms: analysis of initial observer study. J Digit Imaging 23: 592-602.
55. Noroozian M, Hadjiiski L, Rahnama-Moghadam S, Klein KA, Jeffries DO, et al. (2012) Digital breast tomosynthesis is comparable to mammographic spot views for mass characterization. Radiology 262: 61-68.
56. Ohgiya Y, Suyama J, Seino N, Hashizume T, Kawahara M, et al. (2012) Diagnostic accuracy of ultra-high- b-value 3.0-T diffusion-weighted MR imaging for detection of prostate cancer. Clin Imaging 36: 526-531.
57. Otani H, Nitta N, Ikeda M, Nagatani Y, Tanaka T, et al. (2012) Flat-panel detector computed tomography imaging: observer performance in detecting pulmonary nodules in comparison with conventional chest radiography and multidetector computed tomography. J Thorac Imaging 27: 51-57.
58. Padilla F, Roubidoux MA, Paramagul C, Sinha SP, Goodsitt MM, et al. (2013) Breast mass characterization using 3-dimensional automated ultrasound as an adjunct to digital breast tomosynthesis: a pilot study. J Ultrasound Med 32: 93-104.
59. Pollard BJ, Samei E, Chawla AS, Beam C, Heyneman LE, et al. (2012) The effects of ambient lighting in chest radiology reading rooms. J Digit Imaging 25: 520-526.
60. Purysko AS, Remer EM, Coppa CP, Obuchowski NA, Schneider E, et al. (2012) Characteristics and distinguishing features of hepatocellular adenoma and focal nodular hyperplasia on gadoxetate disodium-enhanced MRI. AJR Am J Roentgenol 198: 115-123.
61. Rafferty EA, Park JM, Philpotts LE, Poplack SP, Sumkin JH, et al. (2013) Assessing radiologist performance using combined digital mammography and breast tomosynthesis compared with digital mammography alone: results of a multicenter, multireader trial. Radiology 266: 104-113.
62. Saade C, Bourne R, Wilkinson M, Evanoff M, Brennan P (2013) A reduced contrast volume acquisition regimen based on cardiovascular dynamics improves visualisation of head and neck vasculature with carotid MDCT angiography. Eur J Radiol 82: e64-69.
63. Salazar AJ, Camacho JC, Aguirre DA (2011) Comparison between differently priced devices for digital capture of X-ray films using computed tomography as a gold standard: a multireader-multicase receiver operating characteristic curve study. Telemed J E Health 17: 275-282.
64. Shimauchi A, Giger ML, Bhooshan N, Lan L, Pesce LL, et al. (2011) Evaluation of clinical breast MR imaging performed with prototype computer-aided diagnosis breast MR imaging workstation: reader study. Radiology 258: 696-704.
65. Shiraishi J, Abe H, Ichikawa K, Schmidt RA, Doi K (2010) Observer study for evaluating potential utility of a super-high-resolution LCD in the detection of clustered microcalcifications on digital mammograms. J Digit Imaging 23: 161-169.
66. Subhas N, Kao A, Freire M, Polster JM, Obuchowski NA, et al. (2011) MRI of the knee ligaments and menisci: comparison of isotropic-resolution 3D and conventional 2D fast spin-echo sequences at 3 T. AJR Am J Roentgenol 197: 442-450.
67. Sung YM, Chung MJ, Lee KS, Choe BK (2010) The influence of liquid crystal display monitors on observer performance for the detection of interstitial lung markings on both storage phosphor and flat-panel- detector chest radiography. Eur J Radiol 74: 275-279.
68. Svahn TM, Chakraborty DP, Ikeda D, Zackrisson S, Do Y, et al. (2012) Breast tomosynthesis and digital mammography: a comparison of diagnostic accuracy. Br J Radiol 85: e1074-1082.
69. Takahashi N, Tsai DY, Lee Y, Kinoshita T, Ishii K, et al. (2010) Usefulness of z-score mapping for quantification of extent of hypoattenuation regions of hyperacute stroke in unenhanced computed tomography: analysis of radiologists' performance. J Comput Assist Tomogr 34: 751-756.
70. Tan T, Platel B, Huisman H, Sanchez CI, Mus R, et al. (2012) Computer-aided lesion diagnosis in automated 3-D breast ultrasound using coronal spiculation. IEEE Trans Med Imaging 31: 1034-1042.
71. Timp S, Varela C, Karssemeijer N (2010) Computer-aided diagnosis with temporal analysis to improve radiologists' interpretation of mammographic mass lesions. IEEE Trans Inf Technol Biomed 14: 803-808.
72. Toomey RJ, Ryan JT, McEntee MF, Evanoff MG, Chakraborty DP, et al. (2010) Diagnostic efficacy of handheld devices for emergency radiologic consultation. AJR Am J Roentgenol 194: 469-474.
73. Uchiyama Y, Asano T, Kato H, Hara T, Kanematsu M, et al. (2012) Computer-aided diagnosis for detection of lacunar infarcts on MR images: ROC analysis of radiologists' performance. J Digit Imaging 25: 497-503.
74. Visser R, Veldkamp WJ, Beijerinck D, Bun PA, Deurenberg JJ, et al. (2012) Increase in perceived case suspiciousness due to local contrast optimisation in digital screening mammography. Eur Radiol 22: 908-914.
75. Wallis MG, Moa E, Zanca F, Leifland K, Danielsson M (2012) Two-view and single-view tomosynthesis versus full-field digital mammography: high-resolution X-ray imaging observer study. Radiology 262: 788-796.
76. Wardlaw JM, von Kummer R, Farrall AJ, Chappell FM, Hill M, et al. (2010) A large web-based observer reliability study of early ischaemic signs on computed tomography. The Acute Cerebral CT Evaluation of Stroke Study (ACCESS). PLoS One 5: e15757.
77. Way T, Chan HP, Hadjiiski L, Sahiner B, Chughtai A, et al. (2010) Computer-aided diagnosis of lung nodules on CT scans: ROC study of its effect on radiologists' performance. Acad Radiol 17: 323-332.
78. Yamada Y, Jinzaki M, Hasegawa I, Shiomi E, Sugiura H, et al. (2011) Fast scanning tomosynthesis for the detection of pulmonary nodules: diagnostic performance compared with chest radiography, using multidetector-row computed tomography as the reference. Invest Radiol 46: 471-477.
79. Yamada Y, Mori H, Hijiya N, Matsumoto S, Takaji R, et al. (2012) Extrahepatic bile duct cancer: invasion of the posterior hepatic plexuses-evaluation using multidetector CT. Radiology 263: 419-428.
80. Yoshida A, Tha KK, Fujima N, Zaitsu Y, Yoshida D, et al. (2013) Detection of brain metastases by 3-dimensional magnetic resonance imaging at 3 T: comparison between T1-weighted volume isotropic turbo spin echo acquisition and 3-dimensional T1-weighted fluid-attenuated inversion recovery imaging. J Comput Assist Tomogr 37: 84-90.
81. v2.1 DMs. Available: http://www-radiology.uchicago.edu/krl/KRL-ROC/software-index6.htm.
82. Gur D, Rockette HE, Bandos AI (2007) "Binary" and "non-binary" detection tasks: are current performance measures optimal? Acad Radiol 14: 871-876.
83. Dorfman DD, Berbaum KS, Brandser EA (2000) A contaminated binormal model for ROC data: Part I. Some interesting examples of binormal degeneracy. Acad Radiol 7: 420-426.
84. Baker ME, Bogoni L, Obuchowski NA, Dass C, Kendzierski RM, et al. (2007) Computer-aided detection of colorectal polyps: can it improve sensitivity of less-experienced readers? Preliminary findings. Radiology 245: 140-149.
85. Zhou XH, Obuchowski N, McClish DK (2002) Statistical methods in diagnostic medicine. New York NY: Wiley.
86. Mallett S, Halligan S, Collins GS, Altman DG (2014) Exploration of Analysis Methods for Diagnostic Imaging Tests: Problems with ROC AUC and Confidence Scores in CT Colonography. PLoS One 9: e107633.
87. Shiraishi J, Pesce LL, Metz CE, Doi K (2009) Experimental design and data analysis in receiver operating characteristic studies: lessons learned from reports in radiology from 1997 to 2006. Radiology 253: 822-830.
88. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, et al. (2003) Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative. Radiology 226: 24-28.
89. Harrington MB (1990) Some methodological questions concerning receiver operating characteristic (ROC) analysis as a method for assessing image quality in radiology. J Digit Imaging 3: 211-218.
90. Petrick N, Haider M, Summers RM, Yeshwant SC, Brown L, et al. (2008) CT colonography with computer-aided detection as a second reader: observer performance study. Radiology 246: 148-156.
91. Hanley JA (1988) The robustness of the "binormal" assumptions used in fitting ROC curves. Med Decis Making 8: 197-203.
92. Gur D, Bandos AI, Rockette HE (2008) Comparing areas under receiver operating characteristic curves: potential impact of the "Last" experimentally measured operating point. Radiology 247: 12-15.