Correlation of free-response and receiver-operating-characteristic area-under-the-curve estimates: Results from independently conducted FROCROC studies in mammography

Medical Physics

Volumn 39, Issue 10, 2012, Pages 5917-5929

  Zanca, Federica ^a   Hillis, Stephen L ^b,c   Claus, Filip ^a   Van Ongeval, Chantal ^a   Celis, Valerie ^a   Provoost, Veerle ^a   Yoon, Hong Jun ^d   Bosmans, Hilde ^a  

^a UNIVERSITY HOSPITALS LEUVEN   (Belgium)

^b UNIVERSITY OF IOWA   (United States)

^c VETERANS AFFAIRS MEDICAL CENTER   (United States)

^d UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

Author keywords

AFROC; AUC estimates correlations; ROC

Indexed keywords

CORRELATION METHODS;

AFROC; AREA UNDER THE CURVES; AREA UNDER THE ROC CURVE; AUC ESTIMATES CORRELATIONS; DEGREE OF ASSOCIATION; FREE-RESPONSE RECEIVERS; IMAGE PROCESSING ALGORITHM; RECEIVER OPERATING CHARACTERISTICS;

IMAGE PROCESSING;

EID: 84867288387     PISSN: 00942405     EISSN: None     Source Type: Journal    
DOI: 10.1118/1.4747262     Document Type: Article

References (63)

1. H. Kundel, K. Berbaum, D. Dorfman, D. Gur, R. Swensson, and C. Metz, Receiver operating characteristic analysis in medical imaging., J. ICRU 8 (1), Report 79 (2008).
2. E. Samei and E. A. Krupinski, The Handbook of Medical Image Perception and Techniques (Cambridge University Press, Cambridge, New York, 2010).
3. D. K. McClish, Analyzing a portion of the ROC curve., Med. Decis Making 9, 190-195 (1989). 10.1177/0272989X8900900307
4. Y. Jiang, C. E. Metz, and R. M. Nishikawa, A receiver operating characteristic partial area index for highly sensitive diagnostic tests., Radiology 201, 745-750 (1996).
5. S. V. Beiden, R. F. Wagner, and G. Campbell, Components-of-variance models and multiple-bootstrap experiments: An alternative method for random-effects, receiver operating characteristic analysis., Acad. Radiol. 7, 341-349 (2000). 10.1016/S1076-6332(00)80008-2
6. S. V. Beiden, R. F. Wagner, G. Campbell, and H. P. Chan, Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis., Acad. Radiol. 8, 616-622 (2001). 10.1016/S1076-6332(03)80686-4
7. S. V. Beiden, R. F. Wagner, G. Campbell, C. E. Metz, and Y. Jiang, Components-of-variance models for random-effects ROC analysis: The case of unequal variance structures across modalities., Acad. Radiol. 8, 605-615 (2001). 10.1016/S1076-6332(03)80685-2
8. D. D. Dorfman, K. S. Berbaum, C. E. Metz, R. V. Lenth, J. A. Hanley, and D. H. Abu, Proper receiver operating characteristic analysis: The bigamma model., Acad. Radiol. 4, 138-149 (1997). 10.1016/S1076-6332(97)80013-X
9. D. D. Dorfman, K. S. Berbaum, R. V. Lenth, Y. F. Chen, and B. A. Donaghy, Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: Factorial experimental design., Acad. Radiol. 5, 591-602 (1998). 10.1016/S1076-6332(98)80294-8
10. H. Ishwaran and A. C. Gatsonis, A general class of hierarchical ordinal regression models with applications to correlated ROC analysis., Can. J. Stat. 28 (4), 731-750 (2000). 10.2307/3315913
11. N. A. Obuchowski and H. E. Rockette Jr., Hypothesis testing of the diagnostic accuracy for multiple diagnostic tests: An ANOVA approach with dependent observations., Commun. Stat.-Simul. Comput. 24, 285-308 (1995). 10.1080/03610919508813243
12. N. A. Obuchowski, Multireader, multimodality receiver operating characteristic curve studies: Hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations., Acad. Radiol. 2 (Suppl. 1), S22-S29 (1995). 10.1016/S1076-6332(05)80441-6
13. D. D. Dorfman, K. S. Berbaum, and C. E. Metz, Receiver operating characteristic rating analysis: Generalization to the population of readers and patients with the jackknife method., Invest. Radiol. 27, 723-731 (1992). 10.1097/00004424-199209000-00015
14. K. S. Berbaum, K. M. Schartz, and S. L. Hillis (2012). OR/DBM MRMC (Version 2.4) [Computer software]. Iowa City, IA: The University of Iowa. Available from http://perception.radiology.uiowa.edu.
15. S. L. Hillis, N. A. Obuchowski, K. M. Schartz, and K. S. Berbaum, A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data., Stat. Med. 24, 1579-1607 (2005). 10.1002/sim.2024
16. S. L. Hillis, A comparison of denominator degrees of freedom methods for multiple observer ROC analysis., Stat. Med. 26, 596-619 (2007). 10.1002/sim.2532
17. S. L. Hillis, K. S. Berbaum, and C. E. Metz, Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis., Acad. Radiol. 15, 647-661 (2008). 10.1016/j.acra.2007.12.015
18. P. C. Bunch, J. F. Hamilton, G. K. Sanderson, and A. H. Simmons, A free-response approach to the measurement and characterization of radiographic-observer performance., J. Appl. Photogr. Eng. 4, 166-171 (1978). 10.1117/12.955926
19. D. P. Chakraborty and K. S. Berbaum, Observer studies involving detection and localization: Modeling, analysis, and validation., Med. Phys. 31, 2313-2330 (2004). 10.1118/1.1769352
20. R. G. Swensson, Unified measurement of observer performance in detecting and localizing target objects on images., Med. Phys. 23, 1709-1725 (1996). 10.1118/1.597758
21. S. Paquerault, F. W. Smelson, N. Petrick, and K. J. Myers, Comparing signal-based and case-based methodologies for CAD assessment in a detection task., Proc. SPIE 6917, pp. 691708-691708-9 (2008). 10.1117/12.771498
22. S. Paquerault, F. W. Samuelson, K. J. Myers, and R. C. Smith, Non-localization and localization ROC analyses using clinically based scoring., Proc. SPIE 7263, 72630U (2009). 10.1117/12.813761
23. F. W. Samuelson and N. Petrick, Comparing image detection algorithms using resampling., in Proceedings of the International Symposium on Biomedical Imaging: From Nano to Micro, Arlington, VA (IEEE, 2006), pp. 1312-1315.
24. D. P. Chakraborty, jafroc software v3f, 2010 (available URL: www.devchakraborty.com).
25. D. P. Chakraborty, ROC curves predicted by a model of visual search., Phys. Med. Biol. 51, 3463-3482 (2006). 10.1088/0031-9155/51/14/013
26. D. P. Chakraborty, A search model and figure of merit for observer data acquired according to the free-response paradigm., Phys. Med. Biol. 51, 3449-3462 (2006). 10.1088/0031-9155/51/14/012
27. D. P. Chakraborty and H. J. Yoon, Operating characteristics predicted by models for diagnostic tasks involving lesion localization., Med. Phys. 35, 435-445 (2008). 10.1118/1.2820902
28. P. C. Brennan, The impact of acoustic noise found within clinical departments on radiology performance., Acad. Radiol. 15, 472-476 (2008). 10.1016/j.acra.2007.12.005
29. M. Bruegel, Diagnosis of hepatic metastasis: Comparison of respiration-triggered diffusion-weighted echo-planar MRI and five T2-weighted turbo spin-echo sequences., Am. J. Roentgenol. 191, 1421-1429 (2008). 10.2214/AJR.07.3279
30. V. A. Fisichella, Computer-aided detection (CAD) as a second reader using perspective filet view at CT colonography: Effect on performance of inexperienced readers., Clin. Radiol. 64, 972-982 (2009). 10.1016/j.crad.2009. 05.012
31. J. Ge, Computer-aided detection system for clustered microcalcifications: Comparison of performance on full-field digital mammograms and digitized screen-film mammograms., Phys. Med. Biol. 52, 981-1000 (2007). 10.1088/0031-9155/52/4/008
32. J. M. Goo, Is the computer-aided detection scheme for lung nodule also useful in detecting lung cancer, J. Comput. Assist. Tomogr. 32, 570-575 (2008). 10.1097/RCT.0b013e318146261c
33. W. F. Good, Digital breast tomosynthesis: A pilot observer study., Am. J. Roentgenol. 190, 865-869 (2008). 10.2214/AJR.07.2841
34. T. Hirose, N. Nitta, J. Shiraishi, Y. Nagatani, M. Takahashi, and K. Murata, Evaluation of computer-aided diagnosis (CAD) software for the detection of lung nodules on multidetector row computed tomography (MDCT): JAFROC study for the improvement in radiologists' diagnostic accuracy., Acad. Radiol. 15, 1505-1512 (2008). 10.1016/j.acra.2008.06.009
35. J. Nappi and H. Yoshida, Virtual tagging for laxative-free CT colonography: Pilot evaluation., Med. Phys. 36, 1830-1838 (2009). 10.1118/1.3113893
36. M. Penedo, Free-response receiver operating characteristic evaluation of lossy JPEG2000 and object-based set partitioning in hierarchical trees compression of digitized mammograms., Radiology 237, 450-457 (2005). 10.1148/radiol.2372040996
37. L. Pikus, Artificial multiple sclerosis lesions on simulated FLAIR brain MR images: Echo time and observer performance in detection., Radiology 239, 238-245 (2006). 10.1148/radiol.2383050211
38. M. Ruschin, Dose dependence of mass and microcalcification detection in digital mammography: Free response human observer studies., Med. Phys. 34, 400-407 (2007). 10.1118/1.2405324
39. J. Shiraishi, D. Appelbaum, Y. Pu, Q. Li, L. Pesce, and K. Doi, Usefulness of temporal subtraction images for identification of interval changes in successive whole-body bone scans: JAFROC analysis of radiologists' performance., Acad. Radiol. 14, 959-966 (2007). 10.1016/j.acra.2007.04.005
40. J. Vikgren, Comparison of chest tomosynthesis and chest radiography for detection of pulmonary nodules: Human observer study of clinical cases., Radiology 249, 1034-1041 (2008). 10.1148/radiol.2492080304
41. J. Wei, Dual system approach to computer-aided detection of breast masses on mammograms., Med. Phys. 33, 4157-4168 (2006). 10.1118/1.2357838
42. J. H. Woo, L. P. Henry, J. Krejza, and E. R. Melhem, Detection of simulated multiple sclerosis lesions on T2-weighted and FLAIR images of the brain: Observer performance., Radiology 241, 206-212 (2006). 10.1148/radiol.2411050792
43. M. Yanagawa, Commercially available computer-aided detection system for pulmonary nodules on thin-section images using 64 detectors-row CT: Preliminary study of 48 cases., Acad. Radiol. 16, 924-933 (2009). 10.1016/j.acra.2009.01.030
44. S. Zachrisson, Effect of clinical experience of chest tomosynthesis on detection of pulmonary nodules., Acta Radiol. 50, 884-891 (2009). 10.1080/02841850903085584
45. B. Zheng, D. P. Chakraborty, H. E. Rockette, G. S. Maitz, and D. Gur, A comparison of two data analyses from two observer performance studies using Jackknife ROC and JAFROC., Med. Phys. 32, 1031-1034 (2005). 10.1118/1.1884766
46. F. Zanca, Evaluation of clinical image processing algorithms used in digital in digital mammography., Med. Phys. 36, 765-775 (2009). 10.1118/1.3077121
47. A. Burgess and S. Chakraborty, Producing lesions for hybrid mammograms: Extracted tumors and simulated microcalcifications., Proc. SPIE 3663, 316-322 (1999). 10.1117/12.349659
48. F. Zanca, An improved method for simulating microcalcifications in digital mammograms., Med. Phys. 35, 4012-4018 (2008). 10.1118/1.2968334
49. J. Jacobs, F. Zanca, G. Marchal, and H. Bosmans, Implementation of a novel software framework for increased efficiency in observer performance studies in digital radiology. in Proceedings of the Scientific Assembly and Annual Meeting of the RSNA, Chicago, IL, 2008.
50. B. Efron, Jackknife-after-bootstrap standard errors and influence functions., J. R. Statist. Soc. B 54, 83-127 (1992).
51. H. Hartly and E. Pearson, Biometrika Tables for Statisticians, 3rd ed. (Cambridge University Press, New York, 1996), Vol. I.
52. C. E. Metz and J. H. Shen, Gains in accuracy from replicated readings of diagnostic images: Prediction and assessment in terms of ROC analysis., Med. Decis Making 12, 60-75 (1992). 10.1177/0272989X9201200110
53. D. Gur, Agreement of the order of overall performance levels under different reading paradigms., Acad. Radiol. 15, 1567-1573 (2008). 10.1016/j.acra.2008.07.011
54. G. Coppini, S. Diciotti, M. Falchini, N. Villari, and G. Valli, Neural networks for computer-aided diagnosis: Detection of lung nodules in chest radiograms., IEEE Trans. Inf. Technol. Biomed. 7, 344-357 (2003). 10.1109/TITB.2003.821313
55. A. Oliver, A review of automatic mass detection and segmentation in mammographic images., Med. Image Anal. 14, 87-110 (2010). 10.1016/j.media.2009. 12.005
56. L. Volokh, C. Liu, and B. M. W. Tsui, Exploring FROC paradigm: Initial experience with clinical applications., Proc. SPIE 6146, 94-105 (2006). 10.1117/12.653247
57. A. S. Chawla, E. Samei, R. S. Saunders, J. Y. Lo, and J. A. Baker, A mathematical model platform for optimizing a multiprojection breast imaging system., Med. Phys. 35, 1337-1345 (2008). 10.1118/1.2885367
58. N. T. Ranger, J. Y. Lo, and E. Samei, A technique optimization protocol and the potential for dose reduction in digital mammography., Med. Phys. 37, 962-969 (2010). 10.1118/1.3276732
59. S. Richard and E. Samei, Quantitative imaging in breast tomosynthesis and CT: Comparison of detection and estimation task performance., Med. Phys. 37, 2627-2637 (2010). 10.1118/1.3429025
60. E. Samei, R. S. Saunders Jr., J. A. Baker, and D. M. DeLong, Digital mammography: Effects of reduced radiation dose on diagnostic performance., Radiology 243, 396-404 (2007). 10.1148/radiol.2432061065
61. R. Saunders, E. Samei, J. Baker, and D. Delong, Simulation of mammographic lesions., Acad. Radiol. 13, 860-870 (2006). 10.1016/j.acra.2006.03. 015
62. C. M. Shafer, E. Samei, and J. Y. Lo, The quantitative potential for breast tomosynthesis imaging., Med. Phys. 37, 1004-1016 (2010). 10.1118/1.3285038
63. D. Gur, The laboratory effect: Comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations., Radiology 249, 47-53 (2008). 10.1148/radiol.2491072025