Evaluating Imaging and Computer-aided Detection and Diagnosis Devices at the FDA

Academic Radiology

Volumn 19, Issue 4, 2012, Pages 463-477

  Gallas, Brandon D ^a   Chan, Heang Ping ^b   D'Orsi, Carl J ^c   Dodd, Lori E ^d   Giger, Maryellen L ^e   Gur, David ^f   Krupinski, Elizabeth A ^g   Metz, Charles E ^e   Myers, Kyle J ^a   Obuchowski, Nancy A ^h   Sahiner, Berkman ^a   Toledano, Alicia Y ⁱ   Zuley, Margarita L ^f  

^a CENTER FOR DEVICES AND RADIOLOGICAL HEALTH   (United States)

^b UNIVERSITY OF MICHIGAN MEDICAL SCHOOL   (United States)

^c EMORY UNIVERSITY   (United States)

^d NATIONAL INSTITUTE OF ALLERGY AND INFECTIOUS DISEASES   (United States)

^e UNIVERSITY OF CHICAGO   (United States)

^f UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

^g UNIVERSITY OF ARIZONA   (United States)

^h LERNER RESEARCH INSTITUTE   (United States)

ⁱ University of Iowa   (United States)

Author keywords

Consensus; Designs; Methods; Postmarket; Premarket; Reader studies; ROC

Indexed keywords

COMPUTER ASSISTED RADIOGRAPHY; DIAGNOSTIC ACCURACY; FOOD AND DRUG ADMINISTRATION; IMAGE ANALYSIS; IMAGING; MEDICAL DECISION MAKING; MEDICAL DEVICE; POSTMARKETING SURVEILLANCE; PRIORITY JOURNAL; REVIEW; SENSITIVITY AND SENSIBILITY;

DEVICE APPROVAL; IMAGE INTERPRETATION, COMPUTER-ASSISTED; TECHNOLOGY ASSESSMENT, BIOMEDICAL; UNITED STATES; UNITED STATES FOOD AND DRUG ADMINISTRATION;

EID: 84858786052     PISSN: 10766332     EISSN: 18784046     Source Type: Journal    
DOI: 10.1016/j.acra.2011.12.016     Document Type: Review

References (114)

1. Determination of safety and effectiveness. 21 CFR § 860 (2011).
2. Premarket approval application. 21 CFR § 814 (2011).
3. US Food and Drug Administration. Radiological Devices Panel meeting: computer-aided detection devices. Available at: . Accessed November 17, 2011. http://www.fda.gov/ohrms/dockets/ac/08/minutes/2008%134349m1.pdf.
4. Breast Imaging Reporting and Data System (BI-RADS) 2003, American College of Radiology, American College of Radiology, Reston, VA.
5. Metz C.E. ROC methodology in radiologic imaging. Invest Radiol 1986, 21:720-733.
6. Green D.M., Swets J.A. Signal detection theory and psychophysics 1966, John Wiley, New York.
7. Lusted L.B. Introduction to medical decision making 1968, Charles C. Thomas, Springfield, IL.
8. McNeil B.J., Keller E., Adelstein S.J. Primer on certain elements of medical decision making. N Engl J Med 1975, 293:211-215.
9. Metz C.E. Basic principles of ROC analysis. Semin Nucl Med 1978, 8:283-298.
10. Patton D.D. Introduction to clinical decision making. Semin Nucl Med 1978, 8:273-282.
11. Weinstein M.C., Fineberg H.V. Clinical decision analysis 1980, Saunders, Philadelphia, PA.
12. Hanley J.A., McNeil B.J. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982, 143:29-36.
13. Metz C.E. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol 1989, 24:234-245.
14. Gur D., Bandos A.I., Rockette H.E. Comparing areas under receiver operating characteristic curves: potential impact of the " last" experimentally measured operating point. Radiology 2008, 247:12-15.
15. Pesce L.L., Metz C.E., Berbaum K.S. On the convexity of ROC curves estimated from radiological test results. Acad Radiol 2010, 17:960-968.
16. Jiang Y., Metz C.E. BI-RADS data should not be used to estimate ROC curves. Radiology 2010, 256:29-31.
17. Metz C.E. Fundamental ROC analysis. Handbook of medical imaging, vol. 1: physics and psychophysics 2000, 751-769. SPIE Press, Bellingham, WA. J. Beutel, H.L. Kundel, R.L. Van Metter (Eds.).
18. Obuchowski N.A. ROC analysis. AJR Am J Roentgenol 2005, 184:364-372.
19. Receiver operating characteristic analysis in medical imaging. ICRU Report 79 2008, International Commission of Radiation Units and Measurements, International Commission of Radiation Units and Measurements, Bethesda, MD.
20. Elmore J.G., Wells C.K., Lee C.H., et al. Variability in radiologists' interpretations of mammograms. N Engl J Med 1994, 331:1493-1499.
21. Beam C., Layde P.M., Sullivan D.C. Variability in the interpretation of screening mammograms by US radiologists. Arch Intern Med 1996, 156:209-213.
22. D'Orsi C.J., Swets J.A. Variability in the interpretation of mammograms. N Engl J Med 1995, 332:1172.
23. Gallas B.D., Pennello G.A., Myers K.J. Multi-reader multi-case variance analysis for binary data. J Opt Soc Am A 2007, 24:B70-B80.
24. Gallas B.D., Brown D.G. Reader studies for validation of CAD systems. Neural Networks 2008, 21:387-397.
25. Obuchowski N.A. Reducing the number of reader interpretations in MRMC studies. Acad Radiol 2009, 16:209-217.
26. University of Chicago. Metz ROC software. Available at: . Accessed November 17, 2011. http://metz-roc.uchicago.edu.
27. University of Iowa Medical Image Perception Laboratory. Home page. Available at: . Accessed November 17, 2011. http://perception.radiology.uiowa.edu.
28. Cleveland Clinic, Quantitative Health Sciences. Research activities: ROC analysis. Available at: . Accessed November 17, 2011. http://www.bio.ri.ccf.org/html/rocanalysis.html.
29. Bankier A.A., Levine D., Halpern E.F., et al. Consensus interpretation in imaging research: is there a better way?. Radiology 2010, 257:14-17.
30. Hillman B.J. ACRIN-lessons learned in conducting multi-center trials of imaging and cancer. Cancer Imaging 2005, 5:S97-S101.
31. Roe C.A., Metz C.E. Variance-component modeling in the analysis of receiver operating characteristic (ROC) index estimates. Acad Radiol 1997, 4:587-600.
32. Rockette H.E., Campbell W.L., Britton C.A., et al. Empiric assessment of parameters that affect the design of multireader receiver operating characteristic studies. Acad Radiol 1999, 6:723-729.
33. Wagner R.F., Beiden S.V., Campbell G., et al. Assessment of medical imaging and computer-assist systems: lessons from recent experience. Acad Radiol 2002, 9:1264-1277.
34. Wagner R.F., Beam C.A., Beiden S.V. Reader variability in mammography and its implications for expected utility over the population of readers and cases. Med Decis Making 2004, 24:561-572.
35. Zhou X.-H., Obuchowski N.A., McClish D.K. Statistical methods in diagnostic medicine 2002, John Wiley, Hoboken, NJ.
36. Wagner R.F., Metz C.E., Campbell G. Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol 2007, 14:723-748.
37. Jiang Y., Miglioretti D.L., Metz C.E., et al. Breast cancer detection rate: designing imaging trials to demonstrate improvements. Radiology 2007, 243:360-367.
38. Barlow W.E., Chi C., Carney P.A., et al. Accuracy of screening mammography interpretation by characteristics of radiologists. J Natl Cancer Inst 2004, 96:1840-1850.
39. Taplin S.H., Rutter C.M., Lehman C.D. Testing the effect of computer-assisted detection on interpretive performance in screening mammography. AJR Am J Roentgenol 2006, 187:1475-1482.
40. Otero H.J., Rybicki F.J., Greenberg D., et al. Cost-effective diagnostic cardiovascular imaging: when does it provide good value for the money?. Int J Cardiovasc Imaging 2010, 26:605-612.
41. Tosteson A.N.A., Stout N.K., Fryback D.G., et al. Cost-effectiveness of digital mammography breast cancer screening. Ann Intern Med 2008, 148:1-10.
42. Shaw L.J., Wilson P.W.F., Hachamovitch R., et al. Improved near-term coronary artery disease risk classification with gated stress myocardial perfusion SPECT. JACC Cardiovasc Imaging 2010, 3:1139-1148.
43. Niklason L.T., Christian B.T., Niklason L.E., et al. Digital tomosynthesis in breast imaging. Radiology 1997, 205:399-406.
44. Gur D., Abrams G.S., Chough D.M., et al. Digital breast tomosynthesis: observer performance study. AJR Am J Roentgenol 2009, 193:586-591.
45. Pisano E.D., Gatsonis C., Hendrick E., et al. Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med 2005, 353:1773-1783.
46. Gur D., Rockette H.E., Armfield D.R., et al. Prevalence effect in a laboratory environment. Radiology 2003, 228:10-14.
47. Gur D., Bandos A.I., Fuhrman C.R., et al. The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol 2007, 14:49-53.
48. Franke R.H., Kaul J.D. The Hawthorne experiments: first statistical interpretation. Am Sociol Rev 1978, 43:623-643.
49. McCarney R., Warner J., Iliffe S., et al. The Hawthorne effect: a randomised, controlled trial. BMC Med Res Methodol 2007, 7:30.
50. Gur D., Bandos A.I., Cohen C.S., et al. The " laboratory" effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations. Radiology 2008, 249:47-53.
51. US Food and Drug Administration. Summary of safety and effectiveness data: GE FFDM (P990066). Available at: . Accessed November 17, 2011. http://www.accessdata.fda.gov/cdrh_docs/pdf/P990066b.pdf.
52. US Food and Drug Administration. Summary of safety and effectiveness data: Fischer FFDM (P010017). Available at: . Accessed November 17, 2011. http://www.accessdata.fda.gov/cdrh_docs/pdf/P010017b.pdf.
53. US Food and Drug Administration. Summary of safety and effectiveness data: Hologic FFDM (P010025). Available at: . Accessed November 17, 2011. http://www.accessdata.fda.gov/cdrh_docs/pdf/P010025b.pdf.
54. US Food and Drug Administration. Summary of safety and effectiveness data: Fuji FFDM (P050014). Available at: . Accessed November 17, 2011. http://www.accessdata.fda.gov/cdrh_docs/pdf5/P050014b.pdf.
55. Samuelson F., Gallas B.D., Myers K.J., et al. The importance of ROC data. Acad Radiol 2010, 18:257-258.
56. Kundel H.L., Revesz G., Stauffer H.M. Evaluation of a television image processing system. Invest Radiol 1968, 3:44-50.
57. Dean J.C., Ilvento C.C. Improved cancer detection using computer-aided detection with diagnostic and screening mammography: prospective study of 104 cancers. AJR Am J Roentgenol 2006, 187:20-28.
58. Lewin J.M. Full-field digital mammography. A candid assessment. Diagn Imaging 1999, 21:40-45.
59. Cole E., Pisano E.D., Brown M., et al. Diagnostic accuracy of Fischer Senoscan digital mammography versus screen-film mammography in a diagnostic mammography population. Acad Radiol 2004, 11:879-886.
60. Pinsky P, Gallas BD. Enriched designs for assessing predictive performance-analysis of bias and variance. Stat Med. In press.
61. Horvitz D.G., Thompson D.J. A generalization of sampling without replacement from a finite universe. J Am Stat 1952, 47:663-685.
62. Katz J., Zeger S.L. Estimation of design effects in cluster surveys. Ann Epidemiol 1994, 4:295-301.
63. Zhou X.H. Correcting for verification bias in studies of a diagnostic test's accuracy. Stat Methods Med Res 1998, 7:337-353.
64. Alonzo T.A., Pepe M.S. Assessing accuracy of a continuous screening test in the presence of verification bias. J Roy Stat Soc C-App 2005, 54:173-190.
65. He H., Lyness J.M., McDermott M.P. Direct estimation of the area under the receiver operating characteristic curve in the presence of verification bias. Stat Med 2009, 28:361-376.
66. Swets J.A., Pickett R.M. Evaluation of diagnostic systems: methods from signal detection theory 1982, Academic Press, New York.
67. Dorfman D.D., Berbaum K.S., Metz C.E. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol 1992, 27:723-731.
68. Roe C.A., Metz C.E. Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic (ROC) data: validation with computer simulation. Acad Radiol 1997, 4:298-303.
69. Obuchowski N.A., Beiden S.V., Berbaum K.S., et al. Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol 2004, 11:980-995.
70. Gallas B.D. One-shot estimate of MRMC variance: AUC. Acad Radiol 2006, 13:353-362.
71. Obuchowski N.A. Multireader receiver operating characteristic studies: a comparison of study designs. Acad Radiol 1995, 2:709-716.
72. US Food and Drug Administration. Summary of safety and effectiveness data: Hologic DBT (P080003). Available at: . Accessed November 17, 2011. http://www.accessdata.fda.gov/cdrh_docs/pdf8/P080003b.pdf.
73. Kobayashi T., Xu X.W., MacMahon H., et al. Effect of a computer-aided diagnosis scheme on radiologists' performance in detection of lung nodules on radiographs. Radiology 1996, 199:843-848.
74. Beiden S.V., Wagner R.F., Doi K., et al. Independent versus sequential reading in ROC studies of computer-assist modalities: analysis of components of variance. Acad Radiol 2002, 9:1036-1043.
75. Hadjiiski L., Chan H.-P., Sahiner B., et al. Improvement in radiologists' characterization of malignant and benign breast masses on serial mammograms with computer-aided diagnosis: an ROC study. Radiology 2004, 233:255-265.
76. Obuchowski N.A., Meziane M., Dachman A.H., et al. What's the control in studies measuring the effect of computer-aided detection (CAD) on observer performance?. Acad Radiol 2010, 17:761-767.
77. Obuchowski N.A., Mazzone P.J., Dachman A.H. Bias, underestimation of risk, and loss of statistical power in patient-level analyses of lesion detection. Eur Radiol 2010, 20:584-594.
78. Bunch P.C., Hamilton J.F., Sanderson G.K., et al. A free-response approach to the measurement and characterization of radiographic-observer performance. J Appl Photographic Eng 1978, 4:166-171.
79. Egan J.P., Greenberg G.Z., Schulman A.I. Operating characteristics, signal detectability, and the method of free response. J Acoust Soc Am 1961, 33:993-1007.
80. Chakraborty D.P. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys 1989, 16:561-568.
81. Starr S.J., Metz C.E., Lusted L.B., et al. Visual detection and localization of radiographic images. Radiology 1975, 116:533-538.
82. Swensson R.G. Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys 1996, 23:1709-1725.
83. Obuchowski N.A. Nonparametric analysis of clustered ROC curve data. Biometrics 1997, 53:567-578.
84. Chakraborty D.P., Berbaum K.S. Observer studies involving detection and localization: modeling, analysis and validation. Med Phys 2004, 31:2313-2330.
85. Chakraborty D.P. Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol 2006, 13:1187-1193.
86. Edwards D.C., Kupinski M.A., Metz C.E., et al. Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Med Phys 2002, 29:2861-2870.
87. Obuchowski N.A., Lieber M.L., Powell K.A. Data analysis for detection and localization of multiple abnormalities with application to mammography. Acad Radiol 2000, 7:516-525.
88. Zanca F., Chakraborty D.P., Marchal G., et al. Consistency of methods for analysing location-specific data. Radiat Prot Dosimetry 2010, 139:52-56.
89. Chakraborty D.P. Validation and statistical power comparison of methods for analyzing free-response observer performance studies. Acad Radiol 2008, 15:1554-1566.
90. Good W.F., Abrams G.S., Catullo V.J., et al. Digital breast tomosynthesis: a pilot observer study. AJR Am J Roentgenol 2008, 190:865-869.
91. Babinski P.J., Babinski B.S. Adaptability through cross-training in radiology departments. Radiol Manage 2011, 33:45-49.
92. Yablon C.M., Wu J.S., Slanetz P.J., et al. A report on the current status of grand rounds in radiology residency programs in the United States. Acad Radiol 2011, 18:1593-1597.
93. Gur D., Sumkin J.H., Hardesty L.A., et al. Recall and detection rates in screening mammography. Cancer 2004, 100:1590-1594.
94. Jiang Y., Metz C.E., Nishikawa R.M. A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology 1996, 201:745-750.
95. Obuchowski N.A., McClish D.K. Sample size determination for diagnostic accuracy studies involving binormal ROC Curve indices. Stat Med 1997, 16:1529-1542.
96. Baker S.G., Pinsky P.F. A proposed design and analysis for comparing digital and analog mammography: special ROC methods for cancer screening. J Am Stat Assoc 2001, 96:421-428.
97. Zhang D.D., Zhou X.-H., Freeman D.H., et al. A non-parametric method for the comparison of partial areas under ROC curves and its application to large health care data sets. Stat Med 2002, 21:701-715.
98. Wang F., Gatsonis C.A. Hierarchical models for ROC curve summary measures: design and analysis of multi-reader, multi-modality studies of medical tests. Stat Med 2008, 27:243-256.
99. Li C.-R., Liao C.-T., Liu J- P. A non-inferiority test for diagnostic accuracy based on the paired partial areas under ROC curves. Stat Med 2008, 27:1762-1776.
100. Pepe M.S. The statistical evaluation of medical tests for classification and prediction 2003, Oxford University Press, Oxford, United Kingdom.
101. Hadjiiski L., Sahiner B., Helvie M.A., et al. Breast masses: computer-aided diagnosis with serial mammograms. Radiology 2006, 240:343-356.
102. Berbaum K.S., Dorfman D.D., Franken J.E.A., et al. An empirical comparison of discrete ratings and subjective probability ratings. Acad Radiol 2002, 9:756-763.
103. Jiang Y., Nishikawa R.M., Schmidt R.A., et al. Improving breast cancer diagnosis with computer-aided diagnosis. Acad Radiol 1999, 6:22-33.
104. Horsch K., Giger M.L., Vyborny C.J., et al. Classification of breast lesions with multimodality computer-aided diagnosis: observer study results on an independent clinical data set. Radiology 2006, 240:357-368.
105. Horsch K., Giger M.L., Metz C.E. Potential effect of different radiologist reporting methods on studies showing benefit of CAD. Acad Radiol 2008, 15:139-152.
106. Armato S.G., Roberts R.Y., Kocherginsky M., et al. Assessment of radiologist performance in the detection of lung nodules: dependence on the definition of " truth" . Acad Radiol 2009, 16:28-38.
107. Miller D.P., O'Shaughnessy K.F., Wood S.A., et al. Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions. SPIE Proc Med Imaging 2004, 5372:173-184.
108. US Food and Drug Administration. Radiological Devices Panel. Available at: . Accessed November 17, 2011. http://www.fda.gov/ohrms/dockets/ac/08/transcripts/2008-4349t1-05.pdf.
109. US Food and Drug Administration. Radiological Devices Panel: two draft guidance documents for computer-aided-detection (CAD) devices. Available at: . Accessed November 17, 2011. http://www.fda.gov/downloads/AdvisoryCommittees/CommitteesMeetingMaterials/MedicalDevices/MedicalDevicesAdvisoryCommittee/RadiologicalDevicesPanel/UCM197419.pdf.
110. Obuchowski N.A. Predicting readers' diagnostic accuracy with a new CAD algorithm. Acad Radiol 2011, 18:1412-1419.
111. Chan H.-P., Sahiner B., Wagner R.F., et al. Classifier design for computer-aided diagnosis: effects of finite sample size on the mean performance of classical and neural network classifiers. Med Phys 1999, 26:2654-2668.
112. Sahiner B., Chan H.-P., Petrick N., et al. Feature selection and classifier performance in computer-aided diagnosis: the effect of finite sample size. Med Phys 2000, 27:1509-1522.
113. Kupinski M.A., Giger M.L. Feature selection with limited datasets. Med Phys 1999, 26:2176-2182.
114. Simon R., Radmacher M.D., Dobbin K., et al. Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification. J Natl Cancer Inst 2003, 95:14-18.