Assessment of medical imaging and computer-assist systems: Lessons from recent experience

Academic Radiology

Volumn 9, Issue 11, 2002, Pages 1264-1277

  Wagner, Robert F ^a   Beiden, Sergey V ^a   Campbell, Gregory ^a   Metz, Charles E ^b   Sacks, William M ^a  

^a CENTER FOR DEVICES AND RADIOLOGICAL HEALTH   (United States)

^b UNIVERSITY OF CHICAGO   (United States)

Author keywords

Computers, diagnostic aid; Diagnostic radiology, observer performance; Receiver operating characteristic curve (ROC)

Indexed keywords

ARTICLE; BIOMEDICAL TECHNOLOGY ASSESSMENT; BREAST CANCER; CANCER DIAGNOSIS; COMPUTER ASSISTED DIAGNOSIS; DIAGNOSTIC ACCURACY; DIAGNOSTIC IMAGING; DIAGNOSTIC VALUE; HUMAN; IMAGE QUALITY; LUNG CANCER; MAMMOGRAPHY; MEDICAL RESEARCH; OBSERVER VARIATION; PERFORMANCE; PRIORITY JOURNAL; PROCESS MODEL; SENSITIVITY AND SPECIFICITY; SIGNAL NOISE RATIO; THORAX RADIOGRAPHY;

EID: 0036840083     PISSN: 10766332     EISSN: None     Source Type: Journal    
DOI: 10.1016/S1076-6332(03)80560-3     Document Type: Article

References (90)

1. Kessler LG. Digital mammography screening study design. Food and Drug Administration Radiological Devices Advisory Panel Meeting. March 6, 1995, Rockville, Md.
2. Lewin JM. Full-field digital mammography: a candid assessment. Diagn Imaging 1999; 21:40-45.
3. Lewin JM. Digital mammography faces off with film. Diagn Imaging 2001; 23(suppl):10-12.
4. Elmore JG, Wells CK, Lee CH, Howard DH, Feinstein AR. Variability in radiologists' interpretations of mammograms. N Engl J Med 1994; 331:1493-1499.
5. D'Orsi CJ, Swets JA. Variability in the interpretation of mammograms (letter). N Engl J Med 1995; 332:1172.
6. Beam C, Layde PM, Sullivan DC. Variability in the interpretation of screening mammograms by US radiologists. Arch Intern Med 1996; 156:209-213.
7. Metz CE, Shen JH. Gains in accuracy from replicated readings of diagnostic images: Prediction and assessment in terms of ROC analysis. Med Decis Making 1992; 12:60-75.
8. Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis: Generalization to the population of readers and patients with the jackknife method. Invest Radiol 1992; 27:723-731.
9. Swets JA, Pickett RM. Evaluation of Diagnostic Systems. New York, NY: Academic Press, 1982.
10. Metz CE. Basic principles of ROC analysis. Semin Nucl Med 1978; 8:283-298.
11. Metz CE. ROC methodology in radiologic imaging. Invest Radiol 1986; 21:720-733.
12. Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol 1989; 24:234-245.
13. Lusted LB. Signal detectability and medical decision-making. Science 1971; 171:1217-1219.
14. Lusted LB. ROC recollected (editorial). Med Decis Making 1984; 4:131-135.
15. McClish DK. Analyzing a portion of the ROC curve. Med Decis Making 1989; 9:190-195.
16. Jiang Y, Metz CE, Nishikawa RM. A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology 1996; 201:745-750.
17. Unpublished data from C.E.M., July 2, 2002, regarding registrations at The University of Chicago Web site: www-radiology.uchicago.edu/krl /toppage11.htm#software. Accessed February 15, 2002.
18. Gatsonis CA, Begg CB, Wieand S, Advances in statistical methods for diagnostic radiology: a symposium. Acad Radiol 1995; 2(suppl 1): S1-S84.
19. Rockette HE, Gur D, Kurs-Lasky M, King JL. On the generalization of the receiver operating characteristic analysis to the population of readers and cases with the jackknife method: An assessment. Acad Radiol 1995; 2:66-69.
20. University of Iowa MRMC Software. Available at: ftp://perception.radiology.uiowa.edu/PUBLIC. Accessed February 15, 2002.
21. Web address for University of Chicago LABMRMC software. Available at: www-radiology.uchicago.edu/krl/toppage11.htm#software. Accessed February 15, 2002.
22. Roe CA, Metz CE, Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: Validation with computer simulation. Acad Radiol 1997; 4:298-303.
23. Dorfman DD, Berbaum KS, Lenth RV, Chen YF, Donaghy BA. Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: Factorial experimental design. Acad Radiol 1998; 5:591-602.
24. Obuchowski NA. Multireader, multimodality receiver operating characteristic curve studies: Hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Acad Radiol 1995; 2(suppl 1):S22-S29.
25. Toledano A, Gatsonis CA. Regression analysis of correlated receiver operating characteristic data. Acad Radiol 1995: 2(suppl 1):S30-S36.
26. Toledano A, Gatsonis C. Ordinal regression methodology for ROC curves derived from correlated data. Stat Med 1996; 15:1807-1826.
27. Ishwaran H, Gatsonis CA. A general class of hierarchical ordinal regression models with applications to correlated ROC analysis. Can J Stat 2000; 28:731-750.
28. Belden SV, Wagner RF, Campbell G. Components-of-variance models and multiple-bootstrap experiments: An alternative method for random-effects receiver operating characteristic analysis. Acad Radiol 2000; 7:341-349.
29. Belden SV, Wagner RF, Campbell G, Metz CE, Jiang Y. Components-of-variance models for random-effects ROC analysis: The case of unequal variance structures across modalities. Acad Radiol 2001; 8:605-615.
30. Belden SV, Wagner RF, Campbell G, Chan HP. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol 2001; 8:616-622.
31. American College of Radiology. Breast imaging recording and data system (BIRADS). Reston, Va: American College of Radiology, 1993.
32. Wagner RF, Belden SV, Metz CE. Continuous versus categorical data for ROC analysis: Some quantitative considerations. Acad Radiol 2001; 8:328-334.
33. Begg CG. Invited discussion. Acad Radiol 1995; 2(suppl 1):S57-S60.
34. Hanley JA. Receiver operating characteristic (ROC) methodology: The state of the art. Crit Rev Diagn Imaging 1989; 23:307-335.
35. Food and Drug Administration Radiological Devices Advisory Panel Meeting. December 16, 1999, Rockville Md. Available at: www. accessdata.fda,gov/scripts/cdrh/cfdocs/cfAdvisory/results.cfm?panel =24&month=0&year=0&searchtype=1&pagenum=1&maxrows=10. Accessed February 15, 2002.
36. Hendrick RE, Lewin JM, D'Orsi CJ, et al. Non-inferiority study of FFDM in an enriched diagnostic cohort: Comparison with screen-film mammography in 625 women. In: Yaffe MJ, ed. Proceedings of the 5th International Workshop on Digital Mammography, Toronto, Canada. Madison, Wis: Medical Physics Publishing, 2001; 475-481.
37. Lewin JM, Hendrick RE, D'Orsi CJ, et al. Interim clinical evaluation of FFDM in a screening cohort: Comparison of full-field digital mammography to screen-film mammography for cancer detection - Results of 4945 paired examinations. Radiology 2001; 218:873-880.
38. Wagner RF, Belden SV, Campbell G. Multiple-reader studies, digital mammography, computer-aided diagnosis-and the Holy Grail of imaging physics. I. Proc SPIE 2001; 4320:611-618.
39. Jiang Y, Nishikawa RM, Schmidt RA, Metz CE, Giger ML, Doi K. Improving breast cancer diagnosis with computer-aided diagnosis. Acad Radiol 1999: 6:22-33.
40. Chan HP, Sahiner B, Helvie MA, et al. Improvement of radiologists' characterization of mammographic masses by using computer-aided diagnosis: An ROC study. Radiology 1999; 212:817-827.
41. Nishikawa RM, Gatsonis C, Schnall MD, et al. Large scale observer study to measure the benefits of computer-aided detection to screening mammography (abstr). Radiology 1999; 213(P):150.
42. Warren Burhenne LJ, Wood SA, D'Orsi CJ, et al. Potential contribution of computer-aided detection to the sensitivity of screening mammography. Radiology 2000; 215:554-562.
43. Food and Drug Administration Radiological Devices Advisory Panel Meeting, May 11, 1998. Available at: www.accessdata.fda.gov/scripts /cdrh/cfdocs/cfAdvisory/results.cfm?panel=24&month=0&year= 0&searchtype=1&pagenum=1&maxrows=10. Accessed February 15, 2002.
44. Kundel HL, Polansky M. Comparing observer performance with mixture distribution analysis when there is no external gold standard. Proc SPIE 1998; 3340:78-84.
45. Begg CB, McNeil BJ. Assessment of radiologic tests: Control of bias and other design considerations. Radiology 1988; 167:565-569.
46. Birdwell RL, Ikeda DM, O'Shaughnessy KF, Sickles EA. Mammographic characteristics of 115 missed cancers later detected with screening mammography and the potential utility of computer-aided detection. Radiology 2001; 219:192-202.
47. Zheng B, Ganott MA, Britton CA, et al. Soft-copy mammographic readings with different computer-assisted detection cuing environments: Preliminary findings. Radiology 2001; 221:633-640.
48. D'Orsi CJ. Computer-aided detection: There is no free lunch (editorial). Radiology 2001; 221:585-586.
49. Berbaum KS, Franken EA Jr, Dorfman DD, et al. Satisfaction of search in diagnostic radiology. Invest Radiol. 1990; 25:133-140.
50. Food and Drug Administration Radiological Devices Advisory Panel Meeting, March 5, 2001, Rockville, Md. Available at: www.accessdata .fda.gov/scripts/cdrh/cfdocs/cfAdvisory/results.cfm?panel =24&month=0&year=0&searchtype=1&pagenum=1&maxrows=10. Accessed February 15, 2002.
51. Freedman M, Lo SCB, Lure F, et al. Design choices in the Deus Technologies RS-2000 lung cancer on chest radiograph ROC clinical trial. Presented at Medical Image Perception Conference 1X, September 20-23, 2001, Warrenton, Va.
52. Belden SV, Wagner RF, Doi K, et al. Independent versus sequential reading in ROC studies of computer-assist modalities: Analysis of components of variance. Acad Radiol (in press).
53. Sacks WM. The key points regard the appropriate steady-state level of actual cancers to be used for the normalization, and the arbitrariness of the time interval used for collecting the priors (letter). Radiology (in press).
54. Efron B, Tibshirani RJ. An introduction to the bootstrap. Monographs on Statistics and Applied Probability 57. New York, NY: Chapman & Hall, 1993).
55. Beiden SV, Wagner RF, Campbell G, et al. Analysis of components of variance in multiple-reader studies of computer-aided diagnosis with different tasks. Proc SPIE 2001; 4324:167-176.
56. Software available from S.V.B. at svb@cdrh.fda.gov or at rfw@cdrh.fda.gov.
57. Obuchowski NA. Multireader, multimodality receiver operating characteristic curve studies: Hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Acad Radiol 1995; 2(suppl 1):S22-S29.
58. Rockette HE, Campbell WL, Britton CA, Holbert JM, King JL, Gur D. Empiric assessment of parameters that affect the design of multireader receiver operating characteristic studies. Acad Radiol 1999; 6:723-729.
59. Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983; 148:839-843.
60. Obuchowski NA. Computing sample size for receiver operating characteristic studies. Invest Radiol 1994; 29:238-243.
61. Metz CE. Statistical analysis of ROC data in evaluating diagnostic performance. In: Herbert DE, Myers RH, eds. Multiple Regression Analysis: Applications in the Health Sciences. New York, NY: American Institute of Physics/American Association of Physicists in Medicine, 1986; 365-384.
62. Chan HP, Helvie MA, Petrick N, et al. Digital mammography: Observer performance study of effects of pixel size on radiologists' characterization of malignant and benign microcalcifications. Acad Radiol 2001; 8:454-466.
63. Henkelman RM, Kay I, Bronskill MJ. Receiver operator characteristic (ROC) analysis without truth. Med Decis Making 1990; 10:24-29.
64. Begg CB, Metz CE. Consensus diagnoses and "gold standards." Med Decis Making 1990; 10:29-30.
65. Beiden SV, Campbell G, Meier KL, Wagner RF. On the problem of ROC analysis without truth: The EM algorithm and the information matrix. Proc SPIE 2000; 3981:126-134.
66. Revesz G, Kundel HL, Bonitatibus M. The effect of verification on the assessment of imaging techniques. Invest Radiol 1983; 18:194-198.
67. Brown DG. The evaluation of computer-aided diagnosis systems: An FDA perspective. Proceedings of the 30th Applied Imagery Pattern Recognition Workshop (AIPR 2001). Los Alamitos, Calif: IEEE Computer Society, 2001; 17-20.
68. Guidance for the Submission of 510(k)'s for solid state x-ray imaging devices. U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health, August 6, 1999. Available at: www.fda.gov/cdrh/ode/ssxiguid.pdf. Accessed February 15, 2002.
69. Premarket applications for digital mammography systems: final guidance for industry and FDA. U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health, February 4, 1999. Available at: www.fda.gov/cdrh /ode/guidance/983.html. Accessed February 15, 2002.
70. Houn F, Bright RA, Bushar HF, et al. Study design in the evaluation of breast cancer imaging technologies. Acad Radiol 2000; 7:684-692.
71. International Commission on Radiation Units and Measurements. Medical Imaging: The Assessment of Image Quality. ICRU Report no. 54. Bethesda, Md: International Commission on Radiation Units and Measurements, 1996.
72. Metz CE, Wagner RF, Doi K, Brown DG, Nishikawa RM, Myers KJ. Toward consensus on quantitative assessment of medical imaging systems. Med Phys 1995; 22:1057-1061.
73. Belden SV, Wagner RF, Campbell G, Metz CE, Jiang Y, Chan HP. Multiple-reader studies, digital mammography, computer-aided diagnosis - and the Holy Grail of imaging physics. II. Proc SPIE 2001; 4320:619-626.
74. Dorfman DD, Berbaum KS, Lenth RV, Chen YF. Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: Split plot experimental design. Proc SPIE 1999; 3663:91-99.
75. Costigan T. Bonferroni inequalities and intervals. In: Armitage P, Colton T, eds. Encyclopedia of Biostatistics. Vol 1. New York, NY: Wiley, 1998; 421-425.
76. Swensson RG. Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys 1996; 23:1709-1725.
77. Chakraborty DP, Winter L. Free-response methodology: Alternate analysis and a new observer-performance experiment. Radiology 1990; 174:873-881.
78. Chakraborty D. Statistical power in observer performance studies: A comparison of the ROC and free-response methods in tasks involving localization. Acad Radiol 2002; 9:147-156.
79. Metz CE. Evaluation of digital mammography by ROC analysis. In: Doi K, Giger ML, Nishikawa RM, Schmidt RA, eds. Digital Mammography '96. Amsterdam, The Netherlands: Elsevier Science, 1996; 61-68.
80. Metz CE. Evaluation of CAD methods. In: Doi K, MacMahon H, Giger ML, Hoffmann KR, eds. Computer-Aided Diagnosis in Medical Imaging. Amsterdam, The Netherlands: Elsevier Science, 1999; 43-554.
81. Nishikawa RM, Yarusso LM. Variations in measured performance of CAD schemes due to database composition and scoring protocol. Proc SPIE 1998; 3338:840-844.
82. Giger ML. Currrent issues in CAD for mammography. In: Doi K, Giger ML, Nishikawa RM, Schmidt RA, eds. Digital Mammography '96. Amsterdam, The Netherlands: Elsevier Science, 1996; 53-59.
83. Starr SJ, Metz CE, Lusted LB, Goodenough DJ. Visual detection and localization of radiographic images. Radiology 1975; 116:533-538.
84. , Metz CE. Fundamental ROC analysis. In: Beutel J, Kundel HL, Van Metter RL, eds. Handbook of medical imaging. Vol 1. Physics and Psychophysics. Bellingham, Wash: SPIE Press, 2000; 751-769.
85. Chakraborty DP. The FROC, AFROC and DROC variants of the ROC analysis. In: Beutel J, Kundel HL, Van Metter RL, eds. Handbook of medical imaging. Vol 1. Physics and psychophysics. Bellingham, Wash: SPIE Press, 2000; 771-796.
86. Ransohoff DF, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med 1978; 299:926-930.
87. Baker SG, Pinsky P. A proposed design and analysis for comparing digital and analog mammography: special ROC methods for cancer screening. J Am Stat Assoc 2001; 96:421-428.
88. Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991; 11:88-94.
89. National Council on Radiation Protection and Measurements. An Introduction to Efficacy in Diagnostic Radiology and Nuclear Medicine. NCRP Commentary no, 13. Bethesda, Md: National Council on Radiation Protection and Measurements, 1995.
90. American College of Radiology Imaging Network. Information available at: www.acrin.org. Accessed February 15, 2002.