Unified measurement of observer performance in detecting and localizing target objects on images

Medical Physics

Volumn 23, Issue 10, 1996, Pages 1709-1726

  Swensson, Richard G ^a  

^a BRIGHAM AND WOMEN S HOSPITAL   (United States)

Author keywords

AFROC; FROC; image interpretation; localization; LROC; observer performance; ROC analysis; signal detection

Indexed keywords

MAXIMUM LIKELIHOOD; OBJECT DETECTION; SIGNAL DETECTION;

AFROC; FROC; IMAGE INTERPRETATION; LOCALIZATION; LROC; OBSERVER PERFORMANCE; ROC ANALYSIS;

CURVE FITTING;

ARTICLE; DIAGNOSTIC IMAGING; HUMAN; IMAGE ANALYSIS; OBSERVER VARIATION; PRIORITY JOURNAL; SIGNAL DETECTION; TASK PERFORMANCE; VISUAL DISCRIMINATION;

EID: 0029822853     PISSN: 00942405     EISSN: None     Source Type: Journal    
DOI: 10.1118/1.597758     Document Type: Article

References (62)

1. K. L. Kundel, "Predictive value and threshold detectability of lung tumors," Radiology 139, 25-29 (1981).
2. S. E. Seltzer, R. G. Swensson, R. D. Nawfel, J. F. Lentini, I. Kazda, and P. F. Judy, "Visualization and detection-localization on computed tomographic images," Invest. Radiol. 26, 285-291 (1991).
3. C. E. Metz, "Some practical issues of experimental design and data analysis in radiological ROC studies," Invest. Radiol. 24, 234-245 (1989).
4. J. A Swets and R. M. Pickett, Evaluation of diagnostic systems: Methods from signal detection theory (Academic, New York, 1982).
5. D. D. Dorfman and E. Alf, "Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals - Rating-method data," J. Math. Psychol. 6, 487-496 (1969).
6. D. M. Green and J. A. Swets, Signal Detection Theory and Psychophysics (Wiley, New York, 1966), reprinted by Peninsula Pub., Los Altos, CA, 1988.
7. N. A. Macmillan and C. D. Creelman, Detection Theory: A User's Guide (Cambridge University, New York, 1991).
8. P. F. Judy, R. G. Swensson, R. D. Nawfel, K. H. Chan, and S. E. Seltzer, "Contrast-detail curves for liver CT," Med. Phys. 19, 1167-1174 (1992).
9. S. E. Seltzer, P. F. Judy, R. G. Swensson, K. H. Chan, and R. D. Nawfel, "Flattening of the contrast-detail curve for large lesions on liver CT images," Med. Phys. 21, 1547-1555 (1994).
10. R F. Wagner, "Decision theory and the detail signal-to-noise ratio of Otto Shade," Photogr. Sci. Eng. 22, 41-46 (1978).
11. R. F. Wagner and D G. Brown, "Unified SNR analysis of medical imaging systems," Phys. Med. Biol. 30, 489-518 (1985).
12. H. H. Barrett, J. Yao, J. P. Rolland, and K J. Myers, "Model observers for assessment of image quality," Proc. Natl. Acad. Sci. 90, 9758-9765 (1993).
13. A. E. Burgess, "High level visual decision efficiencies," in Vision: Coding and Efficiency, edited by C. Blakemore (Cambridge University, New York, 1990), pp. 431-440.
14. A. E. Burgess, R. F. Wagner, R. J. Jennings, and H. B. Barlow, "Efficiency of human visual signal discrimination," Science 214, 93-94 (1981).
15. R. G. Swensson and P. F. Judy, "Measuring performance efficiency and consistency in visual discriminations with noisy images," J. Exp. Psychol. Hum. Percpt. Perf. (in press).
16. D. R. Grey and B. J. T. Morgan, "Some aspects of ROC curve-fitting: Normal and logistic models," J. Math. Psychol. 9, 128-139 (1972).
17. J. C. Ogilvie and C. D Creelman, "Maximum-likelihood estimation of receiver operating characteristic curve parameters," J. Math. Psychol. 5, 377-391 (1968).
18. M. F. Kijewski, R. G. Swensson, and P. F. Judy, "Analysis of rating data from multiple-alternative tasks," J. Math. Psychol. 33, 428-451 (1989).
19. C. E. Metz, P. L. Wang, and H. B. Kronman, "A new approach for testing the significance of differences between ROC curves measured from correlated data," in Information Processing in Medical Imaging, edited by F. Deconink (Nijhoff, The Hague, 1984), pp. 432-435.
20. T. D. Wickens, "Maximum-likelihood estimation of a multivariate Gaussian rating model with excluded data," J. Math. Psychol. 36, 213-234 (1992).
21. K. S. Berbaum, E. A. Franken, Jr., D. D. Dorfman, and T. J. Barloon, "Influence of clinical history upon detection of nodules and other lesions," Invest. Radiol. 23, 48-55 (1988).
22. K. B. Herath and P. F. Sharp, "Effects of 'matched filter' smoothing as measured by receiver operating characteristic curve," Phys. Med. Biol. 21, 442-446 (1976).
23. S. J. Starr, C. E. Metz, L. B Lusted, and D. J Goodenough, "Visual detection and localization of radiographic images," Radiology 116, 533-538 (1975).
24. S. J. Starr, C. E. Metz, and L. B. Lusted, "Letter to the editor: Comments on the generalization of receiver operating characteristic analysis to detection and localization tasks," Phys. Med. Biol. 22, 376-379 (1977).
25. R. G. Swensson and P. F. Judy, "Detection of noisy visual targets: Models for the effects of spatial uncertainty and signal-to-noise ratio," Percept. Psychophy. 29, 521-534 (1981).
26. A. F. Burgess, "Comparison of receiver operating characteristic and forced choice observer performance measurement methods," Med. Phys. 22, 643-655 (1995).
27. A. E. Burgess and H. Ghandeharian, "Visual signal detection. I: Ability to use phase information," J. Opt. Soc. Am. A 1, 900-905 (1984).
28. B. Moulden and F. Kingdom, "Effect of pixel width, display width, and number of alternative signal locations on the detection of a simple vertical-line signal in visual noise," Percept. Psychophy. 43, 592-598 (1988).
29. H-P. Chan, K. Doi, C. J. Vyborny, R. A. Schmidt, C. E. Metz, K. L. Lam, T. Ogura, Y. Wu, and H. MacMahon, "Improvement in radiologists' detection of clustered microcalcifications on mammograms: The potential of computer-aided diagnosis," Invest. Radiol. 25, 1102-1110 (1990).
30. J. E. Goin, J. D. Haberman, M. K. Kinder, and P. A Lambird, "Analysis of mammography: A blind interpretation of BCDDP radiographs," Radiology 148, 393-396 (1983).
31. R. G. Swensson, "Observer performance tests in chest imaging," in Proceedings of Chest Imaging Conference '87, edited by W W. Peppler and A. Alter (Medical Physics Publishing, Madison, WI, 1988), pp. 337-355.
32. S. E. Seltzer, P. F. Judy, D F. Adams, J. L. Jacobson, P. Stark, R. Kikinis, G. G. Swensson, S. Hooton, B. Head, and U. Feldman, "Spiral CT of the chest: Comparison of cine and film-based viewing," Radiology 197, 73-78 (1995).
33. P C. Bunch, J. F. Hamilton, G. K. Sanderson, and A. H. Simmons, "A free response approach to the measurement and characterization of radiographic observer performance," Proc. SPIE 154, 124-135 (1977).
34. D. P. Chakraborty, E. A. Breatnach, M. V. Yester, B. Soto, G. T. Barnes, and R. F. Fraser, "Digital and conventional chest imaging: A modified ROC study of observer performance using simulated nodules," Radiology 158, 35-39 (1986).
35. T. F. Meaney, U. Raudkivi, W. J. McIntyre, J. H. Gallagher, J. R. Haaga, T. R. Havrilla, and N E. Reich, "Detection of low-contrast lesions in computed body tomography: An experimental study of simulated lesions," Radiology 134, 149-154 (1980).
36. C. R. Mitchel and J. A. Sorenson, "Application of the Bunch transform in clinical FROC nodule detection studies," in Ref. 31, pp. 356-370.
37. L. T. Niklason, N. M. Hickey, D. P. Chakraborty, E. A. Sabbagh, M. Y. Yester, R. G. Fraser, and G. T. Barnes, "Simulated pulmonary nodules: Detection with dual-energy digital versus conventional radiology," Radiology 160, 589-594 (1986).
38. G. D. Tourassi and C. E. Floyd, Jr., "Artificial neural networks for single photon emission computed tomography: A study of cold lesion detection and localization," Invest. Radiol. 28, 671-677 (1993).
39. A. Yoshimura, M. L. Giger, K. Doi, H. MacMahon, and S. M. Montner, "Computerized scheme for the detection of pulmonary nodules: A nonlinear filtering technique," Invest. Radiol. 27, 124-129 (1992).
40. D. P. Chakraborty, "Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data," Med. Phys. 16, 561-568 (1989).
41. P. G Herman, J. Drummey, R. G. Swensson, S. J. Hessel, and J. P. Balikian, "350 kV chest radiography has no diagnostic advantage: A comparison with 140 kV technique," Am. J. Roentol. 138, 485-489 (1982).
42. R. G. Swensson, S. J. Hessel, and P. G. Herman, "Radiographic interpretation with and without search: Visual search aids the recognition of chest pathology," Invest. Radiol. 17, 145-151 (1982).
43. R. G. Swensson, S. J. Hessel, and P. G. Herman, "The value of searching films without specific preconceptions," Invest. Radiol. 20, 100-107 (1985).
44. R. G. Swensson, S. J. Hessel, and P. G. Herman, "Different methods of analyzing radiographic interpretations do not alter the value of searching films," Reply to editorial comments, Invest. Radiol. 20, 111-114 (1985).
45. R. G. Swensson and G. H. Theodore, "Search and nonsearch protocols for radiographic consultation," Radiology 177, 851-856 (1990).
46. J. A. Swets, "On the supposed value of searching films," Editorial comment, Invest. Radiol. 20, 108-109 (1985).
47. J. A. Swets, "On searching films and begging the question," Letter to editor, Invest. Radiol. 20, 115-116 (1985).
48. R. G. Swensson, "On the effects of reader variability in searching films," Letter to editor, Invest Radiol. 20, 231-233 (1985).
49. D. P. Chakraborty and L. H L. Winter, "Free-response methodology: Alternative analysis and a new observer-performance experiment," Radiology 174, 873-881 (1990).
50. W. J. Tuddenham, "Visual search, image-organization, and reader error in Roentgen diagnosis: Studies of the psychophysiology of roentgen image perception," Radiology 78, 694-704 (1962).
51. K S. Berbaum, E. A. Franken, Jr., D. D. Dorfman, S. A. Rooholamini, T. J. Barloon, F. M. Behlke, Y. Sato, C. H. Lu, G. Y. E. El-Khoury, F. D. W. Flickinger, and W. J. Montgomery, "Satisfaction of search in diagnostic radiology," Invest. Radiol. 25, 133-140 (1990).
52. K. S. Berbaum, E. A. Franken, Jr., D. D. Dorfman, S. A. Rooholamini, C. E Coffman, S. H. Cornell, A. H. Cragg, F. R. Galvin, H. Honda, S. C. S. Kao, D. A. Kimball, T. J. Ryals, W. J. Sickels, and T. P. Smith, "Time course of satisfaction of search," Invest. Radiol. 26, 640-648 (1991).
53. C. E. Metz, S. J. Starr, L. B. Lusted, and K. Rossmann, "Progress in evaluation of human observer visual detection performance using the ROC curve approach," in Information processing in Scintigraphy, edited by C. Raynaud and A. E. Todd-Pokropek (CEA, Orsay, France, 1975), pp. 420-436.
54. C. E. Metz, "Current problems in ROC analysis," in Ref. 31, pp. 315-336.
55. W. A. Wickelgren, "Unidimensional strength theory and component analysis of noise in absolute and comparative judgments," J. Math. Psychol. 5, 102-122 (1968).
56. W. Feller, An Introduction to Probability Theory and its Applications (Wiley, New York, 1950), Vol. 1.
57. R. Fletcher, Practical Methods of Optimization (Wiley, New York, 1987).
58. W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling, Numerical Recipes: The Art of Scientific Computing (Cambridge University, New York, 1986).
59. The fitting program, written by Karen H. Chan in FORTRAN, currently runs on a Sun Sparc Workstation, and is available from the author on request
60. A. Agresti, Categorical Data Analysis (Wiley, New York, 1990).
61. M. G. Kendall and A. Stuart, The Advanced Theory of Statistics: Vol. 2, Inference and Relationship, 4th ed. (Griffin, London, 1979).
62. M. G. Kendall and A. Stuart, The Advanced Theory of Statistics: Vol. 1, Distribution Theory, 4th ed. (Macmillan, New York, 1977).