ROC analysis applied to the evaluation of medical imaging techniques

Investigative Radiology

Volumn 14, Issue 2, 1979, Pages 109-121

  Swets, John A ^a  

^a NONE

Author keywords

Diagnostic radiology; Radiograph interpretation; Roc analysis

Indexed keywords

DIAGNOSIS; METHODOLOGY; RADIOGRAPHY;

DECISION MAKING; DIAGNOSTIC ERRORS; EVALUATION STUDIES; HUMAN; MATHEMATICS; PSYCHOPHYSICS; RADIOGRAPHY;

EID: 0018390029     PISSN: 00209996     EISSN: 15360210     Source Type: Journal    
DOI: 10.1097/00004424-197903000-00002     Document Type: Article

References (59)

1. Ackerman LV, Gose EE: Breast lesion classification by computer and xeroradiograph. Cancer 30: 1025, 1972.
2. Alcorn FS, O’Donnell E: The training of nonphysician personnel for use in a mammography program. Cancer 23: 879, 1969.
3. Anderson Jr TM, Mintzer RA, Hoffer PB, et al: Nuclear image transmission by picturephone. Invest Radiol 8: 244, 1973.
4. Andrus WS, Dreyfuss JR, Jaffer F. et al.: Interpretation of roentgenograms via interactive television. Radiology 116: 25, 1975.
5. Andrus WS, Hunter CH, Bird KT: Remote interpretation of chest roentgenograms. Chest 64: 463, 1975.
6. Bacus JW: The observer error in peripheral blood cell classification. Am J Clin Pathol 59: 223, 1973.
7. Birdsall TG: Unpublished memorandum, 1954.
8. Blackwell RJ, Shirley I, Farman DJ, et al.: Ultrasonic “B” scanning as a pregnancy test after less than six weeks amenorrhoea. Br J Obstet Gynecol 82: 108, 1975.
9. Brookes BC: The measures of information retrieval effective ness proposed by Swets. J Documentation 24: 41, 1968.
10. Dorfman DD, Alf Jr E: Maximum likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—rating method data. J Math Psych, 6: 487, 1969.
11. Dorfman DD, Beavers LL, Saslow C: Estimation of signal-detection theory parameters from rating-method data: a comparison of the method of scoring and direct search. Bull Psychon Soc 1: 207, 1973.
12. Dusoir AE: Treatments of bias in detection and recognition models: a review. Percept Psychophys 17: 167, 1975.
13. Egan JP, Clarke FR: Psychophysics and signal detection. In Sidowsky JB (ed): Experimental Methods and Instrumentation in Psychology. New York, McGraw-Hill, 1966, p 211.
14. Fechner GT: Elemente der Psychophysik. Leipzig, Breitkopf & Hartel, 1860. English translation of Volume 1, Howes DH, Boring EG (eds). New York, Holt, Rinehart & Winston, Reviewed by Swets JA, Science, 154: 1532, 1966.
15. Garland LH: Scientific evaluation of diagnostic procedures. Radiology 52: 309, 1949.
16. Goodenough DJ, Metz CE, Lusted LB: Caveat on use of the parameter d’ for evaluation of observer performance. Radiology 106: 565, 1973.
17. Goodenough DJ, Rossmann K, Lusted LB: Radiographic applications of signal detection theory. Radiology 105: 199, 1972.
18. Green DM, Swets JA: Signal Detection Theory and Psycho physics. New York, John Wiley & Sons, 1966. Reprinted: Huntington, New York, Robert E. Krieger Publishing Co, 1974.
19. Hershman RL, Levine JR, Lichtenstein M: Signal detection and localization by real observers. Percept Psychophys 6: 53, 1969.
20. Hershman RL, Lichtenstein M: Detection and localizability: an extension of the theory of signal detectability. J Acoust Soc Amer 42: 446, 1967.
21. Hershman RL, Small D: Tables of d’ for detection and localization. Percept Psychophys 3: 321, 1968.
22. Kundel HL: Factors limiting roentgen interpretation—physical and psychologic. In Potchen EJ (ed): Current Concepts in Radiology. St. Louis, C V Mosby, 1972, p 1.
23. Lusted LB: Introduction to Medical Decision Making. Springfield, Charles C Thomas, 1968, p 103.
24. Lusted LB: Observer error, signal detectability and medical decision making. In Jacquez JA (ed): Computer Diagnosis and Diagnostic Methods. Springfield, Charles C Thomas, 1972, p 29.
25. McNeil BJ, Hessel SJ, Branch WT, et al: Measures of clinical efficacy, III. The value of the lung scan in the evaluation of young patients with pleuritic chest pain. J Nucl Med 17: 163, 1976.
26. McNeil, BJ, Varady, MS, Burrows BA, et al: Cost effectiveness in hypertensive renovascular disease. New Eng J Med 293: 216, 1975.
27. McNeil BJ, Weber E, Harrison D, et al: Use of signal detectiontheory in examining the results of a contrast examination: A case study using the lymphangiogram. Radiology 123: 613, 1977.
28. Metz CE: Basic principles of ROC analysis. Semin Nucl Med 8: 283, 1978.
29. Metz CE, Goodenough DJ: On failure to improve observer performance with scan smoothing: a rebuttal. J Nucl Med 14: 873, 1973.
30. Metz CF, Goodenough DJ, Rossmann K: Evaluation of receiver operating characteristic curve data in terms of information theory, with applications in radiography. Radiology 109: 297, 1973.
31. Metz CE, Starr SJ, Lusted LB, et al: Progress in evaluation of human observer visual detection performance using the ROC curve approach. In Raynaud C, Todd-Pokropek A (eds): Information Processing in Scintigraphy. Orsay, France, Commissariat a l’Energie Atomique, Departement de Biologie, Service Hospitalier Frederic Joliot, 1975, p 420.
32. Morgan RH, Donner MW, Gayler BW, et al: Decision processes and observer error in the diagnosis of pneumoconiosis by chest roentgenography. Am J Roentgenol 117: 757, 1973.
33. Moskowitz M, Milbrath J, Gartside P, et al: Lack of efficacy of thermography as a screening tool for minimal and stage 1 breast cancer. New Eng J Med 295: 249, 1976.
34. Nishiyama HJ, Lewis T, Ashare AB, et al: Interpretation of liver images: do training and experience make a difference? J Nucl Med 16: 11, 1975.
35. Norman DA: A comparison of data obtained with different false-alarm rates. Psychol Rev 71: 243, 1964.
36. Peterson WW, Birdsall TG, Fox WC: The theory of signal detectability. Trans IRE Prof Group on Inform Theory PGIT- 4: 171, 1954.
37. Pickett RM: Toward more accurate and efficient visual screening of Pap smears: forced piecewise diagnosis. (Paper in preparation.)
38. Revesz G, Haas C: Television display of radiographic images with superimposed simulated lesions. Radiology 102: 197, 1972.
39. Robertson SE: The parametric description of retrieval tests. Part II: overall measures. J Doc 25: 93, 1969.
40. Schulman AL, Mitchell RR: Operating characteristics from yes-no and forced-choice procedures. J Acoust Soc Am 40: 473, 1966.
41. Simpson AJ, Fitter MJ: What is the best index of detectability? Psychol Bull 80: 481, 1973.
42. Starr SJ, Metz CE: ROC analysis of the efficacy of breast thermography when disease localization is required. Unpublished, 1976.
43. Starr SJ, Metz CE, Lusted LB, et al: Visual detection and localization of radiographic images. Radiology 116:533, 1975.
44. Stevens SS: Is there a quantal threshold? In Rosenblith WA (ed): Sensory Communication. New York, Techology Press and Wiley, 1961, pp. 806-813.
45. Swensson RG, Hessel SJ, Herman PG: Omissions in radiology: faulty search or stringent reporting criteria? Radiology 123: 563, 1977.
46. Swets JA: Indices of signal detectability obtained with various psychophysical procedures. J Acoust Soc Am 31: 511, 1959.
47. Swets JA: The relative operating characteristic in psychology Science 182: 990, 1973.
48. Swets JA, Birdsall TG: Deferred decisions in human signal detection: a preliminary experiment. Percept Psychophys 2: 15 1967.
49. Swets JA, Birdsall TG: The human use of information, III Decision-making in signal detection and recognition situations involving multiple alternatives. Trans Inst Radiol Eng, Prof Group Inform Theory IT 2: 138, 1956.
50. Swets JA, Green DM, Getty DJ, et al: Signal detection and identification at successive stages of observation. Percept Psychophys 23: 275, 1978.
51. Swets JA, Green DM, Getty DJ, et al: Identification and scaling of complex visual patterns. Technical Report No. 3536, Bolt Beranek and Newman Inc, November 1977.
52. Swets JA, Pickett RM, Whitehead SF, et al: Accuracy of computed tomography and radionuclide scanning in detection and diagnosis of intracranial lesions: application of a general protocol for evaluation of imaging techniques in clinical medicine. Technical Report No. 3818, Bolt Beranek and Newman Inc, June 1978.
53. Swets JA, Tanner Jr WP, Birdsall TG: Decision processes in perception. Psychol Rev 68: 301, 1961.
54. Tanner Jr WP: Theory of recognition. J Acoust Soc Am 28: 882, 1956.
55. Tanner Jr WP, Swets JA: A decision-making theory of visual detection. Psychol Rev 61: 401, 1954.
56. Tolies WE, Horvath WJ, Bostrom RC: A study of the quantitative characteristics of exfoliated cells from the female genital tract: II. Suitability of quantitative cytological measurements for automatic prescreening. Cancer 14: 457, 1961.
57. Turner DA, Fordham EW, Pagano JV, et al: Brain scanning with the Anger multiplane tomographic scanner as a second examination. Radiology 121: 115, 1976.
58. Wald A: Statistical Decision Functions. New York, John Wiley & Sons, Inc, 1950.
59. Yerushalmy J, Harkness JT, Cope JH, et al: Role of dual reading in mass radiography. Am Rev Tubercol 61: 443, 1950.