Receiver operating characteristic analysis in medical imaging

Journal of the ICRU

Volumn 8, Issue 1, 2008, Pages 1-62

[No Author Info available]

Author keywords

[No Author keywords available]

Indexed keywords

EID: 52049083009     PISSN: 14736691     EISSN: 17423422     Source Type: Journal    
DOI: 10.1093/jicru/ndn003     Document Type: Article

References (182)

1. Anastasio, M. A., Kupinski, M. A. and Nishikawa, R. M. (1998). "Optimization and FROC analysis of rule-based detection schemes using a multiobjective approach," IEEE Trans. Med. Imaging 17, 1089-1093.
2. Anderson, R. and Bancroft, T. (1952). Statistical Theory in Research (McGraw-Hill, New York).
3. Arvesen, J. and Salsburg, D. (1975). "Approximate tests and confidence intervals using the jackknife," pp. 123-147 in Perspectives in Biometrics, Elashoff, R. Ed. (Academic Press, New York).
4. Arenson, R. L., Seshadri, S. B., Kundel, H. L., DeSimone, D., Van der Voorde, F., Gefter, W. B., Epstein, D. M., Miller, W. T., Aronchick, J. M., Simson, M. B., Lanken, P. N., Khalsa, S., Brikman, I., Davey, M. and Brisbon, N. (1988). "Clinical evaluation of a medical image management system for chest images," Am. J. Roentgenol. 150, 55-59.
5. Bamber, D. (1975). "The area above the ordinal dominance graph and the area below the receiver operating characteristic graph," J. Math. Psychol. 12, 387-415.
6. Beam, C. A., Layde, P. M. and Sullivan, D. C. (1996a). "Variability in the interpretation of screening mammograms by US radiologists. Findings from a national sample check," Arch. Intern. Med. 156, 209-213.
7. Beam, C. A., Sullivan, D. C. and Layde, P. M. (1996b). "Effect of human variability on independent double reading in screening mammography," Acad. Radiol. 3, 891-897.
8. Begg, C. B. and McNeil, B. J. (1988). "Assessment of radiologic tests: Control of bias and other design considerations," Radiology 167, 565-569.
9. Beiden, S. V., Wagner, R. F. and Campbell, G. (2000). "Components-of-variance models and multiple-bootstrap experiments. An alternative method for random-effects, receiver operating characteristic analysis," Acad. Radiol. 7, 341-349.
10. Beiden, S. V., Wagner, R. F., Campbell, G. and Chan, H. P. (2001a). "Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis," Acad. Radiol. 8, 616-622.
11. Beiden, S. V., Wagner, R. F., Campbell, G., Metz, C. E. and Jiang, Y. (2001b). "Components-of-variance models for random-effects ROC analysis: The case of unequal variance structures across modalities," Acad. Radiol. 8, 605-615.
12. Beiden, S. V., Wagner, R. F., Doi, K., Nishikawa, R. M., Freedman, M., Lo, S.-C. B. and Xu, X.-W. (2002). "Independent versus sequential reading in ROC studies of computer-assist modalities: Analysis of components of variance," Acad. Radiol. 9, 1036-1043.
13. Berbaum, K. S., Dorfman, D. D. and Franken, E. A., Jr (1989). "Measuring observer performance by ROC analysis: Indications and complications," Invest. Radiol. 24, 228-233.
14. Berbaum, K. S., Franken, E. A., Jr, Dorfman, D. D., Rooholamini, S. A., Kathol, M. H., Barloon, T. J., Behlke, F. M., Sato, Y., Lu, C. H. and el-Khoury, G. Y. (1990). "Satisfaction of search in diagnostic radiology," Invest. Radiol. 25, 133-140.
15. Berbaum, K. S., Dorfman, D. D., Franken, E. A. and Caldwell, R. T. (2000). "Proper ROC analysis and joint ROC analysis of the satisfaction of search effect in chest radiology," Acad. Radiol. 7, 945-958.
16. Berbaum, K. S., Dorfman, D. D., Franken, E. A., Jr and Caldwell, R. T. (2002). "An empirical comparison of discrete ratings and subjective probability ratings," Acad. Radiol. 9, 756-763.
17. Beutel, J., Kundel, H. L. and van Metter, R. Eds. (2000). Handbook of Medical Imaging. Volume I. Physics and Psychophysics. (SPIE Optical Engineering Press, Bellingham, WA).
18. Blackwell, H. R. (1946). "Contstant thresholds of the human eye," J. Opt. Soc. Am. 36, 624-643.
19. Bunch, P. C., Hamilton, J. F., Sanderson, G. K. and Simmons, A. H. (1978). "A free-response approach to the measurement and characterization of radiographic observer performance," J. Appl. Photogr. Eng. 4, 166-171.
20. Burgess, A. (1995a). "Image quality, the ideal observer, and human performance of radiologic decision tasks," Acad. Radiol. 2, 522-526.
21. Burgess, A. E. (1995b). "Comparison of receiver operating characteristic and forced choice observer performance measurement methods," Med. Phys. 22, 643-655.
22. Burgess, A. E. and Hicken, P. (1982). "Comparative performance of x-ray-intensifying screens," Radiology 143, 551-556.
23. Burgess, A. E., Wagner, R. F., Jennings, R. J. and Barlow, H. B. (1981). "Efficiency of human visual signal discrimination," Science 214, 93-94.
24. Campbell, G. (1994). "Advances in statistical methodology for the evaluation of diagnostic and laboratory tests," Stat. Med. 13, 499-508.
25. Chakraborty, D. P. (1989). "Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data," Med. Phys. 16, 561-568.
26. Chakraborty, D. P. (2002). "Statistical power in observer-performance studies: Comparison of the receiver operating characteristic and free-response methods in tasks involving localization," Acad. Radiol. 9, 147-156.
27. Chakraborty, D. P. and Berbaum, K. S. (2004). "Observer studies involving detection and localization: modeling, analysis and validation," Med. Phys. 31, 2313-2330.
28. Chakraborty, D. P. and Winter, L. (1990). "Free-response methodology: Alternative analysis and a new observer-performance experiment," Radiology 174, 873-881.
29. Chan, H.-P., Doi, K., Vyborny, C. J., Schmidt, R. A., Metz, C. E., Lam, K. L., Ogura, T., Wu, Y. Z. and MacMahon, H. (1990). "Improvement in radiologist" detection of clustered microcalcifications on mammograms. The potential of computer-aided diagnosis," Invest. Radiol. 25, 1102 - 1110.
30. Chan, H.-P., Sahiner, B., Hadjiiski, L. M., Petrick, N. and Zhou, C. (2003). "Design of three-class classifiers in computer-aided diagnosis: Monte Carlo simulation study," pp. 567-578 in Medical Imaging 2003: Image Processing (Proc. SPIE, 5032). Sonka, M. and Fitzpatrick, J. M. Eds. (SPIE, Bellingham, WA).
31. Cochran, W. and Cox, G. (1957). Experimental Designs (John Wiley and Sons, New York).
32. Cunningham, I. A. (2000). "Applied linear systems theory," pp. 79-160 in Handbook of Medical Imaging, Vol. 1, Beutel, J., Kundel, H. L. and Van Meter, R. L. Eds. (SPIE Press, Bellingham, WA).
33. DeLong, E. R., DeLong, D. M. and Clarke-Pearson, D. L. (1988). "Comparing the areas under two or more correlated receiver operating characteristic curves: A non-parametric approach," Biometrics 44, 837-845.
34. Dobbins, J. T. (2000). "Image Quality Metrics for Digital Systems," pp. 161-222 in Handbook of Medical Imaging, Vol. 1, Beutel, J., Kundel, H. L. and Van Meter, R. L. Eds. (SPIE Press, Bellingham, WA).
35. Dorfman, D. and Alf, E. J. (1969). "Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals-rating method data," J. Math. Psychol. 6, 487-496.
36. Dorfman, D. D. and Berbaum, K. S. (2000a). "A contaminated binormal model for ROC data: Part II. A formal model," Acad. Radiol. 7, 427-437.
37. Dorfman, D. D. and Berbaum, K. S. (2000b). "A contaminated binormal model for ROC data: Part III. Initial evaluation with detection ROC data," Acad. Radiol. 7, 438-447.
38. Dorfman, D. D., Berbaum, K. S. and Metz, C. E. (1992). "Receiver operating characteristic analysis. Generalization to the population of readers and patients with the jackknife method," Invest. Radiol. 27, 723-731.
39. Dorfman, D. D., Berbaum, K. S., Metz, C. E., Lenth, R. V., Hanley, J. A. and Abu Dagga, H. (1997). "Proper receiver operating characteristic analysis: The bigamma model," Acad. Radiol. 4, 138-149.
40. Dorfman, D. D., Berbaum, K. S., Lenth, R. V., Chen, Y.-F. and Donaghy, B. A. (1998). "Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: Factorial experimental design," Acad. Radiol. 5, 591-602.
41. Dorfman, D. D., Berbaum, K. S. and Brandser, E. A. (2000). "A contaminated binormal model for ROC data: Part I. Some interesting examples of binormal degeneracy," Acad. Radiol. 7, 420-426.
42. Edwards, D. C., Kupinski, M. A., Metz, C. E. and Nishikawa, R. M. (2002). "Maximum-likelihood fitting of FROC curves under an initial-detection-and- candidate-analysis model," Med. Phys. 29, 2861-2870.
43. Edwards, D. C., Lan, L., Metz, C. E., Giger, M. L. and Nishikawa, R. M. (2004a). "Estimating three-class ideal observer decision variables for computerized detection and classification of mammographic mass lesions," Med. Phys. 31, 81-90.
44. Edwards, D. C., Metz, C. E. and Kupinski, M. A. (2004b). "Ideal observers and optimal ROC hypersurfaces in N-class classification," IEEE Trans. Med. Imaging 23, 891-895.
45. Edwards, D. C., Metz, C. E. and Nishikawa, R. M. (2005). "The hypervolume under the ROC hyper-surface of "Near-Guessing" and "Near-Perfect" observers in N-class classification tasks," IEEE Trans. Med. Imaging 24, 293-299.
46. Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap (Chapman & Hall, London).
47. Egan, J. P. (1975). Signal Detection Theory and ROC Analysis (Academic Press, New York).
48. Egan, J. P., Greenberg, G. Z. and Schulman, A. I. (1961). "Operating characteristics, signal detectability, and the method of free response," J. Acoust. Soc. Am. 33, 993-1007.
49. Elmore, J. G., Miglioretti, D. L., Reisch, L. M., Barton, M. B., Kreuter, W., Christiansen, C. L. and Fletcher, S. W. (2002). "Screening mammograms by community radiologists: Variability in false-positive rates," J. Natl. Cancer Inst. 94, 1373-1380.
50. Eng, J., Mysko, W. K., Weller, G. E. R., Renard, R., Gitlin, J. N., Bluemke, D. A., Magid, D., Kelen, G. D. and Scott, W. W., Jr (2000). "Interpretation of emergency department radiographs. A comparison of emergency medicine physicians with radiologists, residents with faculty, and film with digital display," Am. J. Roentgenol. 175, 1233-1238.
51. Esserman, L., Cowley, H., Eberle, C., Kirkpatrick, A., Chang, S., Berbaum, K. and Gale, A. (2002). "Improving the accuracy of mammography: Volume and outcome relationships," J. Natl. Cancer Inst. 94, 369-375.
52. Fearon, T., Vucich, J., McSweeney, W. J., Potter, B. M., Brallier, D. R., McIlhenny, J., Tepper, J. and Markle, B. M. (1986). "A comparative evaluation of rare earth screen-film systems. Free-response operating characteristic analysis and anatomic criteria analysis," Invest. Radiol. 21, 734-742.
53. Fryback, D. G. and Thornbury, J. R. (1991). "The efficacy of diagnostic imaging," Med. Decis. Making 11, 88-94.
54. Garland, L. H. (1959). "Studies on the accuracy of diagnostic procedures," Am. J. Roentgenol. Radium Ther. Nucl. Med. 82, 25-38.
55. Gold, M. R., Siegel, J. E., Russell, L. B. and Weinstein, M. C. (1996). Cost-effectiveness in Health and Medicine (Oxford University Press, Oxford).
56. Good, W. F., Sumkin, J. H., Dash, N., Johns, C. M., Zuley, M. L., Rockett, H. E. and Gur, D. (1999). "Observer sensitivity to small differences: A multipoint rank-order experiment," Am. J. Roentgenol. 173, 275-278.
57. Gray, R., Begg, C. B. and Greenes, R. A. (1984). "Construction of receiver operating characteristic curves when disease verification is subject to selection bias," Med. Dec. Making 4, 151-164.
58. Green, D. M. and Swets, J. A. (1974). Signal Detection Theory and Psychophysics (Krieger, Huntington, NY).
59. Greenes, R. A. and Begg, C. B. (1985). "Assessment of diagnostic technologies; Methodology for unbiased estimation from samples of selectively verified patients," Invest. Radiol. 20, 751-756.
60. Grey, D. R. and Morgan, B. J. T. (1972). "Some aspects of ROC curve-fitting: Normal and logistic models," J. Math. Psychol. 9, 128-139.
61. Gur, D., King, J. L., Rockette, H. E., Britton, C. A., Thaete, F. L. and Hoy, R. J. (1990). "Practical issues of experimental ROC analysis. Selection of controls," Invest. Radiol. 25, 583-586.
62. Gurcan, M. N., Sahiner, B., Petrick, N., Chan, H. P., Kazerooni, E. A., Cascade, P. N. and Hadjiiski, L. (2002). "Lung nodule detection on thoracic computed tomography images: Preliminary evaluation of a computer-aided diagnosis system," Med. Phys. 29, 2552-2558.
63. Hacker, M. J. and Ratcliff, R. (1979). "A revised table of d' for M-alternative forced choice," Percep. Psychophys. 26, 168-170.
64. Hajian-Tilaki, K. O., Hanley, J. A., Joseph, L. and Collet, J.-P. (1997). "A comparison of parametric and non-parametric approaches to ROC analysis of quantitative diagnostic tests," Med. Dec. Making 17, 94-102.
65. Halpern, E. J., Albert, M., Krieger, A. M., Metz, C. E. and Maidment, A. D. (1996). "Comparison of receiver operating characteristic curves on the basis of optimal operating points," Acad. Radiol. 3, 245-253.
66. Hanley, J. A. (1988). "The robustness of the "binormal" assumptions used in fitting ROC curves," Med. Dec. Making 8, 197-203.
67. Hanley, J. A. (1989a). "The place of statistical methods in radiology (and in the bigger picture)," Invest. Radiol. 24, 10-16.
68. Hanley, J. A. (1989b). "Receiver operating characteristic (ROC) methodology: The state of the art," Crit. Rev. Diagn. Imaging 29, 307-335.
69. Hanley, J. A. and McNeil, B. J. (1982). "The meaning and use of the area under a receiver operating characteristic (ROC) curve," Radiology 143, 29-36.
70. Hanley, J. A. and McNeil, B. J. (1983). "A method of comparing the areas under receiver operating characteristic curves derived from the same cases," Radiology 148, 839-843.
71. Hedges, L. V. and Olkin, I. (1985). Statistical Methods for Meta-Analysis (Academic Press, New York).
72. Hillis, S. L. and Berbaum, K. S. (2004). "Power estimation for the Dorfman-Berbaum-Metz method," Acad. Radiol. 11, 1260-1273.
73. Hillis, S. L., Obuchowski, N. A., Schartz, K. M. and Berbaum, K. S. (2005). "A comparison of the Dorfman- Berbaum-Metz and the Obuchowski-Rockett methods for receiver operating characteristic (ROC) data," Stat. Med. 24, 1579-1607.
74. Hillman, B. J. (1994). "Outcomes research and cost-effectiveness analysis for diagnostic imaging," Radiology 193, 307-310.
75. Hillman, B. J., Hessel, S. J., Swensson, R. G. and Herman, P. G. (1977). "Improving diagnostic accuracy: A comparison of interactive and Delphi consultations," Invest. Radiol. 12, 112-115.
76. Houn, F., Bright, R. A., Bushar, H. F., Croft, B. Y., Finder, C. A., Gohagan, J. K., Jennings, R. J., Keegan, P., Kessler, L. G., Kramer, B. S., Martynec, L. O., Robinowitz, M., Sacks, W. M., Schultz, D. G. and Wagner, R. F. (2000). "Study design in the evaluation of breast cancer imaging technologies," Acad. Radiol. 7, 684-692.
77. Hrung, J. M., Langlotz, C. P., Orel, S. G., Fox, K. R., Schnall, M. D. and Schwartz, J. S. (1999). "Cost-effectiveness of MR imaging and core-needle biopsy in the preoperative work-up of suspicious breast lesions," Radiology 213, 39-49.
78. IAEA (1977). "IAEA co-ordinated research programme on the intercomparison of computer-assisted scintigraphic techniques: third progress report," p. 585 in Medical Radionuclide Imaging, Vol. 1. (International Atomic Energy Agency, Vienna).
79. ICRU (1996). Medical Imaging: The Assessment of Image Quality, International Commission on Radiation Units and Measurements Report No. 54 (Oxford University Press, Oxford).
80. Irwig, L., Tosteson, A. N. A., Gatsonis, C., Lau, J., Colditz, G., Chalmers, T. C. and Mosteller, F. (1994). "Guidelines for meta-analyses evaluating diagnostic tests," Ann. Intern. Med. 120, 667-676.
81. Jiang, Y., Metz, C. E. and Nishikawa, R. M. (1996). "A receiver operating characteristic partial area index for highly selective diagnostic tests," Radiology 201, 745-750.
82. Jiang, Y., Nishikawa, R. M., Schmidt, R. A., Metz, C. E., Giger, M. L. and Doi, K. (1999). "Improving breast cancer diagnosis with computer-aided diagnosis," Acad. Radiol. 6, 22-33.
83. Kido, S., Ikezoe, J., Naito, H., Arisawa, J., Tamura, S., Kozuka, T., Ito, W., Shimura, K. and Kato, H. (1995). "Clinical evaluation of pulmonary nodules with single-exposure dual-energy subtraction chest radiography with an iterative noise-reduction algorithm," Radiology 194, 407-412.
84. Kijewski, M. F., Swensson, R. G. and Judy, P. F. (1989). "Analysis of rating data from multiple-alternative tasks," J. Math. Psychol. 33, 428-451.
85. King, J. L., Britton, C. A., Gur, D., Rockette, H. E. and Davis, P. L. (1993). "On the validity of the continuous and discrete confidence rating scales in receiver operating characteristic studies," Invest. Radiol. 28, 962-963.
86. Kobayashi, T., Xu, X.-W., MacMahon, H., Metz, C. E. and Doi, K. (1996). "Effect of a computer-aided diagnosis scheme on radiologists' performance in detection of lung nodules on chest radiographs," Radiology 199, 843-848.
87. Kool, L. J., Busscher, D. L., Vlasbloem, H., Hermans, J., vd Merwe, P. C., Algra, P. R. and Herstel, W. (1988). "Advanced multiple-beam equalization radiography in chest radiology: A simulated nodule detection study," Radiology 169, 35-39.
88. Koran, L. M. (1975a). "The reliability of clinical methods, data and judgments (first of two parts)," N. Eng. J. Med. 293, 642-646.
89. Koran, L. M. (1975b). "The reliability of clinical methods, data and judgments (second of two parts)," N. Eng. J. Med. 293, 695-701.
90. Kundel, H. L. and Revesz, G. (1974). "The evaluation of radiographic techniques by observer tests: Problems, pitfalls, and procedures," Invest. Radiol. 9, 166-173.
91. Kundel, H. L., Revesz, G. and Stauffer, H. M. (1968). "Evaluation of a television image processing system," Invest. Radiol. 3, 44-50.
92. Kundel, H. L., Revesz, G. and Stauffer, H. M. (1969). "The electro-optical processing of radiographic images," Radiol. Clin. North Am. 7, 447-460.
93. Kundel, H. L., Gefter, W., Aronchick, J., Miller, W., Jr, Hatabu, H., Whitfill, C. and Miller, W., Sr. (1997). "Accuracy of bedside chest hard-copy screen-film versus hard- and soft-copy computed radiographs in a medical intensive care unit: Receiver operating characteristic analysis," Radiology 205, 859-863.
94. Kundel, H. L., Polansky, M., Dalinka, M. K., Choplin, R. H., Gefter, W. B., Kneeland, J. B., Miller, W. T., Sr. and Miller, W. T., Jr (2001). "Reliability of soft-copy versus hard-copy interpretation of emergency department radiographs: A prototype study," Am. J. Roentgenol. 177, 525-528.
95. LeLorier, J., Gregoire, G., Benhaddad, A., Lapierre, J. and Derderian, F. (1997). "Discrepancies between meta-analyses and subsequent large randomized, controlled trials," N. Eng. J. Med. 337, 536-542.
96. Loop, J. W. and Lusted, L. B. (1978). "American College of Radiology diagnostic efficacy studies," Am. J. Roentgenol. 131, 173-179.
97. Lusted, L. B. (1960). "Logical analysis in roentgen diagnosis," Radiology 74, 178-193.
98. Lusted, L. B. (1968). Introduction to Medical Decision Making (Charles C. Thomas, Springfield).
99. Lusted, L. B. (1969). "Perception of the roentgen image: Applications of signal detection theory," Radiol. Clin. North Am. 7, 435-445.
100. Lusted, L. B. (1971). "Signal detectability and medical decision making," Science 171, 1217-1219.
101. MacMahon, H., Vyborny, C. J., Metz, C. E., Doi, K., Sabeti, V. and Solomon, S. L. (1986). "Digital radiography of subtle pulmonary abnormalities: An ROC study of the effect of pixel size on observer performance," Radiology 158, 21-26.
102. MacMahon, H., Metz, C. E., Doi, K., Kim, T., Giger, M. L. and Chan, H. P. (1988). "Digital chest radiography: Effect on diagnostic accuracy of hard copy, conventional video, and reversed gray scale video display formats," Radiology 168, 669-673.
103. MacMahon, H., Engelmann, R., Behlen, F. M., Hoffmann, K. R., Ishida, T., Roe, C., Metz, C. E. and Doi, K. (1999). "Computer-aided diagnosis of pulmonary nodules: Results of a large-scale observer test," Radiology 213, 723-726.
104. Macmillan, N. A. and Creelman, C. D. (1991). Detection Theory: A User's Guide (Cambridge University Press, Cambridge).
105. McClish, D. K. (1987). "Comparing the areas under more than two independent ROC curves," Med. Decis. Making 7, 149-155.
106. McClish, D. K. (1989). "Analyzing a portion of an ROC curve," Med. Dec. Making 9, 190-195.
107. McNeil, B. J. and Adelstein, S. J. (1976). "Determining the value of diagnostic and screening tests," J. Nucl. Med. 17, 439-448.
108. McNeil, B. J. and Hanley, J. A. (1984). "Statistical approaches to the analysis of receiver operating characteristic (ROC) curves," Med. Dec. Making 4, 137-150.
109. McNeil, B. J., Keeler, E. and Adelstein, S. J. (1975). "Primer on certain elements of medical decision making," N. Engl. J. Med., 293, 211-215.
110. Mehrez, A. and Gafni, A. (1989). "Quality-adjusted life years, utility theory, and healthy-years equivalents," Med. Dec. Making 9, 142-149.
111. Mendez, A. J., Tahoces, P. G., Lado, M. J., Sauto, M. and Vidal, J. J. (1998). "Computer-aided diagnosis: Automatic detection of malignant masses in digitized mammograms," Med. Phys. 25, 957-964.
112. Metz, C. E. (1978). "Basic principles of ROC analysis," Semin. Nucl. Med. 8, 283-298.
113. Metz, C. E. (1986). "ROC methodology in radiologic imaging," Invest. Radiol. 21, 720-723.
114. Metz, C. E. (1989). "Some practical issues of experimental design and data analysis in radiological ROC studies," Invest. Radiol. 24, 234-245.
115. Metz, C. E. (1993). "Quantification of failure to demonstrate statistical significance. The usefulness of confidence intervals," Invest. Radiol. 28, 59-63.
116. Metz, C. E. (2000). "Fundamental ROC analysis," pp. 751-769 in Handbook of Medical Imaging. Vol. 1. Beutel, J., Kundel, H. L. and Van Meter, R. L. Eds. (SPIE Press, Bellingham, WA).
117. Metz, C. E. and Pan, X. (1999). "'Proper' binormal ROC curves: Theory and maximum- likelihood estimation," J. Math. Psychol. 43, 1-33.
118. Metz, C., Wang, P. and Kronman, H. (1984). "A new approach for testing the significance of differences between ROC curves measured from correlated data," pp. 432-445 in Information Processing in Medical Imaging, Deconink, F. Ed. (Nijhoff, The Hague).
119. Metz, C. E., Wagner, R. F., Doi, K., Brown, D. G., Nishikawa, R. M. and Myers, K. J. (1995). "Toward consensus on quantitative assessment of medical imaging systems," Med. Phys. 22, 1057-1061.
120. Metz, C. E., Herman, B. A. and Shen, J. H. (1998). "Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data," Stat. Med. 17, 1033-1053.
121. Miettunen, R., Korhola, O., Bondestam, S., Standertskjold-Nordenstam, C. G., Lamminen, A., Somer, K. and Soiva, M. (1991). "Combination of multiple pencil-beam imaging to computed storage phosphor radiography: A new method," Eur. J. Radiol. 12, 161-166.
122. Morgan, R. H., Donner, M. W., Gaylor, B. W., Margulies, S. I., Rao, P. S. and Wheeler, P. S. (1973). "Decision processes and observer error in the diagnosis of pneumoconiosis by chest Roentgenography," Am. J. Roentgenol. 117, 757-764.
123. Moses, L. E., Shapiro, D. and Littenberg, B. (1993). "Combining independent studies of a diagnostic test into a summary ROC curve: Data-analytic approaches and some additional considerations," Stat. Med. 12, 1293-1316.
124. Mossman, D. (1999). "Three-way ROCs," Med. Decis. Making 19, 78-89.
125. Myers, K. J. (2000). "Ideal observer models of visual signal detection," pp. 559-592 in Handbook of Medical Imaging, Vol. 1, Beutel, J., Kundel, H. L. and Van Meter, R. L. Eds. (SPIE Press, Bellingham, WA).
126. NCRP (1995). An Introduction to Efficacy in Diagnostic Radiology and Nuclear Medicine. National Council on Radiation Protection and Measurements Commentary No. 13 (National Council on Radiation Protection and Measurements, Bethesda).
127. Nishikawa, R. M., Giger, M. L., Doi, K., Metz, C. E., Yin, F. F., Vyborny, C. J. and Schmidt, R. A. (1994). "Effect of case selection on the performance of computer-aided detection schemes," Med. Phys. 21, 265-269.
128. Nodine, C. F. and Krupinski, E. A. (1998). "Perceptual skill, radiology expertise, and visual test performance with NINA and WALDO," Acad. Radiol. 5, 603-612.
129. Nodine, C. F., Kundel, H. L., Mello-Thoms, C., Weinstein, S. P., Orel, S. G., Sullivan, D. C. and Conant, E. F. (1999). "How experience and training influence mammography expertise," Acad. Radiol. 6, 575-585.
130. Obuchowski, N. A. (2000). "Sample size tables for receiver operating characteristic studies," Am. J. Roentgenol. 175, 603-608.
131. Obuchowski, N. A. (2005). "Fundamentals of clinical research for radiologists: ROC Analysis," Am. J. Roentgenol. 184, 364-372.
132. Obuchowski, N. and Rockette, H. E. (1995). "Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: An ANOVA approach with dependent observations," Commun. Statist. Simula Comp. 24, 285-308.
133. Obuchowski, N. A., Lieber, M. L. and Powell, K. A. (2000). "Data analysis for detection and localization of multiple abnormalities with application to mammography," Acad. Radiol. 7, 516-525.
134. Obuchowski, N. A., Beiden, S. V., Berbaum, K. S., Hillis, S. L., Ishwaran, H., Song, H. H. and Wagner, R. F. (2004). "Multireader, multicase receiver operating characteristic analysis: An empirical comparison of five methods," Acad. Radiol. 11, 980-995.
135. Ohara, K., Chan, H. P., Doi, K., Giger, M. L. and Fujita, H. (1986). "Investigation of basic imaging properties in digital radiography. 8. Detection of simulated low-contrast objects in digital subtraction angiographic images," Med Phys. 13, 304-311.
136. Pan, X. and Metz, C. E. (1997). "The 'proper' binormal model: parametric receiver operating characteristic curve estimation with degenerate data," Acad. Radiol. 4, 380-389.
137. Peterson, W. W., Birdsall, T. L. and Fox, W. C. (1954). "The theory of signal detectability," IRE Prof Gp Inf. Theory (Trans.) PGIT-4, 171-212.
138. Petrick, N., Chan, H. P., Wei, D., Sahiner, B., Helvie, M. A. and Adler, D. D. (1996). "Automated detection of breast masses on mammograms using adaptive contrast enhancement and texture classification," Med. Phys. 23, 1685-1696.
139. Quenouille, M. (1949). "Approximate tests of correlation in time series," J. R. Stat. Soc. Ser. B 11, 68-84.
140. Quenouille, M. (1956). "Notes on bias in estimation," Biometrika 43, 353-360.
141. Raab, S. S., Thomas, P. A., Lenel, J. C., Bottles, K., Fitzsimmons, K. M., Zaleski, M. S., Wittchow, R. J., McPhaul, L. W., Slagel, D. D. and Cohen, M. B. (1995). "Pathology and probability. Likelihood ratios and receiver operating characteristic curves in the interpretation of bronchial brush specimens," Am. J. Clin. Pathol. 103, 588-593.
142. Ransohoff, D. F. and Feinstein, A. R. (1978). "Problems of spectrum and bias in evaluating the efficacy of diagnostic tests," N. Eng. J. Med. 299, 926-930.
143. Revesz, G., Kundel, H. L. and Bonitatibus, M. (1983). "The effect of verification on the assessment of imaging techniques," Invest. Radiol. 18, 194-198.
144. Rey, W. (1983). Introduction to Robust and Quasi-robust Statistical Methods (Springer, New York).
145. Rockette, H. E., Gur, D. and Metz, C. E. (1992). "The use of continuous and discrete confidence judgments in receiver operating studies of diagnostic imaging techniques," Invest. Radiol. 27, 169-172.
146. Rockette, H. E., Gur, D., Campbell, W. L. and Thaete, F. L. (1994). "Use of meta-analysis in the evaluation of imaging systems," Acad. Radiol. 1, 63-69.
147. Rockette, H. E., King, J. L., Thaete, F. L., Fuhrman, C. R., Slifko, R. M. and Gur, D. (1998). "Selection of subtle cases for observer-performance studies: The importance of knowing the true diagnosis," Acad. Radiol. 5, 86-92.
148. Rockette, H. E., Li, W., Brown, M. L., Britton, C. A., Towers, J. T. and Gur, D. (2001). "Statistical test to assess rank-order imaging studies," Acad. Radiol. 8, 24-30.
149. Roe, C. A. and Metz, C. E. (1997a). "Dorfman-Berbaum- Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: Validation with computer simulation," Acad. Radiol. 4, 298-303.
150. Roe, C. A. and Metz, C. E. (1997b). "Variance-component modeling in the analysis of receiver operating characteristic index estimates," Acad. Radiol. 4, 587-600.
151. Rossmann, K. (1966). "Comparison of several methods for evaluating image quality of radiographic screen-film systems," Am. J. Roentgenol. 97, 772-775.
152. Sainfort, F. (1991). "Evaluation of medical technologies: A generalized ROC analysis," Med. Decis. Making 11, 208-220.
153. Samuel, S., Kundel, H. L., Nodine, C. F. and Toto, L. C. (1995). "Mechanism of satisfaction of search: Eye position recordings in the reading of chest radiographs," Radiology 194, 895-902.
154. Schaefer, C. M., Prokop, M., Oestmann, J. W., Wiesmann, W., Haubitz, B., Meschede, A., Reichelt, S., Schirg, E., Stender, H. S. and Galanski, M. (1992). "Impact of hard-copy size on observer performance in digital chest radiography," Radiology 184, 77-81.
155. Scott, W. W., Jr, Bluemke, D. A., Mysko, W. K., Weller, G. E., Kelen, G. D., Reichle, R. L., Weller, J. C. and Gitlin, J. N. (1995). "Interpretion of emergency department radiographs by radiologists and emergency medicine physicians: Teleradiology workstation versus radiograph readings," Radiology 195, 223-229.
156. Scurfield, B. K. (1996). "Multiple-event forced-choice tasks in the theory of signal detectability," J. Math Psychol. 40, 253-269.
157. Scurfield, B. K. (1998). "Generalization of the theory of signal detectability to n-event m-dimensional forced-choice tasks," J. Math Psychol. 42, 5-31.
158. Slasky, B. S., Gur, D., Good, W. F., Costa-Greco, M. A., Harris, K. M., Cooperstein, L. A. and Rockette, H. E. (1990). "Receiver operating characteristic analysis of chest image interpretation with conventional, laser-printed, and high-resolution workstation images," Radiology 174, 775-780.
159. Sorenson, J. A., Mitchell, C. R., Armstrong, J. D., II, Mann, H., Bragg, D. G., Mann, F. A., Tocino, I. B. and Wojtowycz, M. M. (1987). "Effects of improved contrast on lung-nodule detection. A clinical ROC study," Radiology 22, 772-780.
160. Starr, S. J., Metz, C. E., Lusted, L. B. and Goodenough, D. J. (1975). "Visual detection and localization of radiographic images," Radiology 116, 533-538.
161. Swensson, R. G. (1996). "Unified measurement of observer performance in detecting and localizing target objects on images," Med . Phys. 23, 1709-1725.
162. Swensson, R. G., King, J. L. and Gur, D. (2001). "A constrained formulation for the receiver operating characteristic (ROC) curve based on probability summation," Med. Phys. 28, 1597-1608.
163. Swets, J. A. (1986). "Indices of discrimination or diagnostic accuracy: Their ROCs and implied models," Psychol. Bul. 99, 100-117.
164. Swets, J. A. (1988). "Measuring the accuracy of diagnostic systems," Science 240, 1285-1293.
165. Swets, J. A. and Pickett, R. M. (1982). Evaluation of Diagnostic Systems. Methods from Signal Detection Theory (Academic Press, New York).
166. Swets, J. A., Pickett, R. M., Whitehead, S. F., Getty, D. J., Schnur, J. A., Swets, J. B. and Freeman, B. A. (1979). "Assessment of diagnostic technologies," Science 205, 753-759.
167. Tanner, W. P., Jr and Swets, J. A. (1954). "A decision-making theory of visual detection," Psychol. Rev. 61, 401-409.
168. Testa, M. A. and Simonson, D. C. (1996). "Assessment of quality-of-life outcomes," N. Eng. J. Med. 334, 835-840.
169. Thornbury, J. R., Fryback, D. G. and Edwards, W. (1975). "Likelihood ratios as a measure of the diagnostic usefulness of excretory urogram information," Radiology 114, 561-565.
170. Toledano, A. Y. and Gatsonis, C. (1995). "Regression analysis of correlated ROC curves," Acad. Radiol. 2, S30-S36.
171. Toledano, A. and Gatsonis, C. (1996). "Ordinal regression methodology for ROC curves derived from correlated data," Stat. Med. 15, 1807-1826.
172. Tourassi, G. D. and Floyd, C. E., Jr (1993). "Artificial neural networks for single-photon emission computed-tomography: A study of cold lesion detection and localization," Invest. Radiol. 28, 671-677.
173. Tukey, J. (1958). "Bias and confidence in not quite large samples," Ann. Math. Statist. 29, 614 (Abstract).
174. Wagner, R. F., Beiden, S. V. and Metz, C. E. (2001). "Continuous versus categorical data for ROC analysis: Some quantitative considerations, " Acad. Radiol. 8, 328-334.
175. Wagner, R. F., Beiden, S. V., Campbell, G., Metz, C. E. and Sacks, W. M. (2002). "Assessment of medical imaging and computer assisted systems," Acad. Radiol. 9, 1264-1277.
176. Wald, A. (1950). Statistical Decision Functions (Wiley, New York).
177. Wieand, S., Gail, M. H., James, B. R. and James, K. L. (1989). "A family of nonparametric statistics for comparing diagnostic markers with paired and unpaired data," Biometrika 76, 585-592.
178. Winer, B. (1971). Statistical Principles in Experimental Design (McGraw-Hill, New York).
179. Xu, X. W., Doi, K., Kobayashi, T., MacMahon, H. and Giger, M. L. (1997). "Development of an improved CAD scheme for automated detection of lung nodules in digital chest images," Med. Phys. 24, 1395-1403.
180. Yoshimura, H., Giger, M. L., Doi, K., MacMahon, H. and Montner, S. M. (1992). "Computerized scheme for the detection of pulmonary nodules. A nonlinear filtering technique," Invest. Radiol. 27, 124-129.
181. Zhou, X.-H., Obuchowski, N. A. and McClish, D. K. (2002). Statistical Methods in Diagnostic Medicine (John Wiley and Sons, New York).
182. Zweig, M. H. and Campbell, G. (1993). "Receiver-operating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine," Clin. Chem. 39, 561-577.