New developments in observer performance methodology in medical imaging

Seminars in Nuclear Medicine

Volumn 41, Issue 6, 2011, Pages 401-418

  Chakraborty, Dev P ^a  

^a UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; DECISION MAKING; DIAGNOSTIC IMAGING; FALSE POSITIVE RESULT; IMAGE ANALYSIS; IMAGE DISPLAY; IMAGE PROCESSING; IMAGE QUALITY; METHODOLOGY; NUCLEAR MEDICINE; RADIOLOGY; RATING SCALE; RECEIVER OPERATING CHARACTERISTIC; SAMPLE SIZE; SIGNAL NOISE RATIO;

EID: 80053537573     PISSN: 00012998     EISSN: 15584623     Source Type: Journal    
DOI: 10.1053/j.semnuclmed.2011.07.001     Document Type: Review

References (74)

1. C.E. Metz Basic principles of ROC analysis Semin Nucl Med 8 1978 283 298 (Pubitemid 9075026)
2. C.E. Metz ROC methodology in radiologic imaging Invest Radiol 21 1986 720 733 (Pubitemid 16015664)
3. C.E. Metz Some practical issues of experimental design and data analysis in radiological ROC studies Invest Radiol 24 1989 234 245 (Pubitemid 19084791)
4. M.H. Zweig, and G. Campbell Receiver-operating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine Clin Chem 39 1993 561 577 (Pubitemid 23118304)
5. H.L. Kundel, K.S. Berbaum, and D.D. Dorfman Receiver operating characteristic analysis in medical imaging J Intl Commission Radiation Units Measure 8 2008 1 62
6. C. Metz ROC analysis in medical imaging: A tutorial review of the literature Radiol Phys Technol 1 2008 2 12
7. J. Shiraishi, L.L. Pesce, and C.E. Metz Experimental design and data analysis in receiver operating characteristic studies: Lessons learned from reports in radiology from 1997 to 2006 Radiology 253 2009 822 830
8. P.C. Bunch, J.F. Hamilton, and G.K. Sanderson A free-response approach to the measurement and characterization of radiographic-observer performance J Appl Photogr Eng 4 1978 166 171 (Pubitemid 9408892)
9. J.P. Egan, G.Z. Greenburg, and A.I. Schulman Operating characteristics, signal detectability and the method of free response J Acoust Soc Am 33 1961 993 1007
10. H. Miller The FROC curve: A representation of the observer's performance for the method of free response J Acoust Soc Am 46 1969 1473 1476
11. J.I. Marcum A Statistical Theory of Target Detection by Pulsed Radar 1947 RAND Corporation Santa Monica, CA RM-754-PR
12. Y. Jiang, C.E. Metz, and R.M. Nishikawa A receiver operating characteristic partial area index for highly sensitive diagnostic tests Radiology 201 1996 745 750 (Pubitemid 26392624)
13. D.K. McClish Analyzing a portion of the ROC curve Med Decis Mak 9 1989 190 195
14. D.M. Green, and J.A. Swets Signal Detection Theory and Psychophysics 1966 John Wiley & Sons New York
15. C.A. Beam, P.M. Layde, and D.C. Sullivan Variability in the interpretation of screening mammograms by US radiologists Findings from a national sample Arch Intern Med 156 1996 209 213 (Pubitemid 26024757)
16. D.D. Dorfman, and E. Alf Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervalsrating-method data J Math Psychol 6 1969 487 496
17. C.E. Metz, B.A. Herman, and J.H. Shen Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data Stat Med 17 1998 1033 1053 (Pubitemid 28203832)
18. D.D. Dorfman, and K.S. Berbaum A contaminated binormal model for ROC data: Part III Initial evaluation with detection ROC data Acad Radiol 7 2000 438 447 (Pubitemid 30341576)
19. D.D. Dorfman, K.S. Berbaum, and C.E. Metz Proper receiver operating characteristic analysis: The bigamma model Acad Radiol 4 1997 138 149 (Pubitemid 127469601)
20. C.E. Metz, and X. Pan "Proper" binormal ROC curves: Theory and maximum-likelihood Estimation J Math Psychol 43 1999 1 33
21. A.E. Burgess Comparison of receiver operating characteristic and forced choice observer performance measurement methods Med Phys 22 1995 643 655
22. J. Eng ROC analysis: Web-based calculator for ROC curves, 2006 http://www.rad.jhmi.edu/jeng/javarad/roc/JROCFITi.html Accessed July 13, 2011
23. L.L. Pesce, and C.E. Metz Reliable and computationally efficient maximum-likelihood estimation of "proper" binormal ROC curves Acad Radiol 14 2007 814 829 (Pubitemid 46898850)
24. B.J. McNeil, and J.A. Hanley Statistical approaches to the analysis of receiver operating characteristic (ROC) curves Med Decis Mak 4 1984 137 150 (Pubitemid 14001462)
25. J.A. Hanley The place of statistical methods in radiology (and in the bigger picture) JAMA 24 1988 10 16 (Pubitemid 19035147)
26. Chakraborty DP, Krupinski EA, Short Course: Statistical Methods in Bioengineering and ROC and FROC Analyses, 2011, Orlando, FL
27. H.E. Rockette, D. Gur, and C.E. Metz The use of continous and discrete confidence judgments in receiver operating characteristic studies of diagnostic imaging techniques Investig Radiol 27 1992 169 172
28. R.F. Wagner, S.V. Beiden, and C.E. Metz Continuous versus categorical data for ROC analysis: Some quantitative considerations Acad Radiol 8 2001 328 334 (Pubitemid 32666455)
29. C.E. Metz, B.A. Herman, and J.H. Shen Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data Stat Med 17 1998 1033 1053 (Pubitemid 28203832)
30. K.S. Berbaum, D.D. Dorfman, and E.A. Franken An empirical comparison of discrete ratings and subjective probability ratings Acad Radiol 9 2002 756 763 (Pubitemid 34761357)
31. C.J. D'Orsi, L.W. Bassett, and S.A. Feig Illustrated Breast Imaging Reporting and Data System 1998 American College of Radiology Reston, VA
32. C.J. D'Orsi, L.W. Bassett, and W.A. Berg ACR BI-RADS-Mammography 2003 American College of Radiology Reston, VA
33. W.E. Barlow, C. Chi, and P.A. Carney Accuracy of screening mammography interpretation by characteristics of radiologists J Natl Cancer Inst 96 2004 1840 1850 (Pubitemid 40081321)
34. Y. Jiang, and C.E. Metz BI-RADS data should not be used to estimate ROC curves Radiology 256 2010 29 31
35. B. Efron The Jackknife, the Bootstrap and Other Resampling Plans 1982 Capital City Publishing Group Montpelier
36. D.D. Dorfman, K.S. Berbaum, and C.E. Metz Receiver operating characteristic rating analysis Generalization to the population of readers and patients with the jackknife method Invest Radiol 27 1992 723 731
37. Report 79 Receiver Operating Characteristic Analysis in Medical Imaging http://www.oxfordjournals.org/our-journals/jicru/backissues/reports.html
38. N.A. Obuchowski Multireader, multimodality receiver operating characteristic curve studies: Hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations Acad Radiol 2 Suppl 1 1995 S22 S29
39. N.A. Obuchowski Sample size calculations in studies of test accuracy Stat Methods Med Res 7 1998 371 392 (Pubitemid 28559255)
40. N.A. Obuchowski Sample size tables for receiver operating characteristic studies AJR Am J Roentgenol 175 2000 603 608
41. S.L. Hillis, and K.S. Berbaum Power estimation for the Dorfman-Berbaum-Metz method Acad Radiol 11 2004 1260 1273
42. S.J. Starr, C.E. Metz, and L.B. Lusted Visual detection and localization of radiographic images Radiology 116 1975 533 538
43. R.G. Swensson Unified measurement of observer performance in detecting and localizing target objects on images Med Phys 23 1996 1709 1725
44. R.G. Swensson Using localization data from image interpretations to improve estimates of performance accuracy Med Decis Mak 20 2000 170 184
45. D. Bamber The area above the ordinal dominance graph and the area below the receiver operating characteristic graph J Math Psychol 12 1975 387 415
46. L.M. Popescu Nonparametric ROC and LROC analysis Med Phys 35 2007 1556 1564 (Pubitemid 46669152)
47. N.A. Obuchowski, M.L. Lieber, and K.A. Powell Data analysis for detection and localization of multiple abnormalities with application to mammography Acad Radiol 7 2000 516 525 (Pubitemid 30432337)
48. D. Chakraborty, H.J. Yoon, and C. Mello-Thoms Spatial localization accuracy of radiologists in free-response studies: inferring perceptual froc curves from mark-rating Data Acad Radiol 14 2007 4 18 (Pubitemid 44909513)
49. D.P. Chakraborty Recent developments in imaging system assessment methodology, FROC analysis and the search model Nucl Instrum Methods A Phys Res A 648 2011 S297 S301
50. D.P. Chakraborty Maximum Likelihood analysis of free-response receiver operating characteristic (FROC) data Med Phys 16 1989 561 568 (Pubitemid 19224306)
51. D.P. Chakraborty, and L.H. Winter Free-response methodology: alternate analysis and a new observer-performance experiment Radiology 174 1990 873 881 (Pubitemid 20089690)
52. B.M. Geller, W.E. Barlow, and R. Ballard-Barbash Use of the American College of Radiology BI-RADS to report on the mammographic evaluation of women with signs and symptoms of breast disease Radiology 222 2002 536 542 (Pubitemid 34106132)
53. H.L. Kundel, and C.F. Nodine A visual concept shapes image perception Radiology 146 1983 363 368 (Pubitemid 13225474)
54. H.L. Kundel, and C.F. Nodine Modeling visual search during mammogram viewing Proc SPIE 5372 2004 110 115 (Pubitemid 40108791)
55. C.F. Nodine, and H.L. Kundel Using eye movements to study visual search and to improve tumor detection Radiographics 7 1987 1241 1250
56. H.L. Kundel, C.F. Nodine, and E.F. Conant Holistic Component of Image Perception in Mammogram Interpretation: Gaze-tracking Study Radiology 242 2007 396 402 (Pubitemid 46146192)
57. I. Rock The Logic of Perception 1983 MIT Press Cambridge, MA
58. I. Rock Perception 1984 W H Freeman and Company New York
59. R.L. Gregory The Intelligent Eye 1970 McGraw-Hill New York
60. J.M. Wolfe Guided. Search 2.0: A revised model of visual search Psychonomic Bull Rev 1 1994 202 238
61. Chakraborty DP, Svahn T: Estimating the parameters of a model of visual search from ROC data: An alternate method for fitting proper ROC curves. Paper presented at Medical Imaging 2011: Image Perception,Observer Performance, and Technology Assessment, February 16, 2011; Lake Buena Vista, FL
62. D.P. Chakraborty A search model and figure of merit for observer data acquired according to the free-response paradigm Phys Med Biol 51 2006 3449 3462 (Pubitemid 44026411)
63. D.P. Chakraborty ROC Curves predicted by a model of visual search Phys Med Biol 51 2006 3463 3482 (Pubitemid 44026412)
64. D.P. Chakraborty, and H.J. Yoon Operating characteristics predicted by models for diagnostic tasks involving lesion localization Med Phys 35 2008 435 445 (Pubitemid 351190863)
65. D. Levine, A.A. Bankier, and E.F. Halpern Submissions to Radiology: Our top 10 list of statistical errors1 Radiology 253 2009 288 290
66. S. Paquerault, F.W. Samuelson, and K. Myers Non-localization and localization ROC analyses using clinically based scoring Proc SPIE Medical Imaging 2009 7263
67. S. Paquerault, F.W. Samuelson, and N. Petrick Comparing signal-based and case-based methodologies for CAD assessment in a detection task Proc SPIE Medical Imaging 2008 6917
68. Samuelson FW, Petrick N: Comparing image detection algorithms using resampling. Paper presented at: 3rd International Symposium on Biomedical Imaging: Nano to Micro, April 6-9, 2006, pp 1312-1315
69. 2006 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. Arlington, VA, April 6-9, 2006
70. D.P. Chakraborty Validation and statistical power comparison of methods for analyzing free-response observer performance studies Acad Radiol 15 2008 1554 1566
71. K.S. Berbaum, E.A. Franken, and D.D. Dorfman Satisfaction of search in diagnostic radiology Invest Radiol 25 1990 133 140 (Pubitemid 20072671)
72. Chakraborty DP, Yoon HJ: JAFROC analysis revisited: Figure-of-merit considerations for human observer studies. Paper presented at: Proceedings SPIE Medical Imaging: Image Perception, Observer Performance, and Technology Assessment. Lake Buena Vista, FL, March 12, 2009
73. M.S. Pepe The Statistical Evaluation of Medical Tests for Classification and Prediction 2003 Oxford University Press New York
74. A.I. Bandos, H.E. Rockette, and T. Song Area under the free-response ROC curve (FROC) and a related summary index Biometrics 65 2009 247 256