Assessment of Medical Imaging Systems and Computer Aids: A Tutorial Review

Academic Radiology

Volumn 14, Issue 6, 2007, Pages 723-748

  Wagner, Robert F ^a   Metz, Charles E ^b   Campbell, Gregory ^a  

^a CENTER FOR DEVICES AND RADIOLOGICAL HEALTH   (United States)

^b UNIVERSITY OF CHICAGO   (United States)

Author keywords

Assessment methodologies; Multiple reader multiple case (MRMC) ROC analysis; ROC analysis

Indexed keywords

ARTICLE; BREAST CANCER; COMPUTER ASSISTED RADIOGRAPHY; COMPUTER PROGRAM; DIAGNOSTIC IMAGING; DIGITAL MAMMOGRAPHY; HUMAN; IMAGING SYSTEM; LABORATORY DIAGNOSIS; LABORATORY TEST; METHODOLOGY; NONHUMAN; PILOT STUDY; PRIORITY JOURNAL; RADIOLOGIST; RECEIVER OPERATING CHARACTERISTIC;

DIAGNOSIS, COMPUTER-ASSISTED; DIAGNOSTIC IMAGING; EQUIPMENT DESIGN; HUMANS; MODELS, STATISTICAL; OBSERVER VARIATION; ROC CURVE; SENSITIVITY AND SPECIFICITY; SOFTWARE; TECHNOLOGY ASSESSMENT, BIOMEDICAL;

EID: 34250162528     PISSN: 10766332     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.acra.2007.03.001     Document Type: Article

References (136)

1. Wagner R.F., Beiden S.V., Campbell G., et al. Assessment of medical imaging and computer-assist systems: lessons from recent experience. Acad Radiol 9 (2002) 1264-1277
2. Wagner R.F., Beiden S.V., Campbell G., et al. Contemporary issues for experimental design in assessment of medical imaging and computer-assist systems. Proc SPIE 5034 (2003) 213-224
3. Beiden S.V., Wagner R.F., Doi K., et al. Independent versus sequential reading in ROC studies of computer-assist modalities: analysis of components of variance. Acad Radiol 9 (2002) 1036-1043
4. Dodd L.E., Wagner R.F., Armato S.G., et al. Assessment methodologies and statistical issues for computer-aided diagnosis of lung nodules in computed tomography: contemporary research topics relevant to the lung image database consortium (LIDC). Acad Radiol 11 (2004) 462-475
5. Wagner R.F. An overview of contemporary ROC methodology in medical imaging and computer-assist modalities (2004). http://www.fda.gov/ohrms/dockets/ac/04/slides/4024s1.htm Tutorial presented at the FDA Radiological Devices Public Panel Meeting of the Medical Devices Advisory Board, February 3. Available online at: http://www.fda.gov/ohrms/dockets/ac/04/slides/4024s1.htm
6. Houn F., Bright R.A., Bushar H.F., et al. Study design in the evaluation of breast cancer imaging technologies. Acad Radiol 7 (2000) 684-692
7. Metz C.E. Basic principles of ROC analysis. Semin Nucl Med 8 (1978) 283-298
8. Zweig M.H., and Campbell G. Receiver operating characteristic plots: a fundamental evaluation tool in clinical medicine. Clin Chem 39 (1993) 561-577
9. Gur D., Rockette H.E., Armfield D.R., et al. Prevalence effect in a laboratory environment. Radiology 228 (2003) 10-14
10. Patton D.D. Introduction to clinical decision making. Semin Nucl Med (1978) 273-282 VIII
11. Green D.M., and Swets J.A. Signal detection theory and psychophysics (revised ed) (1974), Krieger, Huntington, NY
12. Wagner R.F., Beam C.A., and Beiden S.V. Reader variability in mammography and its implications for expected utility over the population of readers and cases. Med Decis Making 24 (2004) 561-572
13. Metz C.E., and Pesce L. Private communication of draft manuscript on general principles motivating the use of "proper" binormal ROC curves (2006) June
14. Beam C., Layde P.M., and Sullivan D.C. Variability in the interpretation of screening mammograms by US radiologists. Arch Intern Med 156 (1996) 209-213
15. American College of Radiology. Breast imaging recording and data system (BI-RADS®) (1993), American College of Radiology, Reston, Va First edition 1993; currently in fourth edition 2003
16. Biggerstaff B.J. Comparing diagnostic tests: a simple graphic using likelihood ratios. Stat Med 19 (2000) 649-663
17. Fryback D.G., and Thornbury J.R. The efficacy of diagnostic imaging. Med Decis Making 11 (1991) 88-94
18. NCRP Commentary No. 13. An introduction to efficacy in diagnostic radiology and nuclear medicine (1995), National Council on Radiation Protection and Measurements, Bethesda, Md
19. Federal Food, Drug, and Cosmetic Act. Chapter V-drugs and devices; subchapter A-drugs and devices. Section 513 [US Code 360c. Washington, DC: US GPO. Available online at http://www.fda.gov/opacom/laws/fdcact/fdcact5a2.htm
20. Swets J.A., Pickett R.M., Whitehead S.F., et al. Assessment of diagnostic technologies. Science 205 (1979) 753-759
21. Swets J.A., and Pickett R.M. Evaluation of Diagnostic Systems (1982), Academic Press, New York, New York
22. Rockette H.E., Gur D., and Metz C.E. The use of continuous and discrete confidence judgements in receiver operating characteristic studies of diagnostic imaging techniques. Invest Radiol 27 (1992) 169-172
23. Wagner R.F., Beiden S.V., and Metz C.E. Continuous versus categorical data for ROC analysis: some quantitative considerations. Acad Radiol 8 (2001) 328-334
24. Jiang Y., Nishikawa R.M., Schmidt R.A., et al. Improving breast cancer diagnosis with computer-aided diagnosis. Acad Radiol 6 (1999) 22-33
25. Wagner R.F. Toward a strategy for consensus development on a quantitative approach to medical imaging. Acad Radiol 13 (2006) 137-139
26. Pisano E.D., Gatsonis C., Hendrick E., et al., Digital Mammographic Imaging Screening Trial (DMIST) Investigators Group. Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med 353 (2005) 1773-1783
27. Cole P., and Morrison A.S. Basic issues in population screening for cancer. J Natl Cancer Inst 64 (1980) 1263-1272
28. Baker S.G., and Pinsky P. A proposed design and analysis for comparing digital and analog mammography: special ROC methods for cancer screening. J Am Stat Assoc 96 (2001) 421-428
29. Hanley J.A. Receiver operating characteristic (ROC) methodology: the state of the art. Crit Rev Diagn Imaging 23 (1989) 307-335
30. Dorfman D.D., Berbaum K.S., and Metz C.E. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol 27 (1992) 723-731
31. Gatsonis C.A., Begg C.B., and Wieand S. Advances in statistical methods for diagnostic radiology: a symposium. Acad Radiol 2 Suppl 1 (1995) S1-S84
32. University of Iowa. ROC software. Available online at: http://perception.radiology.uiowa.edu/
33. The University of Chicago. ROC software. Available online at: http://www-radiology.uchicago.edu/krl/KRL_ROC/software_index6.htm
34. Roe C.A., and Metz C.E. Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation. Acad Radiol 4 (1997) 298-303
35. Dorfman D.D., Berbaum K.S., Lenth R.V., et al. Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design. Acad Radiol 5 (1998) 591-602
36. Obuchowski N.A. Multireader, multimodality receiver operating characteristic curve studies: hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Acad Radiol 2 Suppl 1 (1995) S22-S29
37. Obuchowski N.A., and Rockette H.E. Hypothesis testing of the diagnostic accuracy for multiple diagnostic tests: an ANOVA approach with dependent observations. Commun Stat Simul Comput 24 (1995) 285-308
38. Cleveland Clinic. Quantitative health sciences. Available online at: http://www.bio.ri.ccf.org/html/rocanalysis.html
39. Toledano A., and Gatsonis C.A. Regression analysis of correlated receiver operating characteristic data. Acad Radiol 2 Suppl 1 (1995) S30-S36
40. Toledano A., and Gatsonis C. Ordinal regression methodology for ROC curves derived from correlated data. Stat Medi 15 (1996) 1807-1826
41. Song H.H. Analysis of correlated ROC areas in diagnostic testing. Biometrics 53 (1997) 370-382
42. Ishwaran H., and Gatsonis C.A. A general class of hierarchical ordinal regression models with applications to correlated ROC analysis. Can J Stat 28 (2000) 731-750
43. Beiden S.V., Wagner R.F., and Campbell G. Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects receiver operating characteristic analysis. Acad Radiol 7 (2000) 341-349
44. Beiden S.V., Wagner R.F., Campbell G., et al. Components-of-variance models for random-effects ROC analysis: The case of unequal variance structures across modalities. Acad Radiol 8 (2001) 605-615
45. Beiden S.V., Wagner R.F., Campbell G., et al. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol 8 (2001) 616-622
46. Obuchowski N.A., Beiden S.V., Berbaum K.S., et al. Multi-reader, multi-case ROC analysis: an empirical comparison of five methods. Acad Radiol 11 (2004) 980-995
47. Toledano A. Three methods for analyzing correlated ROC curves: A comparison in real data sets from multi-reader, multi-case studies with a factorial design. Stat Med 22 (2003) 2919-2933
48. Song X., and Zhou X.-H. A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data. Biostatistics 6 (2005) 303-312
49. Hillis S.L., Obuchowski N.A., Schartz K.M., et al. A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data. Stat Med 24 (2005) 1579-1607
50. Hillis S.L., and Berbaum K.S. Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification. Acad Radiol 12 (2005) 1534-1541
51. Hillis S.L. A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Stat Med 26 (2007) 596-619
52. Hillis S.L., and Berbaum K.S. Recent developments in the Dorfman-Berbaum-Metz (DBM) procedure for multi-reader ROC study analysis (2005) Presented at Medical Image Perception Society (MIPS) Conference XI, September, Windermere, UK.
53. Barrett H.H., Kupinski M.A., and Clarkson E. Probabilistic foundations of the MRMC method. In: Eckstein M.P., and Jiang Y. (Eds). Medical imaging: image perception, observer performance, and technology assessment (2005) 21-31 Proc. of the SPIE 5749. Bellingham, WA
54. Clarkson E., Kupinski M.A., and Barrett H.H. A probabilistic model for the MRMC method. Part I. Theoretical development. Acad Radiol 13 (2006) 1410-1421
55. Kupinski M.A., Clarkson E., and Barrett H.H. A probabilistic model for the MRMC method. Part II. Validation and applications. Acad Radiol 13 (2006) 1422-1430
56. Gallas B.D. One-shot estimate of MRMC variance: AUC. Acad Radiol 13 (2006) 353-362
57. Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristic curve. J Math Psych 12 (1975) 387-415
58. Campbell G., Douglas M.A., and Bailey J.J. Nonparametric comparison of two tests of cardiac function on the same patients using the entire ROC curve. Proc Computers Cardiol (1988) 267-270 Published by the IEEE Computer Society Press. Washington, DC, September 25-28, 1988:.
59. Bandos A.I., Rockette H.E., and Gur D. Exact bootstrap variances of the area under the ROC curve. Commun Sta A-Theory 36 13 (2007) in press
60. Gallas BD, Bandos A, Kupinski M, et al. A framework for random-effects ROC analysis: biases with the bootstrap and other variance estimators. Stat Med. Submitted to Stat Med.
61. Samuelson F.W., and Wagner R.F. Bootstrapped MRMC confidence intervals. In: Eckstein M.P., and Jiang Y. (Eds). Proc SPIE 5749, Med Imaging 2005: image perception, observer performance, and technology assessment (2005) 15-20 Bellingham, WA
62. Obuchowski N.A. Multireader receiver operating characteristic studies: a comparison of study designs. Acad Radiol 2 (1995) 709-716
63. ICRU (submitted 2006; in press 2007). ROC analysis in medical imaging. Report Committee # TBD, International Commission for Radiological Units and Measurements (ICRU) Bethesda, MD.
64. DeLong E.R., DeLong D.M., and Clarke-Pearson D.L. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44 (1988) 837-845
65. Hanley J.A., and McNeil B.J. The meaning and use of the area under the receiver operating characteristic (ROC) curve. Radiology 143 (1982) 29-36
66. McClish D.K. Analyzing a portion of the ROC curve. Med Decis Making 9 (1989) 190-195
67. Wieand S., Gail M.H., James B.R., et al. A family of nonparametric statistics for comparing diagnostic markers with paired and unpaired data. Biometrika 76 (1989) 585-592
68. Jiang Y., Metz C.E., and Nishikawa R.M. A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology 201 (1996) 745-750
69. Roe C.A., and Metz C.E. Variance-component modeling in the analysis of receiver operating characteristic index estimates. Acad Radiol 4 (1997) 587-600
70. Hillis S.L., and Berbaum K.S. Power estimation for the Dorfman-Berbaum-Metz method. Acad Radiol 11 (2004) 1260-1273
71. Private communication, originally noted by CE Metz and supported by unpublished calculations of Chan HP, Sahiner B, Wagner RF, et al, 1998.
72. Greenhouse S.W., and Mantel N. The evaluation of diagnostic tests. Biometrics 6 (1950) 399-412
73. Metz C.E. Fundamental ROC analysis. In: Beutel J., Kundel H.L., and VanMetter R.L. (Eds). Handbook of medical imagin. Vol 1. Physics and psychophysics (2000), SPIE Press, Bellingham, WA 751-769
74. Kondratovich M. Personal communication of unpublished report to RF Wagner (1999)
75. Blume JD. Bounding sample size projections for the area under a ROC curve. Draft report available online at http://alexander.stat.brown.edu/∼jblume/slides/.
76. Metz C.E., and Pesce L. Personal communication (2006)
77. Metz C.E., and Shen J.H. Gains in accuracy from replicated readings of diagnostic images: prediction and assessment in terms of ROC analysis. Med Decis Making 12 (1992) 60-75
78. Metz C.E. Statistical analysis of ROC data in evaluating diagnostic performance. In: Herbert D.E., and Myers R.H. (Eds). Multiple regression analysis: applications in the health sciences (1986), American Institute of Physics: American Association of Physicists in Medicine, New York 365-384
79. Zhou X.-H., Obuchowski N.A., and McClish D.K. Statistical methods in diagnostic medicine (Wiley series in probability and statistics) (2002), Wiley-Interscience, New York
80. Rockette H.E., Campbell W.L., Britton C.A., et al. Empiric assessment of parameters that affect the design of multireader receiver operating characteristic studies. Acad Radiol 6 (1999) 723-729
81. Wagner R.F., Beiden S.V., and Campbell G. Multiple-reader studies, digital mammography, computer-aided diagnosis-and the Holy Grail of imaging physics (I). Proc SPIE 4320 (2001) 611-618
82. Metz C.E. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol 24 (1989) 234-245
83. Kobayashi T., Xu X.-W., MacMahon H., et al. Effect of a computer-aided diagnosis scheme on radiologists' performance in detection of lung nodules on radiographs. Radiology 199 (1996) 843-848
84. Nishikawa R.M., Giger M.L., Schmidt R.A., et al. Can computer-aided diagnosis (CAD) help radiologists find mammographically missed screening cancers. Proc SPIE Med Imaging 4324 (2001) 56-63
85. Beiden S.V., Wagner R.F., Doi K., et al. Independent versus sequential reading in ROC studies of computer-assist modalities: analysis of components of variance. Acad Radiol 9 (2002) 1036-1043
86. Freedman M., Lure F., et al. Lo S-C B. Design choices in the Deus Technologies RS-2000 lung cancer on chest radiograph ROC clinical trial (2001) Presented at Medical Image Perception Conference IX, Warrenton, Va, September 20-23
87. Gifford H.C., and King M.A. Case-sampling in LROC: a Monte Carlo analysis. Proc SPIE Med Imaging 4324 (2000) 143-152
88. Dev Chakraborty. What is FROC? Available online at: http://www.devchakraborty.com.
89. Egan J.P., Greenburg G.Z., and Schulman A.I. Operating characteristics, signal detectability and the method of free response. J Acoust Soc Am 33 (1961) 993-1007
90. Starr S.J., Metz C.E., Lusted L.B., et al. Visual detection and localization of radiographic images. Radiology 116 (1975) 533-538
91. Bunch P.C., Hamilton J.F., Sanderson G.K., et al. A free-response approach to the measurement and characterization of radiographic-observer performance. J Appl Photogr Eng 4 (1978) 166-171
92. Chakraborty D.P. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med. Phys 16 (1989) 561-568
93. Chakraborty D.P., and Winter L. Free-response methodology: alternate analysis and a new observer-performance experiment. Radiology 174 (1990) 873-881
94. Swensson R.G. Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys 23 (1996) 1709-1725
95. Metz C.E. Evaluation of digital mammography by ROC analysis. In: Doi K. (Ed). Digital mammography '96 (1996), Elsevier Science, Amsterdam
96. Chakraborty D.P. The FROC, AFROC and DROC variants of the ROC analysis. Vol 1. Physics and psychophysics. In: Beutel J., Kundel H.L., and Van Metter R.L. (Eds). Handbook of medical imaging (2000), SPIE Press, Bellingham, WA 771-796
97. Chakraborty D. Statistical power in observer performance studies: s comparison of the ROC and free-response methods in tasks involving localization. Acad Radiol 9 (2002) 147-156
98. Edwards D.C., Kupinski M.A., Metz C.E., et al. Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Med Phys 29 (2002) 2861-2870
99. Chakraborty D.P., and Berbaum K.S. Observer studies involving detection and localization: Modeling, analysis, and validation. Med Phys 31 (2004) 2313-2330
100. Zheng B., Chakraborty D.P., Rockette H.E., et al. A comparison of two data analyses from two observer performance studies using Jackknife ROC and JAFROC. Med Phys 32 (2005) 1031-1034
101. Chakraborty D.P. A search model and figure of merit for observer data acquired according to the free-response paradigm. Phys Med Biol 51 (2006) 3449-3462
102. Chakraborty D.P. ROC curves predicted by a model of visual search. Phys Med Biol 51 (2006) 3463-3482
103. Nishikawa R.M., and Yarusso L.M. Variations in measured performance of CAD schemes due to database composition and scoring protocol. Proc SPIE 3338 (1998) 840-844
104. Giger M.L. Current issues in CAD for mammography. In: Doi K., Giger M.L., Nishikawa R.M., et al. (Eds). Digital mammography '96 (1996), Elsevier Science B.V, Amsterdam 53-59
105. Kallergi M., Carney G.M., and Gaviria J. Evaluating the performance of detection algorithms in digital mammography. Med Phys 26 (1999) 267-275
106. Kundel H.L., and Nodine C.F. Modeling visual search during mammogram viewing. Proc SPIE Med Imaging 5372 (2004) 110-115
107. Dorfman D.D., and Berbaum K.S. A contaminated binormal model for ROC data. Part II: a formal model. Acad Radiol 7 (2000) 427-437
108. Samuelson F.W., and Petrick N. Comparing image detection algorithms using resampling (2006) 1312-1315 Proceedings of the 2006 IEEE International Symposium on Biomedical Imaging
109. Bornefalk H., and Bornefalk Hermansson A. On the comparison of FROC curves in mammography CAD systems. Med Phys 32 (2005) 412-417
110. Obuchowski N.A., Lieber M.L., and Powell K.A. Data analysis for detection and localization of multiple abnormalities with application to mammography. Acad Radiol 7 (2000) 516-525
111. Rutter C.M. Bootstrap estimation of diagnostic accuracy with patient-clustered data. Acad Radiol 7 (2000) 413-419
112. Chakraborty D., Obuchowski N.A., Lieber M.L., et al. Point-counterpoint. Acad Radiol 7 (2000) 553-556
113. Chakraborty D.P. Personal communication to RF Wagner (2004)
114. Armato S.G., McLennan G., McNitt-Gray M.F., et al. Lung Image Database Consortium: developing a resource for the medical imaging research community. Radiology 232 (2004) 739-748
115. Revesz G., Kundel H.L., and Bonitatibus M. The effect of verification on the assessment of imaging techniques. Invest Radiol 18 (1983) 194-198
116. Miller D.P., O'Shaughnessy K.F., Wood S.A., et al. Gold standards and expert panels: A pulmonary nodule case study with challenges and solutions. Proc SPIE Med Imaging 5372 (2004) 173-184
117. Henkelman R.M., Kay I., and Bronskill M.J. Receiver operator characteristic (ROC) analysis without truth. Med Decis Making 10 (1990) 24-29
118. Begg C.B., and Metz C.E. Consensus diagnoses and "gold standards.". Med Decis Making 10 (1990) 29-30
119. Beiden S.V., Campbell G., Meier K.L., et al. On the problem of ROC analysis without truth: the EM algorithm and the information matrix. Proc SPIE 3981 (2000) 126-134
120. Zhou X.-H. Correcting for verification bias in studies of a diagnostic test's accuracy. Stat Methods Med Res 7 (1998) 337-353
121. Zhou X.-H. Comparing correlated areas under the ROC curves of two diagnostic tests in the presence of verification bias. Biometrics 54 (1998) 453-470
122. Premarket applications for digital mammography systems; final guidance for industry and FDA. US Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health (February 4, 1999). Available online at www.fda.gov/cdrh/ode/guidance/983.html
123. Gur D., Rockette H.E., Warfel T.E., et al. From the laboratory to the clinic: the 'prevalence effect'. Acad Radiol 10 (2003) 1324-1326
124. Kundel H.L., and Polansky M. Comparing observer performance with mixture distribution analysis when there is no external gold standard. Proc SPIE 3340 (1998) 78-84
125. Begg C.B., and McNeil B.J. Assessment of radiologic tests: control of bias and other design considerations. Radiology 167 (1988) 565-569
126. American College of Radiology Imaging Network. Available online at: http://www.acrin.org/committees.html
127. Metz C.E., and Pesce L. Personal communication of expected availability of ROC software upgrades (2006) February
128. Metz C.E., and Pan X. "Proper" binormal ROC curves: theory and maximum-likelihood estimation. J Math Psychol 43 (1999) 1-33
129. Beiden S.V., Maloof M.A., and Wagner R.F. A general model for finite-sample effects in training and testing of competing classifiers. IEEE Trans Pattern Anal Machine Intell (TPAMI) 25 (2003) 1561-1569
130. Yousef W.A., Wagner R.F., and Loew M.H. Comparison of nonparametric methods for assessing classifier performance in terms of ROC parameters. Applied Imagery Pattern Recognition Workshop, 2004 (2004) 190-195 Published by the IEEE Computer Society. Proceedings 33rd AIPR conference
131. Yousef W.A., Wagner R.F., and Loew M.H. Estimating the uncertainty in the estimated mean area under the ROC curve of a classifier. Pattern Recogn Lett 26 (2005) 2600-2610
132. Yousef W.A., Wagner R.F., and Loew M.H. Assessing classifiers from two independent data sets using ROC analysis: a nonparametric approach. IEEE Trans Patt Anal Mach Intell (TPAMI) 28 (2006) 1809-1817
133. Yousef WA, Wagner RF, Loew MH. The partial area under the ROC curve: its properties and nonparametric estimation for assessing classifier performance. Submitted to IEEE Patt Anal Mach Intell.
134. Gur D., Wagner R.F., and Chan H.-P. On the repeated use of databases for testing incremental improvement of computer-aided detection schemes. Acad Radiol 11 (2004) 103-105
135. Pepe M.S. The statistical evaluation of medical tests for classification and prediction (2003), Oxford University Press, New York
136. Interagency Council on Biomedical Imaging in Oncology. Available online at: http://cancer.gov/dctd/icbio