Reader variability in mammography and its implications for expected utility over the population of readers and cases

Medical Decision Making

Volumn 24, Issue 6, 2004, Pages 561-572

  Wagner, Robert F ^a   Beam, Craig A ^b   Beiden, Sergey V ^a  

^a CENTER FOR DEVICES AND RADIOLOGICAL HEALTH   (United States)

^b H LEE MOFFITT CANCER CENTER AND RESEARCH INSTITUTE   (United States)

Author keywords

Expected utility analysis; Mammography; Multiple reader studies; ROC analysis; Technology assessment

Indexed keywords

BIOMEDICAL TECHNOLOGY ASSESSMENT; CANCER DIAGNOSIS; CANCER SCREENING; DECISION THEORY; DIAGNOSTIC IMAGING; INTERMETHOD COMPARISON; LABORATORY TEST; MAMMOGRAPHY; OBSERVER VARIATION; RECEIVER OPERATING CHARACTERISTIC; REVIEW; SAMPLE SIZE; SKILL; UTILIZATION REVIEW;

DATA INTERPRETATION, STATISTICAL; DIAGNOSIS; FEMALE; HUMANS; MAMMOGRAPHY; OBSERVER VARIATION; ROC CURVE; TECHNOLOGY ASSESSMENT, BIOMEDICAL;

EID: 8744247301     PISSN: 0272989X     EISSN: None     Source Type: Journal    
DOI: 10.1177/0272989X04271043     Document Type: Review

References (57)

1. Beam CA, Conant EF, Sickles EA. Association of volume and volume-independent factors with accuracy in screening mammogram interpretation. J Natl Cancer Inst. 2003;95:282-90.
2. Beam C, Layde PM, Sullivan DC. Variability in the interpretation of screening mammograms by US radiologists. Arch Intern Med. 1996;156:209-13.
3. Beam C. Reader strategies: variability and error-methodology, findings and health policy implications from a study of the US population of mammographers. In: Chakraborty DP, Krupinski EA, eds. Medical Imaging 2002: Image Perception, Observer Performance, and Technology Assessment. Proceedings of the International Society for Optical Engineering (SPIE) 2002;4686:157-68.
4. Swets JA, Pickett RM. Evaluation of Diagnostic Systems. New York: Academic Press; 1982.
5. Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol. 1992;27:723-31.
6. University of Iowa MRMC software. Available from: URL: ftp://perception.radiology.uiowa.edu/PUBLIC
7. University of Chicago LABMRMC software. Available from: URL: http://www-radiology.uchicago.edu/krl/toppage11.htm#software
8. Roe CA, Metz CE. Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation. Acad Radiol. 1997;4:298-303.
9. Dorfman DD, Berbaum KS, Lenth RV, Chen YF, Donaghy BA. Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design. Acad Radiol. 1998;5:591-602.
10. Gatsonis CA, Begg CB, Wieand S. Advances in statistical methods for diagnostic radiology: a symposium. Acad Radiol. 1995;2 Suppl 1:S1-S84.
11. Obuchowski NA. Multireader, multimodality receiver operating characteristic curve studies: hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Acad Radiol. 1995;2 Suppl 1:S22-9.
12. Toledano A, Gatsonis CA. Regression analysis of correlated receiver operating characteristic data. Acad Radiol. 1995;2 Suppl 1:530-6.
13. Toledano A, Gatsonis C. Ordinal regression methodology for ROC curves derived from correlated data. Stat Med. 1996;15:1807-26.
14. Ishwaran H, Gatsonis CA. A general class of hierarchical ordinal regression models with applications to correlated ROC analysis. Can J Stat. 2000;28:731-50.
15. Beiden SV, Wagner RF, Campbell G. Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects receiver operating characteristic analysis. Acad Radiol. 2000;7:341-9.
16. Beiden SV, Wagner RF, Campbell G, Metz CE, Jiang Y. Components-of- variance models for random-effects ROC analysis: the case of unequal variance structures across modalities. Acad Radiol. 2001;8:605-15.
17. Beiden SV, Wagner RF, Campbell G, Chan H-P. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol. 2001;8:616-22.
18. Obuchowski NA, Zepp RC. Simple steps for improving multiple-reader studies in radiology. AJR Am J Roentgenol. 1996;166:517-21.
19. Rockette HE, Campbell WL, Britton CA, Holbert JM, King JL, Gur D. Empiric assessment of parameters that affect the design of multireader receiver operating characteristic studies. Acad Radiol. 1999;6:723-9.
20. Wagner RF, Beiden SV, Campbell G, Metz CE, Sacks WM. Assessment of medical imaging and computer-assist systems: lessons from recent experience. Acad Radiol. 2002;9:1264-77.
21. Green DM, Swets JA. Signal Detection Theory and Psychophysics (Rev. Ed.). Huntington (NY): Krieger; 1974.
22. Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8:283-98.
23. Patton DD. Introduction to clinical decision making. Semin Nucl Med. 1978;8:273-82.
24. Wagner RF, Beiden SV, Metz CE. Continuous versus categorical data for ROC analysis: some quantitative considerations. Acad Radiol. 2001;8:328-34.
25. Elmore JG, Wells CK, Lee CH, Howard DH, Feinstein AR. Variability in radiologists' interpretations of mammograms. N Engl J Med. 1994;331:1493-9.
26. D'Orsi CJ, Swets JA. Letter to the editor: variability in the interpretation of mammograms. N Engl J Med. 1995;332:1172-3.
27. McClish DK. Analyzing a portion of the ROC curve. Med Decis Making. 1989;9:190-5.
28. Jiang Y, Metz CE, Nishikawa RM. A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology. 1996;201:745-50.
29. Swets JA. Indices of discrimination or diagnostic accuracy: their ROCs and implied models. Psychol Bull. 1986;99:100-17.
30. Swets JA. Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance. Psychol Bull. 1986;99:181-98.
31. Metz CE. Statistical analysis of ROC data in evaluating diagnostic performance. In: Herbert DE, Myers RH, eds. Multiple Regression Analysis: Applications in the Health Sciences. New York: American Institute of Physics, American Association of Physicists in Medicine; 1986. p 365-84.
32. Swets JA, Pickett RM, Whitehead SF, et al. Assessment of diagnostic technologies. Science. 1979;205:753-9.
33. Getty DJ, Pickett RM, D'Orsi CJ, Swets JA. Enhanced interpretation of diagnostic images. Invest Radiol. 1988;23:240-52.
34. Swets JA, Getty DJ, Pickett RM, D'Orsi CJ, Seltzer SE, McNeil BJ. Enhancing and evaluating diagnostic accuracy. Med Decis Making. 1991;11:9-18.
35. Seltzer SE, Getty DJ, Tempany CMC, et al. Staging prostate cancer with MR imaging: a combined radiologist-computer system. Radiology. 1997;202:219-26.
36. Chan H-P, Sahiner B, Helvie MA, Petrick N, et al. Improvement of radiologists' characterization of mammographic masses by using computer-aided diagnosis: an ROC study. Radiology. 1999;212:817-27.
37. Jiang Y, Nishikawa RM, Schmidt RA, Metz CE, Giger ML, Doi K. Improving breast cancer diagnosis with computer-aided diagnosis. Acad Radiol. 1999;6:22-33.
38. Roe CA, Metz CE. Variance-component modeling in the analysis of receiver operating characteristic index estimates. Acad Radiol. 1997;4:587-600.
39. Goodenough DJ, Metz CE. Implications of a "noisy" observer to data processing techniques. In: Raynaud C, Todd-Pokropek, eds. Information Processing in Scintigraphy. Orsay (France): Commissariat a l'Energie Atomique, Departement de Biologie, Service Hospitalier Frederic Joliot; 1975.
40. Cole P, Morrison AS. Basic issues in population screening for Cancer. J Natl Cancer Inst. 1980;64:1263-72.
41. Patton DD, Woolfenden JM. A utility-based model for comparing the cost-effectiveness of diagnostic studies. Invest Radiol. 1989;24:263-71.
42. Olsen O, Gøtzsche PC. Cochrane review on screening for breast cancer with mammography. Lancet. 2001;358:1340-2.
43. Horton R. Screening mammography: an overview revisited. Lancet. 2001;358:1284-5.
44. Kolata G. Study sets off debate over mammograms' value. The New York Times 2001 Dec 9: Sect. A:1, 32.
45. Burnside E, Belkora J, Esserman L. The impact of alternative practices on the cost and quality of mammographic screening in the United States. Clin Breast Cancer. 2001;2:145-52.
46. Department of Health and Human Services, US Food and Drug Administration. Quality Mammography Standards: Final Rule. Federal Register, Vol. 62, No. 208. Part II (21 CFR Parts 16 and 900). 28 October 1997. p 55963.
47. US Food and Drug Administration. Food Labeling: Trans-fatty Acids in Nutrition Labeling, Nutrient Content Claims, and Health Claims; Proposed Rule. Federal Register, 17 November 1999.
48. Lusted LB. Decision-making studies in patient management. N Engl J Med. 1971;284:416-24.
49. Baker SG. Identifying combinations of cancer biomarkers for further study as triggers of early intervention. Biometrics. 2000;56:1082-7.
50. Christiansen CL, Wang F, Barton MB, et al. Predicting the cumulative risk of false positive mammograms. J Natl Cancer Inst. 2000;92:1657-66.
51. Baker SG, Pinsky P. A proposed design and analysis for comparing digital and analog mammography: special ROC methods for cancer screening. J Am Stat Assoc. 2001;96:421-8.
52. Wagner RF, Beam CA, Beiden SV. A study of the propagation of reader variability in mammography to the variable expected benefit over the population. In: Chakraborty DP, Krupinski EA, eds. Medical Imaging 2003: Image Perception, Observer Performance, and Technology Assessment. Proceedings of the International Society for Optical Engineering (SPIE) 2003;5034:238-40.
53. American College of Radiology. Breast Imaging Recording and Data System (BIRADS). Reston (VA): American College of Radiology; 1993.
54. Orel SG, Sullivan DC, Dambro TJ. BIRADS categorization as a predictor of malignancy [abstract]. Radiology. 1997;205(P):447.
55. Houn F, Bright RA, Bushar HF, et al. Study design in the evaluation of breast cancer imaging technologies. Acad Radiol. 2000;7:684-92.
56. Kopans DG. Recent issues in breast cancer detection and the premarket approval by the Food and Drug Administration of a US system for breast lesion evaluation: what happened to science? Radiology. 1997;202:315-6.
57. Schultz DG, Wagner RF, Campbell G. Science is alive and well at the Food and Drug Administration. Radiology. 1997;202:317-8.