Multireader, multicase receiver operating characteristic analysis: An empirical comparison of five methods

Academic Radiology

Volumn 11, Issue 9, 2004, Pages 980-995

  Obuchowski, Nancy A ^a,b   Beiden, Sergey V ^c   Berbaum, Kevin S ^d   Hillis, Stephen L ^e   Ishwaran, Hemant ^a   Song, Hae Hiang ^f   Wagner, Robert F ^c  

^a Epidemiology and Biostatistics   (United States)

^b LERNER RESEARCH INSTITUTE   (United States)

^c CENTER FOR DRUG EVALUATION AND RESEARCH   (United States)

^d UNIVERSITY OF IOWA   (United States)

^e VETERANS AFFAIRS MEDICAL CENTER   (United States)

^f The Catholic University of Korea College of Medicine   (South Korea)

Author keywords

diagnostic accuracy; multireader multicase (MRMC) study; multireader study; Receiver operating characteristic (ROC) curve; ROC analysis

Indexed keywords

ACCURACY; ARTICLE; CONFIDENCE INTERVAL; DATA ANALYSIS; DIAGNOSTIC TEST; HUMAN; PRIORITY JOURNAL; RANK SUM TEST; RECEIVER OPERATING CHARACTERISTIC;

ANALYSIS OF VARIANCE; ANEURYSM, DISSECTING; AORTIC ANEURYSM; BREAST NEOPLASMS; FALSE POSITIVE REACTIONS; FEMALE; HUMANS; LUNG DISEASES, INTERSTITIAL; MAMMOGRAPHY; MODELS, STATISTICAL; MULTIVARIATE ANALYSIS; OBSERVER VARIATION; RADIOGRAPHY; RADIOGRAPHY, THORACIC; ROC CURVE; SIGNAL PROCESSING, COMPUTER-ASSISTED; STATISTICS, NONPARAMETRIC; X-RAY FILM;

EID: 4544318699     PISSN: 10766332     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.acra.2004.04.014     Document Type: Article

References (37)

1. Metz C.E. ROC methodology in radiologic imaging. Invest Radiol. 21:1986;720-733
2. Dorfman D.D., Alf E. Maximum likelihood estimation of parameters of signal detection theory-a direct solution. Psychometrika. 33:1968;117-124
3. Dorfman D.D., Alf E. Maximum-likelihood estimation of parameters of signal detection theory and determination of confidence intervals-rating method data. J Math Psychology. 6:1969;487-496
4. Hanley J.A., McNeil B.J. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 143:1982;29-36
5. Hanley J.A., McNeil B.J. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 148:1983;839-843
6. DeLong E.R., DeLong D.M., Clarke-Pearson D.L. Comparing the areas under two or more correlated receiver operating characteristic curves a nonparametric approach. Biometrics. 44:1988;837-844
7. McClish D.K. Analyzing a portion of the ROC curve. Med Decis Making. 9:1989;190-195
8. Zhou H.H., Obuchowski N.A., McClish D.K. Statistical methods in diagnostic medicine. 2002;Wiley & Sons, New York, NY
9. Beam C.A., Baker M.E., Paine S.S., Sostman H.D., Sullivan D.C. Answering unanswered questions proposal for a shared resource in clinical diagnostic radiology research. Radiology. 183:1992;619-620
10. Obuchowski N.A. Multi-reader ROC studies a comparison of study designs. Acad Radiol. 2:1995;709-716
11. Hanley J.A. Receiver operating characteristic (ROC) methodology the state of the art. Crit Rev Diagn Imaging. 29:1989;307-335
12. Dorfman D.D., Berbaum K.S., Metz C.E. Receiver operating characteristic rating analysis generalization to the population of readers and patients with the jackknife method. Invest Radiol. 27:1992;723-731
13. Ishwaran H., Gatsonis C.A. A general class of hierarchical ordinal regression models with applications to correlated ROC analysis. Can J Stat. 28:2000;731-750
14. Dorfman D.D., Berbaum K.S., Lenth R.V., Chen Y.F., Donaghy B.A. Monte carlo validation of a multireader method for receiver operating characteristic discrete rating data factorial experimental design. Acad Radiol. 5:1998;591-602
15. LABMRMC. Available at: http://xray.bsd.uchicago.edu/krl/KRL_ROC/ software_index.htm.
16. Schartz KM, Hillis SL, Berbaum KS, Dorfman DD: MRMC2.0. Available at: http://perception.radiology.uiowa.edu.
17. Song H.H. Analysis of correlated ROC areas in diagnostic testing. Biometrics. 53:1997;370-382
18. Obuchowski N.A., Rockette H.E. Hypothesis testing of the diagnostic accuracy for multiple diagnostic tests an ANOVA approach with dependent observations. Commun Stat Simul Computation. 24:1995;285-308
19. Obuchowski N.A. Multi-reader multi-modality ROC studies hypothesis testing and sample size estimation using an ANOVA approach with dependent observations with rejoinder. Acad Radiol. 2:1995;S22-S29
20. OBUMRM. Available at: html.http://www.bio.ri.ccf.org/OBUMRM/OBUMRM.html.
21. Beiden S.V., Wagner R.F., Campbell G. Components-of-variance models and multiple-bootstrap experiments an alternative method for random-effects, receiver operating characteristic analysis. Acad Radiol. 7:2000;341-349
22. Roe C.A., Metz C.E. Variance-component modeling in the analysis of receiver operating characteristic index estimates. Acad Radiol. 4:1997;587-600
23. Beiden S.V., Wagner R.F., Campbell G., Metz C.E., Jiang Y. Components-of-variance models for random-effects ROC analysis the case of unequal variance structure across modalities. Acad Radiol. 8:2001;605-615
24. Beiden S.V., Wagner R.F., Campbell G., Chan H.P. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol. 8:2001;616-622
25. Efron B., Tibshirani R.J. An introduction to the bootstrap. Monographs on statistics and applied probability 57. 1993;Chapman & Hall, New York, NY
26. Ishwaran H. Univariate and multirater ordinal cumulative link regression with covariate specific cutpoints. Can J Stat. 28:2000;715-730
27. Toledano A.Y., Gatsonis C. Ordinal regression methodology for ROC curves derived from correlated data. Stat Med. 15:1996;1807-1826
28. Van Dyke C.W., White R.D., Obuchowski N.A., Geisinger M.A., Lorig R.J., Meziane M.A. Cine MRI in the diagnosis of thoracic aortic dissection. 1993;79th RSNA Meetings, Chicago, IL. November 28-December 3
29. Powell K.A., Obuchowski N.A., Chilcote W.A., Barry M.M., Ganobcik S.N., Cardenosa G. Clinical evaluation of digital versus film-screen mammograms diagnostic accuracy and patient management. AJR Am J Roentgenol. 173:1990;889-894
30. Herron J.M., Bender T.M., Campbell W.L., Sumkin J.H., Rockette H.E., Gur D. Effects of luminance and resolution on observer performance with chest radiographs. Radiology. 215:2000;169-174
31. CORROC2. a FORTRAN program written by C. Metz. Replaced by ROCKIT. Available at: http://xray.bsd.uchicago.edu/krl/KRL_ROC/software_index.htm.
32. Hillis S.L. A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data. 2003;International Biometric Society ENAR Spring Meeting, Tampa, FL. March 30-April 2
33. PROC MIXED. SAS Institute Inc, Cary, NC. Available at http://www.sas.com
34. Roe C.A., Metz C.E. Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data validation wtih computer simulation. Acad Radiol. 4:1997;298-303
35. Beiden S.V., Maloof M., Wagner R.F. A general model for finite-sample effects in training and testing of competing classifiers. IEEE Trans Patt Anal Machine Intell. 25:2003;1561-1569
36. Obuchowski N.A. Sample size tables for receiver operating characteristic studies. AJR Am J Roentgenol. 175:2000;603-608
37. Hillis SL, Berbaum KS. Power estimation for the Dorfman-Berbaum- Metz method. Acad Radiol (in press).