Multi-reader ROC Studies with Split-plot Designs. A Comparison of Statistical Methods

Academic Radiology

Volumn 19, Issue 12, 2012, Pages 1508-1517

  Obuchowski, Nancy A ^a   Gallas, Brandon D ^b   Hillis, Stephen L ^c  

^a LERNER RESEARCH INSTITUTE   (United States)

^b CENTER FOR DEVICES AND RADIOLOGICAL HEALTH   (United States)

^c UNIVERSITY OF IOWA   (United States)

Author keywords

MRMC study; Multireader imaging study; ROC analysis; Split plot design

Indexed keywords

ARTICLE; HUMAN; INTERMETHOD COMPARISON; MARGINAL MEAN ANALYSIS OF VARIANCE APPROACH; METHODOLOGY; MODEL; OBUCHOWSKI ROCKETTE METHOD; PRIORITY JOURNAL; PROCEDURES; ROE METZ MODEL; SPLIT PLOT DESIGN; STATISTICAL ANALYSIS; U STATISTIC METHOD;

HUMANS; MODELS, STATISTICAL; OBSERVER VARIATION; RESEARCH DESIGN; ROC CURVE; STATISTICS AS TOPIC;

EID: 84868287850     PISSN: 10766332     EISSN: 18784046     Source Type: Journal    
DOI: 10.1016/j.acra.2012.09.012     Document Type: Article

References (29)

1. Metz C.E. Basic principles of ROC analysis. Semin Nucl Med 1978, 8:283-298.
2. Zweig M.H., Campbell G. Receiver operating characteristic plots: a fundamental evaluation tool in clinical medicine. Clin Chem 1993, 39:561-577.
3. Pepe M.S. The Statistical Evaluation of Medical Tests for Classification and Prediction 2004, Oxford University Press, New York.
4. Zhou X.H., Obuchowski N.A., McClish D.L. Statistical Methods in Diagnostic Medicine 2011, John Wiley, Hoboken, NJ. 2nd ed.
5. Wagner R.F., Metz C.E., Campbell G. Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol 2007, 14:723-748.
6. Beam C.A., Layde P.M., Sullivan D.C. Variability in the interpretation of screening mammograms by US radiologists: findings from a national sample. Arch Intern Med 1996, 156:209-213.
7. Chakraborty D.P. Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol 2006, 13:1187-1193.
8. Dorfman D.D., Berbaum K.S., Metz C.E. ROC rating analysis: generalization to the population of readers and cases with the jackknife method. Invest Radiol 1992, 27:723-731.
9. Obuchowski N.A., Rockette H.E. Hypothesis testing of the diagnostic accuracy for multiple diagnostic tests: an ANOVA approach with dependent observations. Commun Stat Simul Comput 1995, 24:285-308.
10. Beiden S.V., Wagner R.F., Campbell G. Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis. Acad Radiol 2000, 7:341-349.
11. Beiden S.V., Wagner R.F., Campbell G., et al. Components-of-variance models for random-effects ROC analysis: the case of unequal variance structures across modalities. Acad Radiol 2001, 8:605-615.
12. Beiden S.V., Wagner R.F., Campbell G., et al. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol 2001, 8:616-622.
13. Obuchowski N.A., Beiden S.V., Berbaum K.S., et al. Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol 2004, 11:980-995.
14. Hillis S.L. A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Stat Med 2007, 26:596-619.
15. Gallas B.D. One-shot estimate of MRMC variance: AUC. Acad Radiol 2006, 13:353-362.
16. Bandos A.I., Rockette H.E., Gur D. A permutation test for comparing ROC curves in multireader studies. Acad Radiol 2006, 13:414-420.
17. Obuchowski N.A. Multi-reader ROC studies: a comparison of study designs. Acad Radiol 1995, 2:709-716.
18. Obuchowski N.A. Reducing the number of reader interpretations in MRMC studies. Acad Radiol 2009, 16:209-217.
19. Gallas B.D., Pennello G.A., Myers K.J. Multireader multicase variance analysis for binary data. J Opt Soc Am A Opt Image Sci Vis 2007, 24:B70-B80.
20. Gallas B.D., Brown D.G. Reader studies for validation of CAD systems. Neural Networks 2008, 21:387-397.
21. Hillis SL. A marginal-mean ANOVA approach for analyzing multi-reader radiological imaging data. Unpublished manuscript submitted for review.
22. Bhat B.R. On the distribution of certain quadratic forms in normal variates. J R Stat Soc B 1962, 24:148-151.
23. DeLong E., DeLong D., Clarke-Pearson D. Comparing areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988, 44:837-845.
24. Gallas B.D., Bandos A., Samuelson F., et al. A framework for random-effects ROC analysis: biases with the bootstrap and other variance estimators. Commun Stat A Theory 2009, 38:2586-2603.
25. Barrett H.H., Kupinski M.A., Clarkson E. Probabilistic foundations of the MRMC method. Proc SPIE 2005, 5749:21-31.
26. Clarkson E., Kupinski M.A., Barrett H.H. A probabilistic model for the MRMC method. Part 1. Theoretical development. Acad Radiol 2006, 13:1410-1421.
27. Roe C.A., Metz C.E. Variance-component modeling in the analysis of receiver operating characteristic index estimates. Acad Radiol 1997, 4:587-600.
28. Dorfman D.D., Berbaum K.S., Lenth R.V., et al. Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design. Acad Radiol 1998, 5:9-19.
29. Hillis S.L., Obuchowski N.A., Berbaum K.S. Power estimation for multireader ROC methods an updated and unified approach. Acad Radiol 2011, 18:129-142.