Receiver operating characteristic rating analysis: Generalization to the population of readers and patients with the jackknife method

Investigative Radiology

Volumn 27, Issue 9, 1992, Pages 723-731

  Dorfman, Donald D ^a   Berbaum, Kevin S ^a   Metz, Charles E ^b  

^a UNIVERSITY OF IOWA   (United States)

^b UNIVERSITY OF CHICAGO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANALYTIC METHOD; ARTICLE; AVERAGING; CALCULATION; DATA ANALYSIS; INTERMETHOD COMPARISON; NEWBORN; PRIORITY JOURNAL; RADIOLOGIST; RADIOLOGY; STATISTICAL ANALYSIS; VARIANCE; X RAY FILM;

ANALYSIS OF VARIANCE; COMPARATIVE STUDY; CONFIDENCE INTERVALS; HUMAN; LIKELIHOOD FUNCTIONS; METHODS; MODELS, STATISTICAL; RADIOGRAPHY; ROC CURVE; SUPPORT, U.S. GOV'T, NON-P.H.S.; SUPPORT, U.S. GOV'T, P.H.S.;

EID: 0026664983     PISSN: 00209996     EISSN: 15360210     Source Type: Journal    
DOI: 10.1097/00004424-199209000-00015     Document Type: Article

References (40)

1. Hanley JA. Receiver operating characteristic (ROC) methodology: the state of the art. Crit Rev Diagn Imaging 1989;29:307-335.
2. Brownlee KA. Statistical theory and methodology in science and engineering, 2nd ed. New York, NY: John Wiley and Sons; 1965:504-529.
3. Winer BJ. Statistical principles in experimental design, 2nd ed. New York, NY: McGraw-Hill; 1971:375-380.
4. Quenouille MH. Approximate tests of correlation in time series. J Roy Stat Soc Ser B 1949;11:68-84.
5. Quenouille MH. Notes on bias in estimation. Biometrika 1956;43: 353-360.
6. Tukey JW. Bias and confidence in not quite large samples. Ann Math Statist 1958;29:614. Abstract.
7. Swets JA, Pickett RM. Evaluation of diagnostic systems: Methods from signal detection theory. New York; NY: Academic Press; 1982:68-93.
8. Hanley JA, Mc Neil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983;148:839-843.
9. McNeil BJ, Hanley JA. Statistical approaches to the analysis of receiver operating characteristic (ROC) curves. Med Decis Making 1984;4:137-150.
10. Metz CE, Wang PL, Kronman HB. A new approach for testing the significance of differences between ROC curves measured from correlated data. In: Deconinck F, ed. Information processing in medical imaging. The Hague, The Netherlands: Martinus Nijhoff; 1984:432-445.
11. Metz CE. ROC methodology in radiologic imaging. Invest Radiol 1986;21:720-733.
12. Swets JA. Measuring the accuracy of diagnostic systems. Science 1988;240;1285-1293.
13. Metz CE. Some practical issues of experimental design and data analysis in radiographic ROC studies. Invest Radiol 1989;24:234-245.
14. Berbaum KS, Dorfman DD, Franken EA Jr. Measuring observer performance by ROC analysis: indications and complications. Invest Radiol 1989;24:228-233.
15. Kendall MG, Stuart A. The advanced theory of statistics. Vol. 3. New York, NY: Hafner; 1966:40-43.
16. Hinckley D. Jackknife Methods. In: Kotz S, Johnson NL, eds. Encyclopedia of statistical sciences. Vol 4. New York, NY: John Wiley and Sons; 1983:280-287.
17. Miller RG. The jackknife-a review. Biometrika 1974;61:1-15.
18. Duncan GT, Layard WJ. A Monte Carlo study of asymptotically robust tests for correlation coefficients. Biometrika 1973;60:551-558.
19. Faraone SV, Dorfman DD. Lag sequential analysis: robust statistical Methods. Psychol Bull 1987;101:312-323.
20. Thompson ML, Zucchini W. On the statistical analysis of ROC curves. Stat Med 1989;8:1277-1290.
21. Arvesen JN, Salsburg DS. Approximate tests and confidence intervals using the jackknife. In: RM Elashoff, ed. Perspectives in biometrics. New York, NY: Academic Press; 1975:123-147.
22. Rey WJJ. Introduction to robust and quasi-robust statistical Methods. New York, NY: Springer-Verlag; 1983;55.
23. Brillinger DR. The asymptotic behavior of Tukey’s general method of setting approximate confidence limits (the jackknife) when applied to maximum likelihood estimates. Rev Inst Int Stat 1964;32:202-206.
24. Reeds JA. Jackknifing maximum likelihood estimates. Ann Stat 1978; 6:727-739.
25. Efron B. The jackknife, the bootstrap and other resampling plans. Philadephia, PA: Society for Industrial and Applied Mathematics; 1982:5-6.
26. Dorfman DD, Berbaum KS. RSCORE-J: pooled rating method data: a computer program for analyzing pooled ROC curves. Behav Res Methods Instrum Comput 1986;18:452-462.
27. Berbaum KS, Franken EA, Dorfman DD, Barloon TJ. Influence of clinical history upon detection of nodules and other lesions. Invest Radiol 1988;23:48-55.
28. McClish DK. Comparing the areas under more than two independent ROC curves. Med Decis Making 1987;7:149-155.
29. Dorfman DD, Alf E Jr. Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals-rating method data. J Math Psychol 1969;6:487-496.
30. Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychol 1975;12:387-415.
31. Cochran WG, Cox GM. Experimental designs, 2nd ed. New York, NY: John Wiley and Sons; 1957:565-567.
32. Anderson RL, Bancroft TA. Statistical theory in research. New York, NY: McGraw-Hill; 1952:338-357.
33. Myers JL. Fundamentals of experimental design. Boston, MA: Allyn & Bacon; 1966:281-282.
34. SAS User’s Guide. Statistics (version 5 edition). Cary, NC: SAS Institute; 1985:113-137, 433-506.
35. Davenport JM, Webster JT. A comparison of some approximate F-tests. Technometrics 1973;15:779-789.
36. Satterthwaite FE. Synthesis of variance. Psychometrika 1941;6:309-316.
37. Satterthwaite FE. An approximate distribution of estimates of variance components. Biom Bull 1946;2:110-114.
38. Franken EA Jr, Berbaum KS, Marley SM, et al. Evaluation of a digital workstation for interpreting neonatal examinations: a receiver operating characteristic study. Invest Radiol 1992;27:732-737.
39. Tukey JW. Analyzing data: sanctification or detective work? Am Psychol 1969;24:83-91.
40. Pollard JH. A handbook of numerical and statistical techniques. New York, NY: Cambridge University Press; 1977:93-94.