Statistical evaluation of diagnostic performance: Topics in ROC analysis

Statistical Evaluation of Diagnostic Performance: Topics in ROC Analysis

Volumn , Issue , 2016, Pages 1-221

  Zou, Kelly H ^a,b,c,d   Liu, Aiyi ^e   Bandos, Andriy I ^f   Ohno Machado, Lucila ^g   Rockette, Howard E ^h,i  

^a PFIZER INC   (United States)

^b HARVARD MEDICAL SCHOOL   (United States)

^c BRIGHAM AND WOMEN S HOSPITAL   (United States)

^d BOSTON CHILDREN S HOSPITAL   (United States)

^e EUNICE KENNEDY SHRIVER NATIONAL INSTITUTE OF CHILD HEALTH AND HUMAN DEVELOPMENT   (United States)

^f UNIVERSITY OF PITTSBURGH   (United States)

^g UNIVERSITY OF CALIFORNIA   (United States)

^h UNIVERSITY OF PITTSBURGH GRADUATE SCHOOL OF PUBLIC HEALTH   (United States)

ⁱ AMERICAN STATISTICAL ASSOCIATION   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85051690686     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Book

References (434)

1. Ash, A. and Schartz, M. 1999. R2: a useful measure of model performance when predicting a dichotomous outcome. Statistics in Medicine 18: 375-384.
2. Bandos, A.I., Rockette, H.E., Song, T. et al. 2009. Area under the free-response ROC curve (FROC) and a related summary index. Biometrics 65: 247-256.
3. Beam, C.A. and Wieand, H.S. 1991. A statistical method for the comparison of a discrete diagnostic test with several continuous diagnostic tests. Biometrics 47: 907-919.
4. Biggerstaff, B.J. 2000. Comparing diagnostic tests: a simple graphic using likelihood ratios. Statistics in Medicine 19: 649-663.
5. Campbell, G. 1994. Advances in statistical methodology for the evaluation of diagnostic and laboratory tests. Statistics in Medicine 13: 499-508.
6. Chen, H. and Su, J.Q. 1995. Simultaneous confidence bands for comparing diagnostic markers. Communications in Statistics: Theory and Methods 24: 375-393.
7. DeLong, E.R., DeLong, D.M., and Clarke-Pearson, D.L. 1988. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44: 837-845.
8. Dorfman, D.D., Berbaum, K.S., and Metz, C.E. 1992. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Investigative Radiology 27: 723-731.
9. Egan, J.P. 1975. Signal Detection Theory and ROC Analysis. New York: Academic Press.
10. Fleiss, J.L. 1986. The Design and Analysis of Clinical Experiments. New York: Wiley.
11. Fluss, R., Faraggi, D., and Reiser, B. 2005. Estimation of the Youden index and its associated cutoff point. Biometrical Journal 47: 458-472.
12. Gastwirth, J.L. 1972. The estimation of the Lorenz curve and Gini index. Review of Economics and Statistics 54: 306-316.
13. Gönen, M. 2007. Analyzing Receiver Operating Characteristic Curves with SAS®. Cary, NC: SAS Institute Inc.
14. Green, D.M. and Swets, J.A. 1974. Signal Detection Theory and Psychophysics. Huntington, NY: Robert E. Krieger Publishing.
15. Greenhouse, S.W. and Mantel, N. 1950. Evaluation of Diagnostic Tests. Biometrics 6: 399-412.
16. Hanley, J.A. and McNeil, B.J. 1983. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148: 839-843.
17. Hilgers, R.A. 1991. Distribution-free confidence bounds for ROC curves. Methods of Information in Medicine 30: 96-101.
18. Hillis, S.L. and Berbaum, K.S. 2005. Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification. Academic Radiolology 12: 1534-1541.
19. Hillis, S.L., Obuchowski, N.A., Schartz, K.M. et al. 2005. A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data. Statistics in Medicine 24: 1579-1607.
20. Hillis, S.L. 2007. A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Statistics in Medicine 26: 596-619.
21. Jensen, K., Müller, H.H., and Schäfer, H. 2000. Regional confidence bands for ROC curves. Statistics in Medicine 19: 493-509.
22. Kendall, M. and Gibbons, J.D. 1990. Rank Correlation Methods, 5th ed. New York: Oxford University Press.
23. Kestler, H.A. 2001. ROC with confidence: a Perl program for receiver operator characteristic curves. Computer Methods and Programs in Biomedicine 64: 133-136.
24. Krzanowski, W.J. and Hand, DJ. 2009. ROC Curves for Continuous Data. Boca Raton: Taylor & Francis.
25. Landis, J.R. and Koch, G.G. 1977. The measurement of observer agreement for categorical data. Biometrics. 33: 159-174.
26. Linnet, K. 1987. Comparison of quantitative diagnostic tests: type I error, power, and sample size. Statistics in Medicine 6: 147-158.
27. Lloyd, C.J. 1988. The use of smoothed ROC curves to summarize and compare diagnostic systems. JASA 93: 1356-1364.
28. Ma, G. and Hall, W.J. 1993. Confidence bands for receiver operating characteristic curves. Medical Decision Making 13: 191-197.
29. Macmillan, N.A. and Creelman, C.D. 1991. Detection Theory: A User's Guide. Cambridge: Cambridge University Press.
30. McClish, D.K. 1989. Analyzing a portion of the ROC curve. Medical Decision Making 9: 190-195.
31. Metz, C.E., Herman, B.A., and Shen, J.H. 1998. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Medical Decision Making 17: 1033-1053.
32. Moses, L.E., Shapiro, D., and Littenberg, B. 1993. Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. Statistics in Medicine 12: 1293-1316.
33. Obuchowski, N.A. 1995. Multireader receiver operating characteristic studies: a comparison of study designs. Academic Radiology 2: 709-716.
34. Obuchowsi, N.A. and Rockette, H.E. 1995. Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: an ANOVA approach with dependent observations. Communications in Statistics Simulations 24: 285-308.
35. O'Malley, A.J., Zou, K.H., Fielding, J.R. et al. 2001. Bayesian regression methodology for estimating a receiver operating characteristic curve with two radiologic applications: prostate biopsy and spiral CT of ureteral stones. Academic Radiology 8: 713-725.
36. O'Malley, A.J. and Zou, K.H. 2006. Bayesian multivariate hierarchical transformation models for ROC analysis. Statistics in Medicine 25: 459-479.
37. Pepe, M.S. 1998. Three approaches to regression analysis of receiver operating characteristic curves for continuous test results. Biometrics 54: 124-135.
38. Pepe, M.S. 2000. An interpretation for the ROC curve and inference using GLM procedures. Biometrics 56: 352-359.
39. Pepe, M.S. 2003. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford: Oxford University Press.
40. Reitsma, J.B., Glas, A.S., Rutjes, A.W. et al. 2005. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. Journal of Clinical Epidemiology. 58: 982-990.
41. Schäfer, H. 1994. Efficient confidence bounds for ROC curves. Statistics in Medicine 13: 1551-1561.
42. Siegel, S. and Castellan, N.J. 1988. Nonparametric Statistics for the Behavioral Sciences, 2nd ed. New York: McGraw-Hill.
43. Sobehart, J.R. and Keenan, S.C. 2001. Measuring default accurately, credit risk special report. Risk 14: 31-33.
44. Su, J.Q. and Liu, J.S. 1993. Linear combinations of multiple diagnostic markers. JASA 88: 1350-1355.
45. Swets, J.A. and Pickett, R.M. 1982. Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. New York: Academic Press.
46. Toledano, A.Y. and Gatsonis, C. 1999. Generalized estimating equations for ordinal categorical data: arbitrary patterns of missing responses and missingness in a key covariate. Biometrics 55: 488-496.
47. Tosteson, A.N. and Begg, C.B. 1988. A general regression methodology for ROC curve estimation. Medical Decision Making 8: 204-215.
48. Venkatraman, E.S. 2000. A permutation test to compare receiver operating characteristic curves. Biometrics 56: 1134-1138.
49. Walter, S.D. 2005. The partial area under the summary ROC curve. Statistics in Medicine 24: 2025-2040.
50. Wan, S. and Zhang, B. 2007. Smooth semiparametric receiver operating characteristic curves for continuous diagnostic tests. Statistics in Medicine 26: 2565-2586.
51. Wieand, S., Gail, M.H., James, B.R. et al. A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika 76: 585-592.
52. Zhou, X.H. and Harezlak, J. 2002. Comparison of bandwidth selection methods for kernel smoothing of ROC curves. Statistics in Medicine 21: 2045-2055.
53. Zhou, X.H., Obuchowski, N.A., and McClish, D.K. 2002. Statistical Methods in Diagnostic Medicine. New York: Wiley.
54. Zou, K.H., DeTora, L.M., Haker, S.J. et al. 2009. Revisiting the p-value: a comparison of statistical evidence in clinical and legal medical decision making. Law, Probability and Risk 8: 159-170.
55. Zou, K.H., Hall, W.J., and Shapiro, D.E. 1997. Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests. Statistics in Medicine 16: 2143-2156.
56. American College of Radiology. 2003. Breast Imaging Reporting and Data System Atlas (BI-RADS® Atlas). Mammography, 4th ed. Reston, VA: American College of Radiology. www.acr.org (accessed 12/17/2010).
57. Coffin, M. and Sukhatme, S. 1997. Receiver operating characteristic studies and measurement errors. Biometrics 53: 823-837.
58. Egan, J.P. 1975. Signal Detection Theory and ROC Analysis. New York: Academic Press.
59. Faraggi, D. 2000. The effect of random measurement error on receiver operating characteristic (ROC) curves. Statistics in Medicine 19: 61-70.
60. Gur, D., Rockette, H.E., and Bandos, A.I. 2007. "Binary" and non-binary detection tasks: are current performance measures optimal? Academic Radiology 14: 871-876.
61. Metz, C.E., Herman, B.A., and Shen, J. 1998. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously distributed data. Statistics in Medicine 17: 1033-1053.
62. Miller, G.A. 1956. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review 63: 81-98.
63. Mossman, D. 1999. Three-way ROCs. Medical Decision Making 19: 78-89.
64. Nakas, C.T. and Yiannoutsos, C.T. 2004. Ordered multiple-class ROC analysis with continuous measurements. Statistics in Medicine 23: 3437-3449.
65. Neyman J. and Pearson, E. 1933. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London Series A 231: 289-337.
66. Obuchowski, N.A. 2006. An ROC-type measure of diagnostic accuracy when the gold standard is continuous scale. Statistics in Medicine 25: 481-493.
67. Pepe, M.S. 2003. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford: Oxford University Press.
68. Rockette, H.E., Gur, D., and Metz, C.E. 1992. The use of continuous and discrete confidence judgment in ROC studies of diagnostic imaging techniques. Investigative radiology 27: 169-172.
69. Swets, J.A. and Pickett, R.M. 1982. Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. New York: Academic Press.
70. Wagner, R.F., Beiden, S.V., and Metz, C.E. 2001. Continuous versus categorical data for ROC analysis: some quantitative considerations. Academic Radiology 8: 328-334.
71. Walsh, S.J. 1997. Limitations to the robustness of binormal ROC curves: effects of model misspecification and location of decision threshold on bias, precision, size and power. Statistics in Medicine 16: 669-679.
72. Zhou, X.H., Obuchowski, N.A., and McClish D.K. 2002. Statistical Methods in Diagnostic Medicine. New York: Wiley.
73. Zou, K.H. and Hall, W.J. 2000. Two transformation models for estimating an ROC curve derived from continuous data. Journal of Applied Statistics 27: 621-631.
74. Agresti, A. and Coull, B.A. 1998. Approximate is better than "exact" for interval estimation of binomial proportions. American Statistician 52: 119-126.
75. Alonzo, T.A. and Pepe, M.S. 2002. Distribution-free ROC analysis using binary regression techniques. Biostatistics 3: 421-432.
76. Bamber, D. 1975. The area above the ordinal dominance graph and the area below the receiver operating graph. Journal of Mathematical Psychology 12: 387-415.
77. Barndorff-Nielsen, O.E., Blæsild, P., and Eriksen, P.S. 1989. Decomposition and Invariance of Measures, and Statistical Transformation Models., New York: Springer.
78. Bartko, J.J. 1966. The intraclass correlation coefficient as a measure of reliability. Psychological Reports 19: 3-11.
79. Bickel, P.J. and Doksum, K.A. 1981. An analysis of transformations revisited. Journal of the American Statistical Association 76: 296-311.
80. Bickel, P.J., Klaassen, C.A.J., Ritov, Y. et al. 1998. Efficient and Adaptive Estimation for Semiparametric Models. New York: Springer.
81. Bickel, P.J. 1986. Efficient testing in a class of transformation models. In Papers on Semiparametric Models at the ISI Centenary Session. Amsterdam, Report MS-R8614. Amsterdam: Centrum voor Wiskunde en Informatica, pp. 63-81.
82. Box, G.E.P. and Cox, D.R. 1964. An analysis of transformations. Journal of the Royal Statistical Society Series B 42: 71-78.
83. Cai, T and Moskowitz, C. 2004. Semiparametric estimation of the binormal ROC curve. Biostatistics 5: 573-586.
84. Cai, T. and Pepe, M.S. 2002. Semi-parametric ROC analysis to evaluate biomarkers for disease. Journal of the American Statistical Association 97: 1099-1107.
85. Cai, T. 2004. Semi-parametric ROC regression analysis with placement values. Biostatistics 5: 45-60.
86. Cappelleri, J.C. and Ting, N. 2003. A modified large-sample approach to approximate interval estimation for a particular intraclass correlation coefficient. Statistics in Medicine 22: 1861-1877.
87. Carroll, B.J. and Ruppert D. 1988. Transformation and Weighting in Regression. New York: Chapman & Hall.
88. D'Agostino, R.B. and Stephens, M.A. 1986. Goodness-of-Fit Techniques. New York: Marcel Dekker.
89. D'Agostino, R.B. 1971. An omnibus test of normality for moderate and large size samples. Biometrika 58: 341-348.
90. D'Agostino, R.B. 1972. Small sample probability points for the D test of normality. Biometrika 59: 319-321.
91. Doksum, K.A. 1987. An extension of partial likelihood methods for proportional hazard models to general transformation models. The Annals of Statistics 15: 325-345.
92. Dorfman, D.D., Berbaum, K.S., Metz, C,E. et al. 1997. Proper receiver operating characteristic analysis: the bigamma model. Academic Radiology 4: 138-149.
93. Du, P. and Tang, L. 2009. Transformation-invariant and nonparametric monotone smooth estimation of ROC curves. Statistics in Medicine 28: 349-359.
94. Esscher, F. 1924. On a method of determining correlation from the ranks of variates. Skandinavisk Aktuarietidskirift 7: 201-219.
95. Fieller, E.C., Hartley, H.O., and Pearson, E.S. 1957. Tests for rank correlation coefficients I. Biometrika 44: 470-481.
96. Fieller, E.C. and Pearson, E.S. 1961. Tests for rank correlation coefficients II. Biometrika 48: 29-40.
97. Filliben, J.J. 1975. The probability plot correlation coefficient test for normality. Technometrics 17: 111-118.
98. Fisher, R.A. and Yates, F. 1948. Statistical Tables for Biological, Agricultural, and Medical Research. New York: Hafner.
99. Fisher, R.A. 1915. Frequency distributions of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika 10: 507-521.
100. Fleiss, J.L. and Shrout, P.E. 1978. Approximate interval estimation for a certain intraclass correlation coefficient. Psychometrika 43: 259-262.
101. Galton, F. 1988. Correlations and their measurement, chiefly from anthropometric data. Proceedings of the Royal Society of London 45: 219-247.
102. Galton, F. 1877. Typical laws of heredity. Proceedings of the Royal Institution of Great Britain 8: 282-301.
103. Gönen, M. 2007. Analyzing Receiver Operating Characteristic Curves with SAS®. Cary, NC: SAS Institute.
104. Goria, M.N. 1982. A survey of two-sample location-scale problem, asymptotic relative efficiencies of some rank tests. Statistica Neerlandica 36: 3-13.
105. Hájek, J. and Sidájk, Z. 1967. Theory of Rank Tests. New York: Academic Press.
106. Hanley, J.A. and McNeil, B.J. 1983. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148: 839-843.
107. Hawkins, D.L. 1989. Using U statistics to derive the asymptotic distribution of Fisher's Z statistic. The American Statistician 43: 235-237.
108. Hernandez, F. and Johnson, R.A. 1980. The large-sample behavior of transformations to normality. Journal of the American Statistical Association 75: 855-861.
109. Hettmansperger, T.P. 1991. Statistical Inference Based on Ranks. Malabar: Krieger Publishing.
110. Hinkley, D.V. and Runger, G. 1984. The analysis of transformed data. Journal of the American Statistical Association 79: 302-309.
111. Hinkley, D.V. 1975. On power transformations to symmetry. Biometrika 62: 101-111.
112. Höeffding, W. 1947. On the distribution of the rank correlation coefficient t when the variates are not independent. Biometrika 34: 183-196.
113. Hsieh, F. and Turnbull, B.W. 1996. Nonparametric and semiparametric estimation of the receiver operating characteristic curve. The Annals of Statistics 24: 25-40.
114. Hsieh, F. 1995. The empirical process approach for semiparametric two-sample models with heterogeneous treatment effect. Journal of the Royal Statistical Society Series B 57: 735-748.
115. Janes, H., Longton, G. and Pepe, M. 2009. Accommodating covariates in ROC analysis. The Strata Journal 9: 17-39.
116. Kendall, M. and Gibbons, J.D. 1990. Rank Correlation Methods. New York: Oxford University Press.
117. Kendall, M.G. 1938. A new measure of rank correlation. Biometrika 30: 81-93.
118. Kochar, S.C. 1979. A class of distribution-free tests for the two sample location-scale problem. Bulletin of International Statistical Institute 47: 288-291.
119. Kruskal, W.H. 1958. Ordinal measures of association. Journal of the American Statistical Association 53: 814-861.
120. Krzanowski, W.J and Hand, D.J. 2009. ROC Curves for Continuous Data. Boca Raton: Taylor & Francis.
121. Krzanowski, W.J. 2000. Principles of Multivariate Analysis: A Users Perspective. Oxford: Oxford University Press.
122. Lambert, D. 1982. Qualitative robustness of tests. Journal of the American Statistical Association 77: 352-357.
123. Lasko, T.A., Bhagwat, J.G., Zou, K.H. et al. 2005. The use of receiver operating characteristic curves in biomedical informatics. Journal of Biomedical Informatics 38: 404-415.
124. Lehmann, E.L. 2006. Nonparametrics: Statistical Methods Based on Ranks. New York: Springer.
125. Lehmann, E.L. 1986. Testing Statistical Hypotheses. New York: Wiley.
126. Li, J. and Zhou, X.H. 2009. Nonparametric and semiparametric estimation of the three way receiver operating characteristic surface. Journal of Statistical Planning and Inference 139: 4133-4142.
127. Lin, C.C. and Mudholkar, G.S. 1980. A simple test for normality against asymmetric alternatives. Biometrika 67: 455-461.
128. Lloyd, C.J. 1988. Using smoothed ROC curves to summarize and compare diagnostic systems. Journal of the American Statistical Association 93: 1356-1364.
129. Madansky, A. 1988. Prescriptions for Working Statisticians. New York: Springer.
130. Mann, H.B. and Whitney, D.R. 1947. On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics 18: 50-60.
131. Maritz, J.S. 1981. Distribution-Free Statistical Methods. London: Chapman & Hall.
132. McClish, D.K. 1989. Analyzing a portion of the ROC curve. Medical Decision Making 9: 190-195.
133. Metz, C.E., Herman, B.A., and Shen, J.H. 1998. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously distributed data. Statistics in Medicine 17: 1033-1053.
134. Nair, V.N. 1982. Some extensions of ANOVA techniques to location-scale models. Communications in Statistics A 11: 1551-1570.
135. O'Malley, A.J., Zou, K.H., Fielding, J.R. et al. 2001. Bayesian regression methodology for estimating a receiver operating characteristic curve with two radiologic applications: prostate biopsy and spiral CT of ureteral stones. Academic Radiology 8: 713-725.
136. Obuchowski, N.A. 2005. Estimating and comparing diagnostic test accuracy when the gold standard is not binary. Academic Radiology 12: 1198-1204.
137. O'Malley, A.J. and Zou, K.H. 2006. Bayesian multivariate hierarchical transformation models for ROC analysis. Statistics in Medicine 25: 459-479.
138. Pearson, K. 1896. Mathematical contributions to the theory of evolution III. Regression, heredity and panmixia. Philosophical Transactions of the Royal Society of London Series A 187: 253-318.
139. Pearson, K. 1900. On the criterion that a given system of deviations from the probable in the case of correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine 50: 157-175.
140. Pepe, M.S. 1998. Three approaches to regression analysis of receiver operating characteristic curves for continuous test results. Biometrics 54: 124-135.
141. Pepe, M.S. 2000. An interpretation for the ROC curve and inference using GLM procedures. Biometrics 56:352-359.
142. Pepe, M.S. 2003. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford: Oxford University Press.
143. Pesarin, F. 2001. Multivariate Permutation Tests: Applications in Biostatistics. Chichester: Wiley.
144. Rajaratnam, N. 1960. Reliability formulas for independent decision data when reliability data are matched. Psychometrika 25: 261-271.
145. Ramasundarahettige, C.F., Donner, A., and Zou, G.Y. 2009. Confidence interval construction for a difference between two dependent intraclass correlation coefficients. Statistics in Medicine 28: 1041-1053.
146. Rao, C.R. 1973. Linear Statistical Inference and Its Applications, 2nd ed. New York: Wiley.
147. Rieder, H. 1982. Qualitative robustness of rank tests. Annals of Statistics 10: 205-211.
148. Satterthwaite, F.E. 1946. An approximate distribution of estimates of variance components. Biometrics 2: 110-114.
149. Shapiro, S.S. and Wilk, M.B. 1968. Approximations for the null distribution of the W statistic. Technometrics 10: 861-866.
150. Spearman, C. 1904. The proof and measurement of association between two things. American Journal of Psychology 15: 72-101.
151. Tijms, H. 2007. Understanding Probability: Chance Rules in Everyday Life. Cambridge: Cambridge University Press.
152. van der Waerden, B.L. 1952. Order tests for the two-sample problem and their power. Proceedings Koninklijke Nederlandse Akademie van Wetenschappen Series A 55: 453-458.
153. Corrigenda, J. 1953. Proceedings Koninklijke Nederlandse Akademie van Wetenschappen Series A 56: 80.
154. Venkatraman, E.S. 2000. A permutation test to compare receiver operating characteristic curves. Biometrics 56: 1134-1138.
155. Wald, A. 1943. Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical Society 54: 426-482.
156. Wieand, S., Gail, M,H., James, B,R. et al. 1989. A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika 76: 585.
157. Wilcoxon, F. 1945. Individual comparisons by ranking methods. Biometrics Bulletin 1: 80-83.
158. Wilk, M.B. and Gnanadesikan, R. 1968. Probability plotting methods for the analysis of data. Biometrika 55: 1-17.
159. Wilson, E.B. 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22: 209-212.
160. Zhou, X.H. and Harezlak, J. 2002. Comparison of bandwidth selection methods for kernel smoothing of ROC curves. Statistics in Medicine 21: 2045-2055.
161. Zou, K.H, Hall, W.J., and Shapiro, D.E. 1997. Smooth nonparametric receiver operating characteristic (ROC) curves for continuous diagnostic tests. Statistics in Medicine 16: 2143-2156.
162. Zou, K.H. and McDermott, M.P. 1999. Higher-moment approaches to approximate interval estimation for a certain intraclass correlation coefficient. Statistics in Medicine 18: 2051-2061.
163. Zou, K.H., Wells, W.M., III, Kikinis, R. et al. 2004. Three validation metrics for automated probabilistic image segmentation of brain tumours. Statistics in Medicine 23: 1259-1282.
164. Zou, K.H., Carlsson, M.O., and Quinn, S.A. 2010. Beta-mapping and beta-regression for changes of ordinal-rating measurements on Likert scales: comparison of the change scores among multiple treatment groups. Statistics in Medicine.
165. Baker, S. 2000. Identifying combinations of cancer markers for further study as triggers of early intervention. Biometrics 56: 1082-1087.
166. Baker, S. 2003. The central role of receiver operating characteristic (ROC) curves in evaluating tests for the early detection of cancer. Journal of the National Cancer Institute 95: 511-515.
167. Barcellos, L.F., Klitz, W., Field, L.L. et al. 1997. Association mapping of disease loci by use of a pooled DNA genomic screen. American Journal of Human Genetics 61: 734-747.
168. Chen, C.L. and Swallow, W.H. 1990. Using group testing to estimate a proportion and to test the binomial model. Biometrics 46: 1035-1046.
169. Constantine, K., Karson, M. et al. 1986. Estimation of P(Y < X) in the gamma case. Communication Statistics Simulation 15: 365-388.
170. Cook, R.D. and Weisberg, S. 1991. Discussion of Li. Journal of the American Statistical Association 86: 328-332.
171. Dorfman, R. 1943. The detection of defective members of large populations. Annals of Mathematical Statistics 14: 436-440.
172. Efron, B. and Tibshirani, R. 1993. An Introduction to the Bootstrap. Boca Raton: Chapman & Hall.
173. Eguchi, S. and Copas, J.B. 2002. A class of logistic-type discriminant functions. Biometrika 89: 1-22.
174. Enard, W., Khaitovich, P., Klose, J. et al. 2002. Intra- and inter-specific variation in primate gene expression patterns. Science 296: 340-343.
175. Evans, M., Hastings, N., and Peacock, B. 2000. Statistical Distributions. 3rd Ed. New York: Wiley.
176. Faraggi, D., Reiser, B., and Schisterman, E.F. 2003. ROC curve analysis for biomarkers based on pooled assessments. Statistics in Medicine 22: 2515-2527.
177. Farrington, C. (1992). Estimation prevalence by group testing using generalized linear models. Statistics in Medicine 11: 1591-1597.
178. Gastwirth, J. and Johnson, W. 1994. Screening with cost-effective quality control: potential applications to HIV and drug testing. Journal of the American Statistical Association 89: 972-981.
179. Green, D.M. and Swets, J.A. 1966. Signal Detection Theory and Psychophysics. New York: Wiley.
180. Hughes-Oliver, J.M. and Swallow, W.H. 1994. A two-stage adaptive group-testing procedure for estimating small proportions. Journal of the American Statistical Association 89: 982-993.
181. Jin, W., Riley, R.M., Wolfinger, R.D. et al. 2001. The contributions of sex: genotype and age to transcriptional variance in Drosophila melanogaster. Nature Genetics 29: 389-395.
182. Kendziorski, C.M., Zhang, Y., Lan, H. et al. 2003. The efficiency of pooling mRNA in microarray experiments. Biostatistics 4: 465-477.
183. Li, K.C. 1991. Sliced inverse regression for dimension reduction. Journal of the American Statistical Association 86: 316-342.
184. Liu, A. and Schisterman, E.F. 2003. Comparison of diagnostic accuracy of biomarkers with pooled assessments. Biometrical Journal 45: 631-644.
185. Liu, A., Schisterman, E.F., and Zhu, Y. 2005. On linear combinations of biomarkers to improve diagnostic accuracy. Statistics in Medicine 24: 37-47.
186. Litvak, E., Tu, X.M., and Pagano, M. 1994. Screening for the presence of a disease by pooling sera samples. Journal of the American Statistical Association 89: 424-434.
187. Ma, S. and Huang, J. 2005. Regularized ROC method for disease classification and biomarker selection with microarray data. Bioinformatics 21: 4356-4362.
188. Ma, S. and Huang, J. 2007. Combining multiple markers for classification using ROC. Biometrics 63: 751-757.
189. Mumford, S.L., Schisterman, E.F., Vexler, A. et al. 2006. Pooling biospecimens and limits of detection: effects on ROC curve analysis. Biostatistics 7: 585-598.
190. Neyman, J. and Pearson, E.S. 1933. On the problem of the most efficient tests of statistical hypothesis. Philosophical Transactions of the Royal Society of London Series A 231: 289-337.
191. O'Malley, A.J. and Zou, K.H. 2006. Bayesian multivariate hierarchical transformation models for ROC analysis. Statistics in Medicine 25: 459-479.
192. Pepe, M.S., Cai, T., and Longton, G. 2006. Combining predictors for classification using the area under the receiver operating characteristic curve. Biometrics 62: 221-229.
193. Pepe, M.S. and Thompson, M.L. 2000. Combining diagnostic test results to increase accuracy. Biostatistics 1: 123-140.
194. Pfeiffer, R.M. and Bura, E. 2008. A model free approach to combining biomarkers. Biometrical Journal 50: 558-570.
195. Rao, C.R. 1972. Linear Statistical Inference and Its Applications. New York: Wiley.
196. Reiser, B. and Faraggi, D. 1997. Confidence intervals for the generalized ROC criterion. Biometrics 53: 644-652.
197. Sobel, M. and Elashoff, R. 1975. Group testing with a new goal: estimation. Biometrika 62: 181-193.
198. Su, J.Q. and Liu, J.S. 1993. Linear combinations of multiple diagnostic markers. Journal of the American Statistical Association 88: 1350-1355.
199. Tu, X.M., Litvak, E., and Pagano, M. 1995. On the informativeness and accuracy of pooled testing in estimating prevalence of a rare disease: application to HIV screening. Biometrika 82: 287-297.
200. Vexler, A., Liu, A., and Schisterman, E.F. 2006. Efficient design and analysis of biospecimens with measurements subject to detection limit. Biometrical Journal 48: 780-791.
201. Vexler, A., Liu, A., and Schisterman, E.F. 2010. Nonparametric deconvolution of density estimation based on observed sums. Journal of Nonparametric Statistics 22: 23-29.
202. Vexler, A., Schisterman, E.F., and Liu, A. 2008. Estimation of ROC based on stably distributed biomarkers subject to measurement error and pooling mixtures. Statistics in Medicine 27: 280-296.
203. Vexler, A., Liu, A., Schisterman, E.F. et al. 2006. Note on distribution-free estimation of maximum linear separation of two multivariate distributions. Journal of Nonparametric Statistics 18: 145-158.
204. Weinberg, C.R. and Umbach, D.M. 1999. Using pooled exposure assessment to improve efficiency in case-control studies. Biometrics 55: 718-726.
205. Black, M.A. and Craig, B.A. 2002. Estimating disease prevalence in the absence of a gold standard. Statistics in Medicine 21: 2653-2669.
206. Branscum, A.J., Johnson, W.O., Hanson, T.E. et al. 2008. Bayesian semiparametric ROC curve estimation and disease diagnosis. Statistics in Medicine 27: 2474-2496.
207. Broemeling, L.D. 2007. Bayesian Biostatistics and Diagnostic Medicine. Boca Raton: Chapman & Hall.
208. Cao, J., Lee, J.J., and Alber, S. 2009. Comparison of Bayesian sample size criteria: ACC, ALC, and WOC. Journal of Statistical Planning and Inference 139: 4111-4122.
209. Carlin, B.P. and Louis, T.A. 2000. Bayes and Empirical Bayesian Methods for Data Analysis. London: Chapman & Hall.
210. Chen, M.H., Shao, Q.M., and Ibrahim, J.G. 2000. Monte Carlo Methods in Bayesian Computation. New York: Springer.
211. Choi, Y.K., Johnson, W.O., Collins, M.T. et al. 2006. Bayesian inferences for receiver operating characteristic curves in the absence of a gold standard. Journal of Agricultural, Biological, and Environmental Statistics 11: 210-229.
212. Dendukuri, N. and Joseph, L. 2001. Bayesian approaches to modeling the conditional dependence between multiple diagnostic tests. Biometrics 57: 158-167.
213. Dendukuri, N., Rahme, E., Bélisle, P. et al. 2004. Bayesian sample size determination for prevalence and diagnostic test studies in the absence of a gold standard. Biometrics 60: 388-397.
214. Donner, A. and Klar, N. 2004. Pitfalls of and controversies in cluster randomization trials. American Journal of Public Health 94: 416-422.
215. Erkanli, A., Sung, M., Costello, E.J. et al. 2006. Bayesian semi-parametric ROC analysis. Statistics in Medicine 25: 3905-3928.
216. Fleiss, J.L. 1986. The Design and Analysis of Clinical Experiments. New York: Wiley.
217. Fluss, R., Faraggi, D., and Reiser, B. 2005. Estimation of the Youden index and its associated cutoff point. Biometrical Journal 47: 458-472.
218. Gammerman, D. and Lopes, H.F. 2006. Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference. Boca Raton: Chapman & Hall.
219. Gelman, A., Carlin, J.B., Stern, H.S. et al. 1995. Bayesian Data Analysis. London: Chapman & Hall.
220. Gu, J., Ghosal, S., and Roy, A. 2008. Bayesian bootstrap estimation of ROC curve. Statistics in Medicine 27: 5407-5420.
221. Hellmich, M., Abrams, K.R., Jones, D.R. et al. A Bayesian approach to a general regression model for ROC curves. Medical Decision Making 18: 436-443.
222. Ishwaran, H. and Gatsonis, C.A. 2000. A general class of hierarchical ordinal regression models with applications to correlated ROC analysis. Canadian Journal of Statistics 28: 731-775.
223. Jeffreys, H. 1961. Theory of Probability. Oxford: Oxford University Press.
224. Joseph, L., du Berger, R., and Bélisle, P. 1997. Bayesian and mixed Bayesian likelihood criteria for sample size determination. Statistics in Medicine 16: 769-781.
225. Joseph, L., Gyorkos, T.W., and Coupal, L. 1995. Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard. American Journal of Epidemiology 141: 263-272.
226. Joseph, L. and Wolfson, D.B. 1997. Interval-based versus decision theoretic criteria for the choice of sample size. Statistician 46: 145-149.
227. Kass, R.E. and Wasserman, L. 1996. The selection of prior distributions by formal rules. Journal of the American Statistical Association 91: 1343-1370.
228. Kass, R.E. and Wasserman, L. 1998. The selection of prior distributions by formal rules. Journal of the American Statistical Association 93: 412.
229. Lunn, D., Spiegelhalter, D., Thomas, A. et al. 2009. The BUGS project: evolution, critique and future directions. Statistics in Medicine 28: 3049-3067.
230. Metz, C.E., Herman, B.A., and Shen, J.H. 1998. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously distributed data. Statistics in Medicine 17: 1033-1053.
231. Miller, S.W., Sinha, D., Slate, E.H. et al. 2009. Bayesian adaptation of the summary ROC Curve method for meta-analysis of diagnostic test performance. Journal of Data Science 7: 349-364.
232. Normand, S.L. and Zou, K,H. 2002. Sample size considerations in observational health care quality studies. Statistics in Medicine 21: 331-345.
233. O'Malley, A.J., Zou, K.H., Fielding, J.R. et al. C.M. 2001. Bayesian regression methodology for estimating a receiver operating characteristic curve with two radiologic applications: prostate biopsy and spiral CT of ureteral stones. Academic Radiology 8: 713-725.
234. O'Malley, A.J. and Zou, K.H. 2006. Bayesian multivariate hierarchical transformation models for ROC analysis. Statistics in Medicine 25: 459-479.
235. Parmigiani, G. 2002. Measuring uncertainty in complex decision analysis models. Statistical Methods in Medical Research 11: 513-537.
236. Peng, F. and Hall, W. J. 1996. Bayesian analysis of ROC curves using Markov chain Monte Carlo methods. Medical Decision Making 16: 404-411.
237. Resnic, F.S., Ohno-Machado, L., Selwyn et al. 2001. Simplified risk score models accurately predict the risk of major in-hospital complications following percutaneous coronary intervention. American Journal of Cardiology 88: 5-9.
238. Rutter, C.M. and Gatsonis, C.A. 2001. A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Statistics in Medicine 20: 2865-2884.
239. Shoukri, M.M. 2004. Measures of Inter-Observer Agreement. Boca Raton: Chapman & Hall.
240. Wang, F. and Gatsonis, C.A. 2008. Hierarchical models for ROC curve summary measures: design and analysis of multi-reader, multi-modality studies of medical tests. Statistics in Medicine 27: 243-256.
241. Yee, T.W. 2010. The VGAM package for categorical data analysis. Journal of Statistical Software 32: 1-34.
242. Zou, G.Y., Donner, A., and Klar, N. 2005. Group sequential methods for cluster randomization trials with binary outcomes. Clinical Trials 2: 479-487.
243. Zou, K.H. and Normand, S.L. 2001. On determination of sample size in hierarchical binomial models. Statistics in Medicine 20: 2163-2182.
244. Zou, K.H. and O'Malley, A.J. 2005. A Bayesian hierarchical nonlinear regression model in receiver operating characteristic analysis of clustered continuous diagnostic data. Biometrical Journal 47: 417-427.
245. Zou, K.H., Resnic, F.S., Gogate, A. et al. 2003. Efficient Bayesian sample size calculation for designing a clinical trial with multicluster outcome data. Biometrical Journal 45: 826-836.
246. Bamber, D. 1975. The area above the ordinal dominance graph and the area below the receiver operating characteristics graph. Journal of Mathematical Psychology 12: 387-415.
247. Chang, M.N. 1989. Confidence intervals for a normal mean following a group sequential test. Biometrics 45: 247-254.
248. DeLong, E.R., DeLong, D.M., and Clarke-Pearson, D.L. 1988. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44: 837-845.
249. Emerson, S.S. and Fleming, T.R. 1990. Parameter estimation following sequential hypothesis testing. Biometrika 77: 875-892.
250. Fairbanks, K. and Madsen, R. 1982. P values for tests using a repeated significance design. Biometrika 69: 69-74.
251. Fleming, T.R. 1982. One-sample multiple testing procedure for phase II clinical trials. Biometrics 38: 143-151.
252. Gehan, E.A. 1961. The determination of the number of patients required in a preliminary and a follow-up trial of a new chemotherapeutic agent. Journal of Chronic Diseases 13: 346-353.
253. Hanley, J.A. and McNeil, B.J. 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143: 29-36.
254. Hwang, I.K., Shih, W.J., and DeCani, J. 1990. Group sequential designs using a family of type I error probability spending functions. Statistics in Medicine 9: 1439-1445.
255. Jennison, C. and Turnbull, B.W. 2000. Group Sequential Methods with Applications to Clinical Trials. New York: Chapman & Hall.
256. Kim, K. and DeMets, D.L. 1987. Design and analysis of group sequential tests based on the type I error spending rate function. Biometrika 74: 149-154.
257. Lan, K.K.G. and DeMets, D.L. 1983. Discrete sequential boundaries for clinical trials. Biometrika 70: 659-663.
258. Lehmann, E.L. 1999. Elements of Large-Sample Theory. New York: Springer.
259. Liu, A. and Hall, W.J. 1999. Unbiased estimation following a group sequential test. Biometrika 86: 71-78.
260. Liu, A. and Hall, W.J. 2001. Unbiased estimation of secondary parameters following a sequential test. Biometrika 88: 895-900.
261. Liu, A. and Schisterman, E.F. 2003. Comparison of diagnostic accuracy of biomarkers with pooled assessments. Biometrical Journal 45: 631-644.
262. Liu, A. Boyett, J., and Xiong, X.P. 2000. Sample size calculation for planning group sequential longitudinal trials. Statistics in Medicine 19: 205-220.
263. Liu, A., Wu, C.Q. and Schisterman, E.F. 2008. Nonparametric sequential evaluation of diagnostic biomarkers. Statistics in Medicine 27: 1667-1678.
264. Mazumdar, M. 2004. Group sequential design for comparative diagnostic accuracy studies: implications and guidelines for practitioners. Medical Decision Making 24: 525-533.
265. Mazumdar, M. and Liu, A. 2003. Group sequential design for comparative diagnostic accuracy studies. Statistics in Medicine 22: 727-739.
266. Obuchowski, N.A. 1994. Computing sample size for receiver operating characteristic studies. Investigative Radiology 29: 238-243.
267. Obuchowski, N.A. 1997. Nonparametric analysis of clustered ROC curve data. Biometrics 53: 567-578.
268. Obuchowski, N.A. 1998. Sample size considerations in studies of test accuracy. Statistical Methods in Medical Research 7: 371-392.
269. Obuchowski, N.A. and McClish, D.K. 1997. Sample size determination for diagnostic accuracy studies involving binormal ROC curve indices. Statistics in Medicine 16: 1529-1542.
270. O'Brien, P.C. and Fleming, T.R. 1979. A multiple testing procedure for clinical trials. Biometrics 35: 549-556.
271. Pepe, M.S. 2003. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford: University Press.
272. Pocock, S.J. 1977. Group sequential methods in the design and analysis of clinical trials. Biometrika 64: 191-199.
273. Proschan, M.A., Lan, K.K.G., and Wittes, J.T. 2006. Statistical Monitoring of Clinical Trials: A Unified Approach. New York: Springer.
274. Rao, C.R. 1973. Linear Statistical Inference and Its Applications, 2nd ed. New York: Wiley.
275. Rosner, G.L. and Tsiatis, A.A. 1988. Exact confidence intervals following a group sequential trial: A comparison of methods. Biometrika 75: 723-729.
276. Siegmund, D. 1978. Estimation following sequential tests. Biometrika 65: 295-297.
277. Shu, Y., Liu, A., and Li, Z.H. 2007. Sequential evaluation of a medical diagnostic test with binary outcomes. Statistics in Medicine 26: 4416-4427.
278. Simon, R. 1989. Optimal two-stage designs for phase II clinical trials. Controlled Clinical Trials 10: 1-10.
279. Tang, L. and Liu, A. 2010. Sample size recalculation in sequential diagnostic trials. Biostatistics 11: 151-163.
280. Tang, L., Emerson, S.S., and Zhou, X.H. 2008. Nonparametric and semiparametric group sequential methods for comparing accuracy of diagnostic tests. Biometrics 64: 1137-1145.
281. Wald, A. 1945. Sequential tests of statistical hypotheses. Annals of Mathematical Statistics 16: 117-186.
282. Wang, S.K. and Tsiatis, A.A. 1987. Approximately optimal one-parameter boundaries for group sequential trials. Biometrics 43: 193-199.
283. Wassmer, G. and Vandemeulebroecke, M. 2006. A brief review on software developments for group sequential and adaptive designs. Biometrical Journal 48: 732-737.
284. Wieand, S., Gail, M.H., James, B.R. et al. 1989. A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika 75: 585-592.
285. Whitehead, J. 1986a. On the bias of maximum likelihood estimation following a sequential test. Biometrika 73: 573-581.
286. Whitehead, J. 1986b. Supplementary analysis at the conclusion of a sequential clinical trial. Biometrics 42: 461-471.
287. Whitehead, J. 1997. The Design and Analysis of Sequential Clinical Trials, 2nd ed. Chichester: Wiley.
288. Whitehead, J. 1999. A unified theory for sequential clinical trials. Statistics in Medicine 18: 2271-2286.
289. Wu, C., Liu, A., and Yu, K.F. 2008. An adaptive approach to designing comparative diagnostic accuracy studies. Journal of Biopharmaceutical Statistics 18: 116-125.
290. Zhou, X.H., Li, S., and Gatsonis, C. 2008. Wilcoxon based group sequential designs for comparison of areas of two correlated ROC curves. Statistics in Medicine 27: 213-223.
291. Arvesen, J.N. 1969. Jack-knifing U-statistics. Annals of Mathematical Statistics 40: 2076-2100.
292. Bamber, D. 1975. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology 12: 387-415.
293. Bandos, A.I., Rockette, H.E., and Gur, D. 2005. A permutation test sensitive to differences in areas for comparing ROC curves from a paired design. Statistics in Medicine 24: 2873-2893.
294. Bandos, A.I., Rockette, H.E., and Gur, D. 2006. Resampling methods for the area under the ROC curve. Proceedings of ICML Workshop on ROC Analysis in Machine Learning pp. 1-8.
295. Bandos, A.I., Rockette, H.E., and Gur, D. 2006. A permutation test for comparing ROC curves in multireader studies. Academic Radiology 13: 414-420.
296. Bandos, A.I., Rockette, H.E., and Gur, D. 2007. Exact bootstrap variances of the area under the ROC curve. Communications in Statistics: Theory & Methods 36: 2443-2461.
297. Beam, C.A. 1998. Analysis of clustered data in receiver operating characteristic studies. Statistical Methods in Medical Research 7: 324-336.
298. Beiden, S.V., Wagner, R.F., and Campbell, G. 2000. Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects receiver operating characteristic analysis. Academic Radiology 7: 341-349.
299. Davison, A.C. and Hinkley, D.V. 1997. Bootstrap Methods and Their Application. Edinburgh: Cambridge University Press.
300. DBM MRMC software. http://perception.radiology.uiowa.edu/Software/Receiver OperatingCharacteristicROC/DBMMRMC/tabid/116/Default.aspx (accessed December 29, 2010).
301. DeLong, E.R., DeLong, D.M., and Clarke-Pearson, D.L. 1988. Comparing the area under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44: 837-845.
302. Dorfman, D.D. and Alf, E., Jr. 1969. Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals: rating-method data. Journal of Mathematical Psychology 6: 487-496.
303. Dorfman, D.D., Berbaum, K.S., and Metz, C.E. 1992. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jack-knife method. Investigative Radiology 27: 723-731.
304. Gallas, B.D., Penello, G.A., and Myers, K.L. 2007. Multireader multicase variance analysis for binary data. Journal of the Optical Society of America 24: B70-B80.
305. Gönen, M. 2007. Analyzing Receiver Operating Characteristic Curves with SAS®. Cary, NC: SAS Institute (http://works.bepress.com/mithat_gonen/).
306. Gu, J., Ghosal, S., and Roy, A. 2008. Bayesian bootstrap estimation of ROC curve. Statistics in Medicine 27: 5407-5420.
307. Gur, D., Bandos, A.I., Fuhrman, C.R. et al. 2007. The prevalence effect in laboratory environment: changing the confidence ratings. Academic Radiology 14: 49-53.
308. Hanley, J.A. and Hajian-Tilaki, K.O. 1997. Sampling variability of nonparametric estimates of the area under the receiver operating characteristic curves: an update. Academic Radiology 4: 49-58.
309. Hedeker, D. and Gibbons, R.D. 1994. A random-effects ordinal regression model for multilevel analysis. Biometrics 50: 993-944.
310. Hillis, S.L. 2007. A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Statistics in Medicine 26: 596-619.
311. Hillis, S. L. 2009. Multireader ROC analysis In Handbook of Medical Image Perception and Techniques, Samei, E. et al., Eds., Cambridge: Cambridge University Press.
312. Hillis, S.L. and Obuchowski, N.A. 2011. Power estimation for multireader ROC methods: an updated and unified approach. Academic Radiology (in press).
313. Hillis, S.L., Berbaum, K.S., and Metz, C.E. 2008. Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Academic Radiology 15: 647-661.
314. Hillis, S.L., Obuchowski, N.A., Schartz, K.M. et al. 2005. Comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data. Statistics in Medicine 24: 1579-1607.
315. Hoffman, E.B., Sen, P.K., and Weinberg, C.R. 2001. Within-cluster resampling. Biometrika 88: 1121-1134.
316. Ishwaran H. 2000. Univariate and multirater ordinal cumulative link regression with covariate specific cut points. Canadian Journal of Statistics 28: 715-730.
317. Ishwaran, H. and Gatsonis, C.A. 2000. A general class of hierarchical ordinal regression models with applications to correlated ROC analysis. Canadian Journal of Statistics 28: 731-750.
318. Janes, H. and Pepe, M.S. 2009. Adjusting for covariate effects on classification accuracy using the covariate-adjusted receiver operating characteristic curve. Biometrika 96: 371-382.
319. Lee, M.L.T. and Rosner, B.A. 2001. The average area under correlated receiver operating characteristic curves: a nonparametric approach based on generalized two-sample Wilcoxon statistics. Applied Statistics 50: 337-344.
320. Liang, K.Y. and Zeger, S.L. 1986. Longitudinal data analysis using generalized linear models. Biometrika 73: 13-22.
321. Obuchowski, N.A. 1995. Multireader receiver operating characteristic studies: a comparison of study designs. Academic Radiology 2: 709-716.
322. Obuchowski, N.A. and Rockette, H.E. 1995. Hypothesis testing of the diagnostic accuracy for multiple diagnostic tests: an ANOVA approach with dependent observations. Communications in Statistics: Simulation and Computation 24: 285-308.
323. Obuchowski, N.A. 1997. Nonparametric analysis of clustered ROC curve data. Biometrics 53: 567-578.
324. Obuchowski, N.A. 2009. Reducing the number of reader interpretations in MRMC studies. Academic Radiology 16: 209-217.
325. Obuchowski, N.A., Lieber, M.L., and Powel, K.A. 2000. Data analysis for detection and localization of multiple abnormalities with application to mammography. Academic Radiology 7: 516-525.
326. Obuchowski, N.A. and McClish, D.K. 1997. Sample size determination for diagnostic accuracy studies involving binormal ROC curve indices. Statistics in Medicine 16: 1529-1542.
327. O'Malley, A.J. and Zou, K.H. 2006. Bayesian multivariate hierarchical transformation models for ROC analysis. Statistics in Medicine 25: 459-479.
328. Pepe, M.S., Urban, N., Rutter, C. et al. 1997. Design of a study to improve accuracy in reading mammograms. Journal of Clinical Epidemiology 50: 1327-1338.
329. Pepe, M.S. 1998. Three approaches to regression analysis of receiver operating characteristic curves for continuous test results. Biometrics 54: 124-135.
330. Pepe, M.S. 2003. The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford: Oxford University Press (http://labs.fhcrc.org/pepe/book/).
331. Rao, J.N.K. and Scott, A.J. 1992. A simple method for the analysis of clustered binary data. Biometrics 48: 577-585.
332. Rockette, H.E., Campbell, W.L., Britton, C.A. et al. 1999. Empiric assessment of parameters that affect the design of multireader receiver operating characteristic studies. Academic Radiology 6: 723-729.
333. Song, H.H. 1997. Analysis of correlated ROC areas in diagnostic testing. Biometrics 53: 370-382.
334. Song, X. and Zhou, X.H. 2005. A marginal model approach for analysis of multi-reader multi-test receiver operating characteristic (ROC) data. Biostatistics 6: 303-312.
335. Swets, J.A. and Pickett, R.M. 1982. Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. New York: Academic Press.
336. Thompson, M.L. and Zucchini, W. 1989. On the statistical analysis of ROC curves. Statistics in Medicine 8: 1277-1290.
337. Tosteson, A.N.A. and Begg, C.B. 1988. A general regression methodology for ROC curve estimation. Medical Decision Making 8: 204-215.
338. Toledano, A.Y. and Gatsonis, G. 1996. Ordinal regression methodology for ROC curves derived from correlated data. Statistics in Medicine 15: 1807-1826.
339. Toledano, A.Y. 2003. Three methods for analyzing correlated ROC curves: a comparison in real data sets from multi-reader, multi-case studies with a factorial design. Statistics in Medicine 22: 2919-2933.
340. Venkatraman, E.S. and Begg, C.B. 1996. A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment. Biometrika 83: 835-848.
341. Wagner, R.F., Beiden, S.V., Campbell, G. et al. 2002. Assessment of medical imaging and computer-assist systems: lessons from recent experience. Academic Radiology 9: 1264-1277.
342. Wang, F. and Gatsonis, C.A. 2008. Hierarchical models for ROC curve summary measures: design and analysis of multi-reader, multi-modality studies of medical tests. Statistics in Medicine 27: 243-256.
343. Wieand, H.S., Gail, M.M., and Hanley, J.A. 1983. A nonparametric procedure for comparing diagnostic tests with paired or unpaired data. IMS. Bulletin 12: 213-214.
344. Zeger, S.L., Liang, K.Y., and Albert, P.S. 1988. Models for longitudinal data: a generalized estimating equation approach. Biometrics 44: 1049-1060.
345. Zhou, X.H., Obuchowski, N.A., and McClish, D.K. 2002. Statistical methods in diagnostic medicine. New York: Wiley (ftp://ftp.wiley.com/public/sci_tech_med/statistical_methods).
346. Arvesen, J.N. 1969. Jackknifing U-statistics. Annals of Mathematical Statistics 40: 2076-2100.
347. Bandos, A.I., Rockette H.E., and Gur D. 2010. Use of likelihood ratios for comparisons of binary diagnostic tests: underlying ROC curves. Medical Physics 37: 5821-5830.
348. Bandos, A.I., Rockette, H.E., and Gur, D. 2006. Resampling methods for the area under the ROC curve. Proceedings of ICML Workshop on ROC Analysis in Machine Learning, pp. 1-8.
349. Bandos, A.I., Rockette, H.E., Song, T. et al. 2009. Area under the free-response ROC curve (FROC) and a related summary index. Biometrics 65: 247-256.
350. Beam, C.A. 1998. Analysis of clustered data in receiver operating characteristic studies. Statistical Methods in Medical Research 7: 324-336.
351. Bunch, P.C., Hamilton, J.F., Sanderson, G.K. et al. 1978. A free-response approach to the measurement and characterization of radiographic-observer performance. Journal of Applied Photographic Engineering 4: 165-171.
352. Chakraborty, D.P. 1989. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Medical Physics 16: 561-568.
353. Chakraborty, D.P. 2000. Data analysis for detection and localization of multiple abnormalities with application to mammography. Academic Radiology 7: 553-556.
354. Chakraborty, D.P. 2006. A search model and figure of merit for observer data acquired to the free-response paradigm. Physics in Medicine and Biology 51: 3449-3462.
355. Chakraborty, D.P. 2008. Validation and statistical power comparison of methods for analyzing free-response observer performance studies. Academic Radiology 15: 1554-1566.
356. Chakraborty, D.P. and Berbaum, K.S. 2004. Observer studies involving detection and localization: modeling, analysis and validation. Medical Physics 31: 2313-2330.
357. Chakraborty, D.P. and Winter, L.H.L. 1990. Free-response methodology: alternate analysis and a new observer-performance experiment. Medical Physics 174: 873-881.
358. Chakraborty, D.P. and Yoon, H.J. 2008. Operating characteristics predicted by models for diagnostic tasks involving lesion localization. Medical Physics 35: 435-445.
359. Chakraborty, D.P. and Yoon, H.J. 2009. JAFROC analysis revisited: figure-of-merit considerations for human observer studies. In Proceedings of SPIE Conference on Medical Imaging: Perception, Observer Performance, and Technology Assessment.
360. Chakraborty, D.P., Yoon, H.J., and Mello-Thoms, C. 2007. Spatial localization accuracy of radiologists in free-response studies: inferring perceptual FROC curves from mark-rating data. Academic Radiology 14: 4-18.
361. Dorfman, D.D. and Alf, E., Jr. 1969. Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals: rating-method data. Journal of Mathematical Psychology 6: 487-496.
362. Edwards, D.C., Kupinski, M.A., Metz, C.E. et al. 2002. Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Medical Physics 29: 2861-2870.
363. Egan, J.P., Greenberg, G.Z., and Schulman, A.I. 1961. Operating characteristics, signal delectability, and the methods of free response. Journal of the Acoustical Society of America 33: 993-1007.
364. Gallas, B.D., Penello, G.A., and Myers, K.L. 2007. Multireader multicase variance analysis for binary data. Journal of the Optical Society of America 24: B70-B80.
365. Gur, D., Bandos, A.I., and Rockette, H.E. 2008. Comparing areas under the receiver operating characteristic curves: the potential impact of the "last" experimentally measured operating point. Radiology 247: 12-15.
366. Gur, D. and Rockette, H.E. 2008. Performance assessments of diagnostic systems under the FROC paradigm: experimental, analytical and results interpretation issues. Academic Radiology 15: 1312-1315.
367. Hajian-Tilaki, K.O., Hanley, J.A., Lawrance, J. et al. 1997. Extension of receiver operating characteristics analysis to data concerning multiple signal detection tasks. Academic Radiology 4: 222-229.
368. Hoffman, E.B., Sen, P.K., and Weinberg, C.R. 2001. Within-cluster resampling. Biometrika 88: 1121-1134.
369. Hutchinson, T.P. 2007. Free-response operator characteristic models for visual search. Physics in Medicine and Biology 52: L1-L3.
370. Irvine, J.M. 2004. Assessing target search performance: the free-response operator characteristic model. Optical Engineering 43: 2926-2934.
371. Metz, C.E., Herman, B.A., and Shen, J. 1998. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously distributed data. Statistics in Medicine 17: 1033-1053.
372. Metz, C.E., Starr, S.J., and Lusted, L.B. 1976. Observer performance in detecting multiple radiographic signals. Radiology 121: 337-347.
373. Norman, D.A. 1964. A comparison of data obtained with different false-alarm rates. Psychological Review 71: 243-246.
374. Obuchowski, N.A. 1997. Nonparametric analysis of clustered ROC curve data. Biometrics 53: 567-578.
375. Obuchowski, N.A. 1998. On the comparison of correlated proportions for clustered data. Statistics in Medicine 17: 1495-1507.
376. Obuchowski, N.A., Mazzone, P.J., and Dachman, A.H. 2010. Bias, underestimation of risk, and loss of statistical power in patient-level analyses of lesion detection. European Radiology 20: 584-594.
377. Obuchowski, N.A., Lieber, M.L., and Powel, K.A. 2000. Data analysis for detection and localization of multiple abnormalities with application to mammography. Academic Radiology 7: 516-525.
378. Perkins, N.J., Schisterman, E.F., and Vexler, A. 2007. Receiver operating characteristic curve inference from a sample with a limit of detection. American Journal of Epidemiology 165: 325-333.
379. Popescu, L.M. 2007. Nonparametric ROC and LROC analysis. Medical Physics 34: 1556-1564.
380. Popescu, L.M. 2008. Model for the detection of signals in images with multiple suspicious locations. Medical Physics 35: 5565-5574.
381. Rao, J.N.K. and Scott, A.J. 1992. A simple method for the analysis of clustered binary data. Biometrics 48: 577-585.
382. Samuelson, F.W. and Petrick, N. 2006. Comparing image detection algorithms using resampling. In Proceedings of Third IEEE International Symposium, Biomedical Imaging: Macro to Nano, Arlington, VA, pp. 1312-1315.
383. Schisterman, E.F., Reiser, B., and Faraggi, D. 2006. ROC analysis for markers with mass at zero. Statistics in Medicine 25: 623-638.
384. Song, T., Bandos, A.I., Rockette, H.E. et al. 2008. On comparing methods for discriminating between actually negative and actually positive subjects with FROC type data. Medical Physics 35: 1547-1558.
385. Starr, S.J., Metz, C.E., Lusted, L.B. et al. 1975. Visual detection and localization of radiographic images. Radiology 116: 533-538.
386. Swensson, R.G. 1996. Unified measurement of observe performance in detecting and localizing target objects on images. Medical Physics 23: 1709-1725.
387. Swensson, R.G. 2000. Using localization data from image interpretations to improve estimates of performance accuracy. Medical Decision Making 20: 170-185.
388. Swensson, R.G., King, J.L., and Gur, D. 2001. A constrained formulation for the receiver operating characteristic (ROC) curve based on probability summation. Medical Physics 28: 1597-1609.
389. Tang, L. and Balakrishnan N. 2011. A random-sum Wilcoxon statistic with applications to various types of diagnostic data. Journal of Statistical Planning and Inference 141: 335-344.
390. Toledano, A.Y. and Gatsonis, G. 1996. Ordinal regression methodology for ROC curves derived from correlated data. Statistics in Medicine 15: 1807-1826.
391. Tosteson, A.N.A. and Begg, C.B. 1988. A general regression methodology for ROC curve estimation. Medical Decision Making 8: 204-215.
392. Wieand, H.S., Gail, M.M., and Hanley, J.A. 1983. A nonparametric procedure for comparing diagnostic tests with paired or unpaired data. IMS Bulletin 12: 213-214.
393. Yoon, H.J., Zheng, B., Sahiner, B. et al. 2007. Evaluating computer-aided detection algorithms. Medical Physics 34: 2024-2038.
394. Zheng, B., Chakraborty, D.P., Rockette, H.E. et al. 2005. Comparison of two data analyses from two observer performance studies using jackknife ROC and JAFROC. Medical Physics 32: 1031-1034.
395. Zhou, X.H., Obuchowski, N.A., and McClish, D.K. 2002. Statistical Methods in Diagnostic Medicine. New York: Wiley.
396. Zou, K.H. and Hall, W.J. 2000. Two transformation models for estimating an ROC curve derived from continuous data. Journal of Applied Statistics 27: 621-631.
397. Zou, K.H., Hall, W.J., and Shapiro, D.E. 1997. Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests. Statistics in Medicine 16: 2143-2156.
398. Adler, W. and Lausen, B. 2009. Bootstrap estimated true and false positive rates and ROC curve. Computational Statistics and Data Analysis 53: 718-729.
399. Asiago, V.M., Alvarado, L.Z., Shanaiah, N. et al. 2010. Early detection of recurrent breast cancer using metabolite profiling. Cancer Research 70: 8309-8318.
400. Ayers, K.L. and Cordell, H.J. 2010. SNP Selection in genome-wide and candidate gene studies via penalized logistic regression. Genetic Epidemiology 34: 879-891.
401. Benjamini, Y. and Hochberg, Y. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B 57: 289-300.
402. Bishop, C.M. 2006. Pattern Recognition and Machine Learning. New York: Springer.
403. Bradley, A.P. 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition 30: 1145-1159.
404. Chen, J.J., Wang, S.J., Tsai, C.A. et al. 2007. Selection of differentially expressed genes in microarray data analysis. Pharmacogenomics Journal. 7: 212-220.
405. Claus, E.B. 2001. Risk models used to counsel women for breast and ovarian cancer: a guide for clinicians. Familial Cancer 1: 197-206.
406. Dorfman, D.D., Berbaum. K.S., and Lenth, RV. 1995. Multireader, multicase receiver operating characteristic methodology: a bootstrap analysis. Academic Radiology 2: 626-633.
407. Efron, B. and Tibshirani, R. 1993. Introduction to the Bootstrap. New York: Chapman & Hall.
408. Efron, B. and Tibshirani, R. 1997. Improvements on cross-validation: the .632+ bootstrap method. Journal of the American Statistical Association 92: 548-560.
409. Fox, C.S., Massaro, J.M, Schlett, C.L. et al. 2010. Periaortic fat deposition is associated with peripheral arterial disease: the Framingham Heart Study. Circulation: Cardiovascular Imaging 3: 515-519.
410. Gerds, T.A. and Cai, T.X. 2008. The performance of risk prediction models. Biometrical Journal 50: 457-479.
411. Gönen, M. 2007. Analyzing Receiver Operating Characteristic Curves with SAS®. Cary, NC: SAS Institute.
412. Hastie, T., Tibshirani, R., and Friedman, J.H.2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. New York: Springer.
413. Hosmer, D.W, and Lemeshow, S. 2000. Applied Logistic Regression. 2nd ed. New York: Wiley.
414. Knaus ,W.A., Draper, E.A., Wagner, D.P. et al. 1985. APACHE II: a severity of disease classification system. Critical Care Medicine 13: 818-829.
415. Lasko, T.A., Bhagwat, J.G., Zou, K.H. et al. 2005. The use of receiver operating characteristic curves in biomedical informatics. Journal of Biomedical Informatics 38: 404-415.
416. Liao, J.G. and Chin, K.V. 2007. Logistic regression for disease classification using microarray data: model selection in a large p and small n case. Bioinformatics 23: 1945-1951.
417. Martin, A.P., Bartels, M., Hauss, J. et al. 2007. Overview of MELD score and UNOS adult liver allocation system. Transplantation Proceedings 39: 3169-3174.
418. Matheny, M.E., Arora, N., Ohno-Machado, L. et al. 2007. Rare adverse event monitoring of medical devices with the use of an automated surveillance tool. AMIA Annual Symposium Proceedings, pp. 518-522.
419. Matheny, M.E., Ohno-Machado, L., and Resnic, F.S. 2005. Discrimination and calibration of mortality risk prediction models in interventional cardiology. Journal of Biomedical Informatics 38: 367-375.
420. Matheny, M.E., Resnic, F.S., Arora, N. et al. 2007. Effects of SVM parameter optimization on discrimination and calibration for post-procedural PCI mortality. Journal of Biomedical Informatics 40: 688-697.
421. Nimgaonkar, A., Karnad, D.R., Sudarshan, S. et al. 2004. Prediction of mortality in an Indian intensive care unit: comparison between APACHE II and artificial neural networks. Intensive Care Medicine 30: 248-253.
422. Ohno-Machado, L. and Rowland, T. 1999. Neural network applications in physical medicine and rehabilitation. American Journal of Physical Medicine and Rehabilitation 78: 392-398.
423. Ohno-Machado, L. 1997. A comparison of Cox proportional hazards and artificial neural network models for medical prognosis. Computers in Biology and Medicine 27: 55-65.
424. Pawitan, Y., Michiels, S., Koscielny, S. et al. 2005. False discovery rate, sensitivity and sample size for microarray studies. Bioinformatics 21: 3017-3024.
425. Racowsky, C., Ohno-Machado, L., Kim, J. et al. 2009. Is there an advantage in scoring early embryos on more than one day? Human Reproduction 24: 2104-2113.
426. Resnic, F.S., Ohno-Machado, L., Selwyn, A. et al. 2001. Simplified risk score models accurately predict the risk of major in-hospital complications following percutaneous coronary intervention. American Journal of Cardiology 88: 5-9.
427. Rowland, T., Ohno-Machado, L., and Ohrn, A. 1998. Comparison of multiple prediction models for ambulation following spinal cord injury. Proceedings of AMIA Symposium, pp. 528-532.
428. Smola, A.J. 2000. Advances in Large Margin Classifiers. Cambridge: MIT Press.
429. Steyerberg, E.W., Borsboom, G.J.J.M., van Houwelingen, H.C. et al. 2004. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Statistics in Medicine 23: 2567-2586.
430. Storey, J.D. 2003. The positive false discovery rate: a Bayesian interpretation and the q-value. Annals of Statistics 31: 2013-2035.
431. Talos, I.F., Zou, K.H., Ohno-Machado, L. et al. 2006. Supratentorial low-grade glioma resectability: statistical predictive analysis based on anatomic MR features and tumor characteristics. Radiology 239: 506-513.
432. Tsai, C.A. and Chen, J.J. 2004. Significance analysis of ROC indices for comparing diagnostic markers: applications to gene microarray data. Journal of Biopharmaceutical Statistics 14: 985-1003.
433. Wang, S.J., Kuperman, G.J., Ohno-Machado, L. et al. 2000. Using electronic data to predict the probability of true bacteremia from positive blood cultures. Proceedings of AMIA Symposium, pp. 893-897.
434. Zou, K.H., Resnic, F.S, Talos, I.F. et al. 2005. A global goodness-of-fit test for receiver operating characteristic curve analysis via the bootstrap method. Journal of Biomedical Informatics 38: 395-403.