ROC curves for continuous data

ROC Curves for Continuous Data

Volumn , Issue , 2009, Pages 1-241

  Krzanowski, Wojtek J ^a,b   Hand, David J ^c  

^a UNIVERSITY OF LEEDS   (United Kingdom)

^b IMPERIAL COLLEGE LONDON   (United Kingdom)

^c UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85122646791     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Book

References (245)

1. Adams, N.M and Hand, D.J. (1999). Comparing classifiers when the misallocation costs are uncertain. Pattern Recognition, 32, 1139-1147.
2. Algarabel, S. and Pitarque, A. (2007). ROC parameters in item and context recognition. Psichothema, 19, 163-170.
3. Allison, P.D. (2001). Missing Data. Sage Publications, Thousand Oaks, CA.
4. Alonzo, T.A. and Pepe, M.S. (2002). Distribution-free ROC analysis using binary regression techniques. Biostatistics, 3, 421-432.
5. Alonzo, T.A. and Pepe, M.S. (2005). Assessing accuracy of a continuous screening test in the presence of verification bias. Applied Statistics, 54, 173-190.
6. Alonzo, T.A., Pepe, M.S. and Lumley, T. (2003). Estimating disease prevalence in two-phase studies. Biostatistics, 4, 313-326.
7. Augustin, N.H., Mugglestone, M.A. and Buckland, S.T. (1996). An autologistic model for the spatial distribution of wildlife. Journal of Applied Ecology, 33, 339-347.
8. Ayalew, L. and Yamagishi, H. (2005). The application of GIS-based logistic regression for landslide susceptibility mapping in the Kakuda- Yahiko mountains, Central Japan. Geomorphology, 65, 15-31.
9. Bamber, D. (1975). The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology, 12, 387-415.
10. Begg, C.B. (1989). Experimental design of medical imaging trials: issues and options. Investigative Radiology, 24, 934-936.
11. Begg, C.B. and Greenes, R.A. (1983). Assessment of diagnostic tests when disease is subject to selection bias. Biometrics, 39, 207-216.
12. Beiden, S.V., Campbell, G., Meier, K.L. and Wagner, R.F. (2000). On the problem of ROC analysis without truth. In: Proceedings of the Society of Photo-Optical Instrumentation Engineers (SPIE), eds. M.D. Salman, P.Morley and R. Ruch-Gallie, vol 3981, pp. 126-134. The International Society for Optical Engineering, Bellingham, WA.
13. Biggerstaff, B.J. and Tweedie, R.L. (1997). Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis. Statistics in Medicine, 16, 753-768.
14. Bloch, D.A. (1997). Comparing two diagnostic tests against the same "gold standard" in the same sample. Biometrics, 53, 73-85.
15. Blochlinger, A. and Leippold, M. (2006). Economic benefit of powerful credit scoring. Journal of Banking and Finance, 30, 851-873.
16. Bradley, A.P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30, 1145-1159.
17. Branscum, A.J., Gardner, I.A. and Johnson, W.O. (2005). Estimation of diagnostic-test sensitivity and specificity through Bayesian modeling. Preventive Veterinary Medicine, 68, 145-163.
18. Brenning, A. (2005). Spatial prediction models for landslide hazards: review, comparison and evaluation. Natural Hazards and Earth System Sciences, 5, 853-862.
19. Briggs W.M. and Zaretzki R. (2008). The skill plot: a graphical technique for evaluating continuous diagnostic tests. Biometrics, 63, 250261.
20. Brooks, S.P. (1998). Markov chain Monte Carlo method and its application. The Statistician, 47, 69-100.
21. Brumback, L.C., Pepe, M.S. and Alonzo, T.A. (2006). Using the ROC curve for gauging treatment effect in clinical trials. Statistics in Medicine, 25, 575-590.
22. Buizza, R., Hollingsworth, A., Lalaurette, F. and Ghelli, A. (1999). Probabilistic predictions of precipitation using the ECMWF ensemble prediction system. Weather and Forecasting, 14, 168-189.
23. Cai, T. and Moskowitz, C.S. (2004). Semi-parametric estimation of the binormal ROC curve for a continuous diagnostic test. Biostatistics, 5, 573-586.
24. Cai, T. and Pepe, M.S. (2003). Semi-parametric ROC analysis to evaluate biomarkers for disease. Journal of the American Statistical Association, 97, 1099-1107.
25. Cai, T. and Zheng, Y. (2007). Model checking for ROC regression analysis. Biometrics, 63, 152-163.
26. Campbell, G. and Ratnaparkhi, M.V. (1993). An application of Lomax distributions in receiver operating characteristic (ROC) curve analysis. Communications in Statistics, 22, 1681-1697.
27. Chi, Y.-Y. and Zhou, X.-H. (2008). Receiver operating characteristic surfaces in the presence of verification bias. Applied Statistics, 57, 1-23.
28. Choi, Y.-K., Johnson, W.O., Collins, M.T. and Gardner, I.A. (2006). Bayesian inferences for Receiver Operating Characteristic curves in the absence of a gold standard. Journal of Agricultural, Biological and Environmental Statistics, 11, 210-229.
29. Cochran, W.G. (1964). Comparison of two methods of handling covariates in discriminant analysis. Annals of the Institute of Statistical Mathematics, 16, 43-53.
30. Cochran, W.G. (1977). Sampling Techniques (3rd Edition). Wiley, New York.
31. Cochran, W.G. and Bliss, C.I. (1948). Discriminant functions with covariance. Annals of Mathematical Statistics, 19, 151-176.
32. Coffin, M. and Sukhatme, S. (1995). A parametric approach to measurement errors in receiver operating characteristic studies. Lifetime Data Models in Reliability and Survival Analysis, 71-75. Kluwer, Boston.
33. Coffin, M. and Sukhatme, S. (1997). Receiver operating characteristic studies and measurement errors. Biometrics, 53, 823-837.
34. Collinson, P. (1998). Of bombers, radiologists, and cardiologists: time to ROC. Heart, 80, 215-217.
35. Cortes, C. and Mohri, M. (2005). Confidence intervals for the area under the ROC curve. Advances in Neural Information Processing Systems vol 17 (NIPS 2004). MIT Press, Cambridge, MA.
36. Cox, D.R. (1972). Regression models and life-tables (with discussion). Journal of the Royal Statistical Society, Series B, 74, 187-220.
37. Cox, D.R. (1992). Planning of Experiments. Wiley, New York.
38. Craig, L.A., Beech, A.R. and Browne, K.D. (2006). Evaluating the predictive accuracy of sex offender risk assessment measures on UK samples: a cross-validation of the risk matrix 2000 scales. Sexual Offender Treatment, 1(1). (Electronic Journal, http://www.sexual- offender-treatment.org/19.98)
39. Davis J. and Goadrich M (2006). The relationship between precision- recall and ROC curves. In: Proceedings of the 23rd International Conference on Machine Learning (ICML), ed. W.W.Cohen and A.Moore, 233-240.
40. Davison, A.C. and Hinkley, D.V. (1997). Bootstrap Methods and their Applications. Cambridge University Press, Cambridge.
41. DeLong, E.R., De Long, D.M. and Clarke-Pearson, D.L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 44, 837-845.
42. Dendurki, N. and Joseph, L. (2001). Bayesian approaches to modelling the conditional dependence between multiple diagnosis tests. Biometrics, 57, 158-167.
43. Denison, D.G.T, Holmes, C.C., Malik, B.K. and Smith, A.F.M. (2002). Bayesian Methods for Nonlinear Classification and Regression. Wiley, Chichester.
44. Der Simonian, R. and Laird, N. (1986). Meta-analysis in clinical trials. Control Clinical Trials, 7, 177-188.
45. Dodd, L.E. and Pepe, M.S. (2003a). Semiparametric regression for the area under the receiver operating characteristic curve. Journal of the American Statistical Association, 98, 409-417.
46. Dodd, L.E. and Pepe, M.S. (2003b). Partial AUC estimation and regression. Biometrics, 59, 614-623.
47. Dodge, H. and Romig, H. (1929). A method of sampling inspection. Bell Systems Technical Journal, 8, 613-631.
48. Dorfman, D.D. and Alf, E. (1969). Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals-rating method data. Journal of Mathematical Psychology, 6, 487-496.
49. Dorfman, D.D., Berbaum, K.S. and Metz, C.E. (1992). Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Investigative Radiology, 27, 307-335.
50. Dorfman, D.D., Berbaum, K.S., Metz, C.E., Lenth, R.V., Hanley, J.A. and Dagga, H.A. (1997). Proper receiver operating characteristic analysis: the bigamma model. Academic Radiology, 4, 138-149.
51. Dreiseitl, S., Ohno-Machado, L., and Binder, M. (2000). Comparing three-class diagnostic tests by three-way ROC analysis. Medical Decision Making, 20, 323-331.
52. Drummond, C. and Holte, R.C. (2000). Explicitly representing expected cost: an alternative to ROC representation. In: Proceedings of the 6th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 198-207.
53. Dudoit, S. and Fridlyand, J. (2003). Classification in microarray experiments. In: Statistical Analysis of Gene Expression Microarray Data, ed. T. Speed, pp. 93-158. Chapman & Hall, London.
54. Edwards, D.C., Metz, C.E., and Kupinski, M.A. (2004). Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Transations on Medical Imaging, 23, 891-895.
55. Edwards, D.C., Metz, C.E., and Nishikawa, R.M. (2005). The hypervolume under the ROC hypersurface of "near-guessing" and "nearperfect" observers in N-class classification tasks. IEEE Transactions on Medical Imaging, 24, 293-299.
56. Efron, B. and Tibshirani, R.J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC, New York.
57. Egan, J.P. (1975). Signal Detection Theory and ROC Analysis. Academic Press, New York.
58. Enpe, C., Georgiadis, M.P. and Johnson, W.O. (2000). Estimation of sensitivity and specificity of diagnostic tests and disease prevalence when the true disease state is unknown. Preventive Veterinary Medicine, 45, 61-81.
59. Erkanli, A., Sung, M., Costello, E.J. and Angold, A. (2006). Bayesian semi-parametric ROC analysis. Statistics in Medicine, 25, 3905-3928.
60. Escobar, M.D. and West, M. (1995). Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90, 577-588.
61. Everson, R.M. and Fieldsend, J.E. (2006). Multi-class ROC analysis from a multi-objective optimisation perspective. Pattern Recognition Letters, 27, 918-927.
62. Faraggi, D. (2000). The effect of random measurement error on receiver operating characteristic (ROC) curves. Statistics in Medicine, 19, 61-70.
63. Faraggi, D. (2003). Adjusting receiver operating characteristic curves and related indices for covariates. Journal of the Royal Statistical Society, Series D, 52, 179-192.
64. Faraggi, D. and Reiser, B. (2002). Estimation of the area under the ROC curve. Statistics in Medicine, 21, 3093-3106.
65. Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27, 861-874.
66. Ferguson, T.S. (1983). Bayesian density estimation by mixtures of normal distributions. In: Recent Advances in Statistics, eds. H. Rizvi and J. Rustagi, pp. 287-302. Academic Press, New York.
67. Ferri C., Hernandez-Orallo J., and Salido M.A. (2004). Volume under the ROC surrface for multi-class problems. Lecture Notes in Computer Science, 2837, 108-120.
68. Fisher, R.A. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7, 179-184.
69. Fleiss, J.L. (1981). Statistical Methods for Rates and Proportions. New York: Wiley, New York.
70. Fluss, R., Faraggi, D. and Reiser, B. (2005). Estimation of the Youden Index and its associated cutoff point. Biometrical Journal, 47, 458-472.
71. Gao, S., Hui, S.L., Hall, K.S. and Hendrie, H.C. (2000). Estimating disease prevalence from two-phase surveys with non-response at the second phase. Statistics in Medicine, 19, 2101-2114.
72. Garrett, E.S., Eaton, E.E. and Zeger, S. (2002). Methods for evaluating the performance of diagnostic tests in the absence of a gold standard: a latent class model approach. Statistics in Medicine, 21, 1289-1307.
73. Gini, C. (1912). Variabilit'a e mutabilit'a. Reprinted in Memorie di Metodologica Statistica, (Ed. Pizetti E. and Salvemini, T.), 1955, Li- breria Eredi Virgilio Veschi, Rome.
74. Girling, A.J. (2000). Rank statistics expressible as integrals under P-P- plots and receiver operating characteristic curves. Journal of the Royal Statistical Society, Series B, 62, 367-382.
75. Goddard, M.J. and Hinberg, I. (1990). Receiver operating characteristic (ROC) curves and non-normal data: an empirical study. Statistics in Medicine, 9, 325-337.
76. Gold, M.R., Siegel, J.E., Russel, L.B. and Weinstein, M.C. (1977). Cost Effectiveness in Health and Medicine. Wiley, New York.
77. Goodenough, D.J., Rossman, K. and Lusted, L.B. (1974). Radiographic applications of receiver operating characteristic (ROC) analysis. Radiology, 110, 89-95.
78. Goovaerts, P., Jacquez, G.M. and Marcus, A. (2005). Geostatistical and local cluster analysis of high resolution hyperspectral imagery for the detection of anomalies. Remote Sensing of Environment, 95, 351367.
79. Gorsevski, P.V., Gessler, P. and Foltz, R.B. (2000). Spatial prediction of landslide hazard using logistic regression and GIS. In: 4th International Conference on Integrating GIS and Environmental Modeling (GIS/EM4), Problems, Prospects and Research Needs, 110. Banff, Alberta.
80. Gotway, C.A. and Stroup, W.W. (1997). A generalized linear model approach to spatial data analysis and prediction. Journal of Agricultural, Biological, and Environmental Statistics, 2, 157-178.
81. Green, D.M. and Swets, J.A. (1966). Signal Detection Theory and Psychophysics, Wiley, New York.
82. Greenhouse, S. and Mantel, N. (1950). The evaluation of diagnostic tests. Biometrics, 6, 399-412.
83. Guido, R., Schiffman, M., Solomon, D. and Burke, L. (2003). Postcolposcopy management strategies for women referred with low-grade squamous intraepithelial lesions or human papillomavirus DNA-positive atypical squamous cells of undetermined significance: a two-year prospective study. American Journal of Obstetrics and Gynecology, 188, 14011405.
84. Gur, D., King, J.L., Rockette, H.E., Britton, C.A., Thaete, F.L. and Hoy, R.J. (1990). Practical issues of experimental ROC analysis: selection of controls. Investigative Radiology, 25, 583-586.
85. Guttman, I., Johnson, R.A., Bhattacharyya, G.K. and Reiser, B. (1988). Confidence limits for stress-strength models with explanatory variables. Technometrics, 30, 161-168.
86. Hajian-Tilaki, K.O., Hanley, J.A., Joseph, L. and Collet, J. (1997). A comparison of parametric and nonparametric approaches to ROC analysis of quantitative diagnostic tests. Medical Decision Making, 17, 94-102.
87. Han, A.K. (1987). Non-parametric analysis of a generalized regression model. Journal of Econometrics, 35, 303-316.
88. Hand, D.J. (1986). Recent advances in error rate estimation, Pattern Recognition Letters, 4, 335-46
89. Hand, D.J. (1997). Construction and Assessment of Classification Rules. Wiley, Chichester.
90. Hand, D.J. (2001). Measuring diagnostic accuracy of statistical prediction rules. Statistica Neerlandica, 55, 3-16.
91. Hand, D.J. (2005). Good practice in retail credit scorecard assessment. Journal of the Operational Research Society, 56, 1109-1117.
92. Hand D.J. and Till R.J. (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning, 45, 171-186.
93. Hand, D.J., Oliver, J.J. and Lunn, A.D. (1998). Discriminant analysis when the classes arise from a continuum. Pattern Recognition, 31, 641-650.
94. Hanley, J.A. (1989) Receiver operating characteristic (ROC) methodology: the state of the art. Critical Reviews in Diagnostic Imaging, 29, 307-335.
95. Hanley, J.A. (1996). The use of the 'binormal' model for parametric ROC analysis of quantitative diagnostic tests. Statistics in Medicine, 15, 1575-1585.
96. Hanley, J.A. and Hajian-Tilaki, K.O. (1997). Sampling variability of nonparametric estimates of the areas under receiver operating characteristic curves: an update. Academic Radiology, 4, 49-58.
97. Hanley, J.A. and McNeil, B.J. (1982). The meaning and use of the area under an ROC curve. Radiology, 143, 29-36.
98. Hanley, J.A. and McNeil, B. (1983). A method of comparing the area under two ROC curves derived from the same cases. Radiology, 148, 839-43.
99. Hardy, R.J. and Thompson, S.G. (1996). A likelihood approach to meta-analysis with random effects. Statistics in Medicine, 15, 619-629.
100. Hastie, T., Tibshirani, R. and Friedman, J. (2001). The Elements of Statistical Learning. Springer, New York.
101. Hayes, S.C. (2002). Early intervention or early incarceration? Using a screening test for intellectual disability in the criminal justice system. Journal of Applied Research in Intellectual Disabilities, 15, 120-128.
102. He X., Metz C.E., Tsui B.M.W., Links J.M., and Frey E.C. (2006). Three-class ROC analysis-a decision theoretic approach under the ideal observer framework. IEEE Transactions on Medical Imaging, 25, 571-581.
103. Heagerty, P.J. and Pepe, M.S. (1999). Semiparametric estimation of regression quantiles with application to standardizing weight for height and age in US children. Applied Statistics, 48, 533-551.
104. Hellmich, M., Abrams, K.R., Jones, D.R. and Lambert, P.C. (1998). A Bayesian approach to a general regression model for ROC curves. Medical Decision Making, 18, 436-443.
105. Hellmich, M., Abrams, K.R. and Sutton, A.J. (1999). Bayesian approaches to meta-analysis of ROC curves. Medical Decision Making, 19, 252-264.
106. Henkelman, R.M., Kay, I. and Bronskill, M.J. (1990). Receiver Operating Characteristic (ROC) analysis without truth. Medical Decision Making, 10, 24-29.
107. Holliday, J.R., Rundle, J.B., Turcotte, D.L., Klein, W., Tiampo, K.F. and Donellan, A. (2006). Space-time clustering and correlations of major earthquakes. Physical Review Letters, 97, 238501.
108. Hsieh, F. and Turnbull, B.W. (1996). Nonparametric and semiparametric estimation of the receiver characteristic curve. Annals of Statistics, 24, 25-40.
109. Irwig, L., Macaskill, P., Glasziou, P. and Fahey, M. (1995). Metaanalytic methods for diagnostic test accuracy. Journal of Clinical Epidemiology, 48, 119-130.
110. Ishwaran, H. and Gatsonis, C.A. (2000). A general class of hierarchical ordinal regression models with applications to correlated ROC analysis. The Canadian Journal of Statistics, 28, 731-750.
111. Ishwaran, H. and James, L.F. (2002). Approximate Dirichlet process computing in finite normal mixtures: smoothing and prior information. Journal of Computational and Graphical Statistics, 11, 508-532.
112. Jamain, A. (2004). Meta-Analysis of Classification Methods. Unpublished PhD thesis, Department of Mathematics, Imperial College, London.
113. Janes, H. and Pepe, M.S. (2006). Adjusting for covariate effects on classification accuracy using the covariate-adjusted ROC curve. Technical Report 283, UW Biostatistics Working Paper Series. Available at http://www.bepress.com/uwbiostat / paper283
114. Janes, H. and Pepe, M.S. (2007). Adjusting for covariates in studies of diagnostic, screening, or prognostic markers: an old concept in a new setting. Technical Report 310, UW Biostatistics Working Paper Series. Available at http://www.bepress.com/uwbiostat/paper310
115. Janes, H. and Pepe, M.S. (2008). Matching in studies of classification accuracy: implications for analysis, efficiency, and assessment of incremental value. Biometrics, 64, 1-9.
116. Janes, H., Longton, G.M. and Pepe, M.S. (2008). Accommodating covariates in ROC analysis. Technical Report 322, UW Biostatistics Working Paper Series. Available at http://www.bepress.com/uwbiostat / paper322
117. Kessler, R.C. (2002). The categorical versus dimensional assessment controversy in the sociology of mental illness. Journal of Health and Social Behavior, 43, 171-188.
118. Krzanowski, W.J. and Marriott, F.H.C. (1995). Multivariate Analysis Part 2: Classification, Covariance Structures and Repeated Measurements. Arnold, London.
119. Kucharski, L.T., Toomey, J.P. and Fila, K. (2007). Detection of malingering of psychiatric disorder with the Personality Assessment Inventory: an investigation of criminal defendants. Journal of Personality Assessment, 88, 25-32.
120. Kuncheva, L.I. (2004). Combining Pattern Classifiers, Methods and Algorithms, Wiley, New Jersey.
121. Lachenbruch, P.A. (1977). Covariance adjusted discriminant functions. Annals of the Institute of Statistical Mathematics, 29, 247-257.
122. Landgrebe, T.C.W. and Duin, R.P.W. (2007). Approximating the multiclass ROC by pairwise analysis. Pattern Recognition Letters, 28, 1747-1758.
123. Landgrebe, T.C.W. and Duin, R.P.W. (2008). Multiclass ROC approximation by decomposition via confusion matrix perturbation analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30, 810-822.
124. Le, C.T. (2006). A solution for the most basic optimization problem associated with an ROC curve. Statistical Methods for Medical Research, 15, 571-584.
125. Lee, P. (2004). Bayesian Statistics: An Introduction (3rd Edition). Arnold, London.
126. Lee, W.C. (1999). Probabilistic analysis of global performances of diagnostic tests: interpreting the Lorenz curve-based summary measures. Statistics in Medicine, 18, 455-471.
127. Lee, W.C. and Hsiao, C.K. (1996). Alternative summary indices for the receiver operating characteristic curve. Epidemiology, 7, 605-611.
128. Lin, J.-H. and Haug, P.J. (2008). Exploiting missing clinical data in Bayesian network modeling for predicting medical problems. Journal of Biomedical Informatics, 41, 1-14.
129. Little, R.J.A. and Rubin, D.B. (2002). Statistical Analysis with Missing Data (2nd Edition). Wiley, New York.
130. Liu, A., Schisterman, E.F. and Wu, C. (2005). Nonparametric estimation and hypothesis testing on the partial area under receiver characteristic curves. Communications in Statistics-Theory and Methods, 34, 2077-2088.
131. Lloyd, C.J. (1998). The use of smoothed ROC curves to summarise and compare diagnostic systems. Journal of the American Statistical Association, 93, 1356-1364.
132. Lloyd, C.D. (2002). Semiparametric estimation of ROC curves based on binomial regression modelling. Australian and New Zealand Journal of Statistics, 44, 75-86.
133. Lorenz, M.O. (1905). Methods of measuring the concentration of wealth. American Statistical Association, 9, 209-219.
134. Lusted, L.B. (1968). Introduction to Medical Decision Making. Thomas, Springfield.
135. Ma, G. and Hall, W.J. (1993). Confidence bands for receiver operating characteristic curves. Medical Decision Making, 13, 191-197.
136. Ma, S. and Huang, J. (2005). Regularized ROC method for disease classification and biomarker selection with microarray data. Bioinformatics, 21, 4356-4362.
137. Ma, S., Song, X. and Huang, J. (2006). Regularized binormal ROC method in disease classification with microarray data. BMC Bioinformatics, 7, 253.
138. Mamitsuka, H. (2006). Selecting features in microarray classification using ROC curves. Pattern Recognition, 39, 2393-2404.
139. Marascuilo, L.A. and McSweeney, M. (1977). Nonparametric and Distribution-free Methods for the Social Sciences. Brooks-Cole Publishing Company, Monterey, California.
140. Mason, S.J. and Graham, N.E. (2002). Areas beneath the relative operating characterists (ROC) and relative operating levels (ROL) curves: statistical significance and interpretation. Quarterly Journal of the Meteorological Society, 128, 2145-2166.
141. McClish, D.K. (1989). Analyzing a portion of the ROC curve. Medical Decision Making, 9, 190-195.
142. McClish, D.K. (1990). Determining the range of false-positives for which ROC curves differ. Medical Decision Making, 10, 283-287.
143. McCullagh, P. and Nelder, J.A. (1989). Generalized Linear Models (2nd Edition). Chapman & Hall, New York.
144. McIntosh, M.W. and Pepe, M.S. (2002). Combining several screening tests: optimality of the risk score. Biometrics, 58, 657-664.
145. McLachlan, G.J. (1992). Discriminant Analysis and Pattern Recognition. Wiley, New York.
146. Mee, R.W. (1990). Confidence intervals for probabilities and tolerance regions based on a generalization of the Mann-Whitney statistic. Journal of the American Statisticsl Association, 85, 793-800.
147. Merz, C.J. and Murphy, P.M. (1996). UCI depository of machine learning database. http://www.ic.uci.edu/ mlearn/MLRepository.html
148. Metz, C.E. (1978) Basic principles of ROC analysis. Seminars in Nuclear Medicine, 8, 283-98.
149. Metz, C.E. (1989). Some practical issues of experimental design and data analysis in radiological ROC studies. Investigative Radiology, 24, 234-245.
150. Metz, C.E. (2008). ROC analysis in medical imaging: a tutorial review of the literature. Radiological Physics & Technology, 1, 2-12.
151. Metz, C. and Kronman, H. (1980). Statistical significance tests for binormal ROC curves. Journal of Mathematical Psychology, 22, 218-243.
152. Metz, C., Herman, B.A., and Roe, C.A. (1998). Statistical comparison of two ROC estimates obtained from partially paired classes. Medical Decision Making, 18, 110-121.
153. Metz, C.E., Herman, B.A. and Shen, J. (1998). Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously distributed data. Statistics in Medicine, 17, 1033-1053.
154. Metz, C., Wang, P., and Kronman, H. (1984). A new approach for testing the significant differences between ROC curves measured from correlated data. In F. Deconinck (ed.) Information Processing in Medical Imaging. Nijihoff, The Hague, The Netherlands.
155. Mossman, D. (1999). Three-way ROCs. Medical Decision Making, 19, 78-89.
156. Müller, P., Erkanli, A. and West, M. (1996). Bayesian curve fitting using multivariate normal mixtures. Biometrika, 83, 67-79.
157. Nakas, C.T. and Yiannoutsos, C.T. (2004). Ordered multiple-class ROC analysis with continuous measurements. Statistics in Medicine, 23, 3437-3449.
158. Neyman, J. and Pearson, E.S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society, London, A231, 289-337.
159. Nielsen, S.S., Gronbak, C., Agger, J.F. andHoue, H. (2002). Maximum- likelihood estimation of sensitivity and specificity of ELISAs and faecal culture for diagnosis of paratuberculosis. Preventitive Veterinary Medicine, 53, 191-204.
160. Nock, R. (2003). Complexity in the case against accuracy estimation. Theoretical Computer Science, 301, 143-165.
161. Normand, S-L.T. (1999). Tutorial in biostatistics. Meta-analysis: formulating, evaluating, combining and reporting. Statistics in Medicine, 18, 321-359.
162. Obuchowski, N.A. (1998). Sample size calculations in studies of test accuracy. Statistical Methods in Medical Research, 7, 371-392.
163. Obuchowski, N.A. (2003). Receiver operating characteristic curves and their use in radiology. Radiology, 229, 3-8.
164. Obuchowski, N.A. (2004). How many observers are needed in clinical studies of medical imaging? American Journal of Radiology, 182, 867-869.
165. Obuchowski, N.A. (2005). ROC analysis. American Journal of Radiology, 184, 364-372.
166. Obuchowski, N.A. and McClish, D.K. (1997). Sample size determination for diagnostic accuracy studies involving binormal ROC curve indices. Statistics in Medicine, 16, 1529-1542.
167. Obuchowski, N.A. and Rockette, H.E. (1995). Hypothesis testing of the diagnostic accuracy for multiple diagnostic tests: an ANOVA approach with dependent observations. Communications in Statistics, Simulation & Computation, 24, 285-308.
168. Obuchowski, N.A., Beiden, S.V., Berbaum, K.S., Hillis, S.L., Ishwaran, H., Song, H.H. and Wagner, R.F. (2004). Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Academic Radiology, 11, 980-995.
169. Obuchowski, N.A., Lieber, M.L. and Wians, F.H. (2004). ROC curves in clinical chemistry: uses, misuses, and possible solutions. Clinical Chemistry, 50, 1118-1125.
170. Ogilvie, J.C. and Creelman, C.D. (1968). Maximum likelihood estimation of ROC curve parameters. Journal of Mathematical Psychology, 5, 377-391.
171. Owen, D.B., Craswell, K.J. and Hanson, D.L. (1964). Non-parametric upper confidence bounds for P(Y X) and confidence limits for P(Y X) when X and Y are normal. Journal of the American Statis- ticsl Association, 59, 906-924.
172. Peng, F.C. and Hall, W.J. (1996). Bayesian analysis of ROC curves using Markov-chain Monte Carlo methods. Medical Decision Making, 16, 404-411.
173. Peng, L. and Zhou, X.-H. (2004). Local linear smoothing of receiver operating characteristic (ROC) curves. Journal of Statistical Planning and Inference, 118, 129-143.
174. Pepe, M.S. (1997). A regression modelling framework for receiver operating characteristic curves in medical diagnostic testing. Biometrika, 84, 595-608.
175. Pepe, M.S. (1998). Three approaches to regression analysis of receiver operating characteristic curves for continuous test results. Biometrics, 54, 124-135.
176. Pepe, M.S. (2000). An interpretation for the ROC curve and inference using GLM procedures. Biometrics, 56, 352-359.
177. Pepe, M.S. (2003). The Statistical Evaluation of Medical Tests for Classification and Prediction. University Press, Oxford.
178. Pepe, M.S. and Cai, T. (2004). The analysis of placement values for evaluating discriminatory measures. Biometrics, 60, 528-535.
179. Pepe, M.S., Cai, T. and Longton, G. (2006). Combining predictors for classification using the area under the receiver operating characteristic curve. Biometrics, 62, 221-229.
180. Perkins, N.J. and Schisterman, E.F. (2006). The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. American Journal of Epidemiology, 163, 670-675.
181. Peterson, W.W., Birdsall, T.G. and Fox, W.C. (1954). The theory of signal detectability. Transactions of the IRE Professional Group on Information Theory, PGIT, 2-4, 171-212.
182. Pfeiffer, R.M. and Castle, P.E. (2005). With or without a gold standard. Epidemiology, 16, 595-597.
183. Provost F. and Fawcett T. (1997). Analysis and visualisation of classifier performance: comparison under imprecise class and cost distributions. In: Proceedings of the Third International Conference on Knowledge Discovery and Data Mining, AAAI Press, Menlo Park, CA, 43-48.
184. Provost F. and Fawcett T. (1998). Robust classification systems for imprecise environments. In: Proceedings of the 15th National Conference on Artificial Intelligence, AAAI Press, Madison, WI, 706-707.
185. Provost, F., Fawcett, T. and Kohavi, R. (1998). The case against accuracy estimation for comparing induction algorithms. In: Proceedings of the 15th International Conference on Machine Learning, pp 445-453, July 24-27 1998.
186. Qin, G. and Zhou, X.-H. (2006). Empirical likelihood inference for the area under the ROC curve. Biometrics, 62, 613-622.
187. Qin, J. and Zhang, B. (2003). Using logistic regression procedures for estimating receiver operating characteristic curves. Biometrika, 90, 585-596.
188. Ransohoff, D.F. and Feinstein, A.R. (1978). Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. New England Journal of Medicine, 299, 926-930.
189. Rao, C.R. (1949). On some problems arising out of discrimination with multiple characters. Sankhya, 9, 343-366.
190. Reiser, B. (2000). Measuring the effectiveness of diagnostic markers in the presence of measurement error through the use of ROC curves. Statistics in Medicine, 19, 2115-2129.
191. Reiser, B. and Faraggi, D. (1994). Confidence bounds for Pr(a'X b'Y). Statistics, 25, 107-111.
192. Reiser, B. and Guttman, I. (1986). Statistical inference for P(Y X): the normal case. Technometrics, 28, 253-257.
193. Ren, H., Zhou, X.-H. and Liang, H. (2004). A flexible method for estimating the ROC curve. Journal of Applied Statistics, 31, 773-784.
194. Ripley, B.D. (1996) Pattern Recognition and Neural Networks. Cambridge University Press, Cambridge.
195. Rotnitzky, A., Faraggi, D. and Schisterman, E. (2006). Doubly robust estimation of the area under the receiver-operating characteristic curve in the presence of verification bias. Journal of the American Statistical Association, 101, 1276-1288.
196. Schafer, J.L. (1999). Multiple imputation: a primer. Statistical Methods in Medical Research, 8, 3-15.
197. Schiavo, R. and Hand, D.J. (2000). Ten more years of error rate research. International Statistical Review, 68, 295-310.
198. Schisterman, E.F., Faraggi, D., Reiser, B. and Trevisan, M. (2001). Statistical inference for the area under the receiver operating characteristic curve in the presence of random measurement error. American Journal of Epidemiology, 154, 174-179.
199. Schisterman, E.F., Faraggi, D. and Reiser, B. (2004). Adjusting the generalized ROC curve for covariates. Statistics in Medicine, 23, 33193331.
200. Scott M.J.J., Niranjan M. and Prager R.W. (1998). Parcel: feature subset selection in variable cost domains. Technical Report CUED/F- INFENG/TR.323, Cambridge University Engineering Department.
201. Sen, P.K. (1960). On some convergence properties of U-statistics. Calcutta Statistical Association Bulletin, 10, 1-18.
202. Shewhart, W. (1931). Economic Control of Quality of Manufactured Products. van Norstand, New York.
203. Shieh, G., Show-Li, J. and Randles, R.H. (2006). On power and sample size determination for the Wilcoxon-Mann-Whitney test. Nonparametric Statistics, 18, 33-43.
204. Silverman, B.W. (1986). Density Estimation in Statistics and Data Analysis. Chapman & Hall, London.
205. Simonoff, J.S., Hochberg, Y. and Reiser, B. (1986). Alternative estimation procedures for P(X Y) in categorized data. Biometrics, 42, 815-907.
206. Smith, P.J. and Thompson, T.J. (1996). Correcting for confounding in analyzing receiver operating characteristic curves. Biometrical Journal, 38, 857-863.
207. Smith, T.C., Spiegelhalter, D.J. and Thomas, A. (1995). Bayesian approaches to random effects meta-analysis: a comparative study. Statistics in Medicine, 14, 2685-2699.
208. Song, H.H. (1997). Analysis of correlated ROC areas in diagnostic testing. Biometrics, 53, 370-382.
209. Spiegelhalter, D.J., Thomas, A., Best, N.G. and Gilks, W.R. (1995). BUGS: Bayesian Inference Using Gibbs Sampling, Version 0.50. Cambridge: MRC Biostatistics Unit.
210. Sprent, P. (1989). Applied Nonparametric Statistical Methods. Chapman & Hall, London.
211. Spritzler, J., DeGruttola, V.G. and Pei, L. (2008). Two-sample tests of area-under-the-curve in the presence of missing data. The International Journal of Biostatistics, 4, 18 pp., The Berkeley Electronic Press.
212. Stepanova, M. and Thomas, L.C. (2002). Survival analysis methods for personal loans data. Operations Research, 50, 277-289.
213. Stephan, C., Wesseling, S., Schink, T. and Jung, K. (2003). Comparison of eight computer programs for receiver-operating characteristic analysis. Clinical Chemistry, 49, 433-439.
214. Stover, L., Gorga, M.P., Neely, S.T. and Montoya, D. (1996). Toward optimizing the clinical utility of distortion product otoacoustic emission measurements. Journal of the Acoustical Society of America, 100, 956-967.
215. Swets, J.A. and Pickett, R.M. (1982). Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. Academic Press, New York.
216. Tanner, M. and Wong, W. (1987). The calculation of posterior distributions by data augmentation. Journal of the American Statistical Association, 82, 528-550.
217. Thomas, L.C., Banasik, J. and Crook, N. (1999). Not if but when loans default. Journal of the Operational Research Society, 50, 11851190.
218. Thompson, M.L. and Zucchini, W. (1989). On the statistical analysis of ROC curves. Statistics in Medicine, 8, 1277-1290.
219. Thuiller, W., Vayreda, J., Pino, J., Sabate, S., Lavorel, S. and Gracia, C. (2003). Large-scale environmental correlates of forest tree distributions in Catalonia (NE Spain). Global Ecology & Biogeography, 12, 313-325.
220. Toh, K-A., Kim, J. and Lee, S. (2008). Maximizing area under ROC curve for biometric scores fusion. Pattern Recognition, 41, 3373-3392.
221. Toledano, A.Y. and Gatsonis, C. (1996). Ordinal regression methodology for ROC curves derived from correlated data. Statistics in Medicine, 15, 1807-1826.
222. Tosteson, A.N.A. and Begg, C.B. (1988). A general regression methodology for ROC curve estimation. Medical Decision Making, 8, 204-215.
223. Tosteson, D., Buonaccorsi, J.P., Demidenko, E. and Wells, W.A. (2005). Measurement error and confidence intervals for ROC curves. Biometrical Journal, 47, 409-416.
224. Van der Heiden, G.J.M.G., Donders, A.R.T., Stijnen, T. and Moons, K.G.M. (2006). Imputation of missing values is superior to complete case analysis and the missing-indicator method in multivariable diagnostic research: a clinical example. Journal of Clinical Epidemiology, 59, 1102-1109.
225. Venables, W.N. and Ripley, B.D. (2002). Modern Applied Statistics with S. 4th Edition. Springer, New York.
226. Venkatraman, E.S. (2000). A permutation test to compare receiver operating characteristic curves. Biometrics, 56, 1134-1138.
227. Venkatraman, E.S. and Begg, C.B. (1996). A distribution-free procedure for comparing Receiver Operating Characteristic curves from a paired experiment. Biometrika, 83, 835-848.
228. Verde, M.F. and Rotello, C.M. (2004). ROC curves show that the revelation effect is not a single phenomenon. Psychonomic Bulletin & Review, 11, 560-566.
229. Vida, S. (1993). A computer program for non-parametric receiver operating characteristic analysis. Computer Methods and Programs in Biomedicine, 40, 95-101.
230. Wang, C., Turnbull, B.W., Grohn, Y.T. and Nielsen, S.S. (2007). Nonparametric estimation of ROC curves based on Bayesian models when the true disease state is unknown. Journal of Agricultural, Biological and Environmental Statistics, 12, 128-146.
231. Webb, A.R. (2002). Statistical Pattern Recognition (2nd Edition). Wiley, Chichester.
232. Weinstein, M.C., Fineberg, H.V., Elstein, A.S., Frasier, H.S., Neuhauser, D., Neutra, R.R. and McNeil, B.J. (1996). Clinical Decision Analysis. W.B. Saunders, Philadelphia.
233. Wieand, S., Gail, M.H., James, B.R. and James, K.L. (1989). A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika, 76, 585-592.
234. Wilks, D.S. (2001). A skill score based on economic value for probability forecasts. Meteorological Applications, 8, 209-219.
235. Williams, P., Hand, D.J. and Tarnopolsky, A. (1982) The problem of screening for uncommon disorders a comment on the Eating Attitudes Test, Psychological Medicine, 12, 431-4.
236. Wolfe, D.A. and Hogg, R.V. (1971). On constructing statistics and reporting data. American Statistician, 25, 27-30.
237. Wolfowitz, J. (1957). The minimum distance method. Annals of Mathematical Statistics, 28, 75-88.
238. Yan L., Dodier R., Mozer M.C. and Wolniewicz R. (2003). Optimizing classifier performance via the Wilcoxon-Mann-Whitney statistics. In: Proceedings of the 20th International Conference on Machine Learning, ICML-2003, eds. T.Fawcett and N.Mishra.
239. Zheng, Y. and Heagerty, P.J. (2004). Semiparametric estimation of time-dependent ROC curves for longitudinal marker data. Biostatistics, 5, 615-632.
240. Zhou, X.-H. and Harezlak, J. (2002). Comparison of bandwidth selection methods for kernel smoothing of ROC curves. Statistics in Medicine, 21, 2045-2055.
241. Zhou, X.-H., Obuchowski, N.A. and McClish, D.K. (2002). Statistical Methods in Diagnostic Medicine. Wiley, New York.
242. Zou, K.H. (2001). Comparison of correlated receiver operating characteristic curves derived from repeated diagnostic test data. Academic Radiology, 8, 225-233.
243. Zou, K.H. and Hall, W.J. (2000). Two transformation models for estimating an ROC curve derived from continuous data. Journal of Applied Statistics, 27, 621-631.
244. Zou, K.H., Hall, W.J. and Shapiro, D.E. (1997). Smooth nonparametric receiver operating characteristic (ROC) curves for continuous diagnostic tests. Statistics in Medicine, 16, 2143-2156.
245. Zweig, M.H. and Campbell, G. (1993). Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clinical Chemistry, 39, 561-577.