ROC confidence bands: An empirical evaluation

ICML 2005 - Proceedings of the 22nd International Conference on Machine Learning

Volumn , Issue , 2005, Pages 537-544

  Macskassy, Sofus A ^a   Provost, Foster ^a   Rosset, Saharon ^b  

^a NEW YORK UNIVERSITY   (United States)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLASS-CONDITIONAL MODEL; EMPIRICAL EVALUATION; ROC CURVES;

BAND STRUCTURE; BENCHMARKING; DATABASE SYSTEMS; MATHEMATICAL MODELS;

LEARNING SYSTEMS;

EID: 31844447615     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1102351.1102419     Document Type: Conference Paper

References (23)

1. Bennett, P. N. (2003). Using Asymmetric Distributions to Improve Text Classifier Probability Estimates. Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Toronto, Canada: ACM Press.
2. Bradley, A. P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 7, 1145-1159.
3. Campbell, G. (1994). Advances in statistical methodology for the evaluation of diagnostic and laboratory tests. Statistics in Medicine, 13, 499-508.
4. Claeskens, G., Jing, B.-Y., Peng, L., & Zhou, W. (2003). Empirical likelihood confidence regions for comparison distributions and roc curves. The Canadian Journal of Statistics, 31, 173-190.
5. Conover, W. J. (1980). Practical nonparametric statistics. New York: Wiley. 2nd edition.
6. Dorfman, D. D., & Alf, E. (1969). Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals-rating method data. Journal of Mathematical Psychology, 6, 487-496.
7. Efron, B., & Tibshirani, R. (1993). An Introduction to the Bootstrap. Chapman & Hall.
8. Fawcett, T. (2003). ROC Graphs: Notes and Practical Considerations for Data Mining Researchers (Technical Report HPL-2003-4). HP Labs.
9. Hall, P. G., Hyndman, R. J., & Fan, Y. (2004). Nonparametric confidence intervals for receiver operating characteristic curves. Biometrika, 91, 743-750.
10. Hilgers, R. A. (1991). Distribution-free confidence bounds for ROC curves. Methods of Information in Medicine, 30, 96-101.
11. Kohavi, R. (1996). Scaling Up the Accuracy of Naive-Bayes Classifiers: a Decision-Tree Hybrid. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining. Menlo Park, CA: AAAI Press/MIT Press.
12. Landwehr, N., Hall, M., & Frank, E. (2003). Logistic Model Trees. Proceedings of the 16th European Conference on Machine Learning.
13. Ma, G., & Hall, W. J. (1993). Confidence bands for receiver operating characteristic curves. Medical Decision Making, 13, 191-197.
14. Macskassy, S., & Provost, F. (2004). Confidence Bands for ROC Curves: Methods and an Empirical Study. Proceedings of the First Workshop on ROC Analysis in AI (ROCAI-2004) at ECAI-2004.
15. Macskassy, S., Provost, F., & Rosset, S. (2005). Pointwise ROC Confidence Bounds: An Empirical Evaluation; Proceedings of the Workshop on ROC Analysis in Machine Learning (ROCML-2005) at ICML-2005.
16. Margineantu, D. D., & Dietterich, T. G. (2000). Bootstrap methods for the cost-sensitive evaluation of classifiers. International Conference on Machine Learning (ICML) (pp. 582-590).
17. Metz, C. E., Herman, B. A., & Roe, C. A. (1998). Statistical Comparison of Two ROC-curve Estimates Obtained from Partially-paired Datasets. Medical Decision Making, 18, 110-121.
18. Perlich, C., Provost, F., & Simonoff, J. (2003). Tree induction vs. logistic regression: A learning-curve analysis. Journal of Machine Learning Research, 4, 211-255.
19. Provost, F., Fawcett, T., & Kohavi, R. (1998). The case against accuracy estimation for comparing induction algorithms. Proceedings of the Fifteenth International Conference on Machine Learning (pp. 445-453). San Francisco, CA: Morgan Kaufman.
20. Stein, R. (2002). Benchmarking Default Prediction Models: Pitfalls and Remedies in Model Validation (Technical Report #030124). Moody's KMV.
21. Witten, I. H., & Frank, E. (2000). In Data Mining: Practical machine learning tools with Java implementations. San Francisco: Morgan Kaufmann.
22. Working, H., & Hotelling, H. (1929). Application of the theory of error to the interpreation of trends. Journal of the American Statistical Assocation, 24, 73-85.
23. Zou, K. H., Hall, W. J., & Shapiro, D. E. (1997). Smooth nonparametric receiver operating characteristic (ROC) curves for continuous diagnostic tests. Statistics in Medicine, 16, 2143-2156.