An introduction to ROC analysis

Pattern Recognition Letters

Volumn 27, Issue 8, 2006, Pages 861-874

  Fawcett, Tom ^a  

^a Institute for the Study of Learning and Expertise   (United States)

Author keywords

Classifier evaluation; Evaluation metrics; ROC analysis

Indexed keywords

DATA MINING; DECISION MAKING; METRIC SYSTEM; PERFORMANCE;

CLASSIFIER EVALUATION; EVALUATION METRICS; RECEIVER OPERATING CHARACTERISTICS (ROC) GRAPHS;

GRAPH THEORY;

EID: 33646023117     PISSN: 01678655     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patrec.2005.10.010     Document Type: Article

References (31)

1. Bradley A.P. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recogn. 30 7 (1997) 1145-1159
2. Breiman L., Friedman J., Olshen R., and Stone C. Classification and Regression Trees (1984), Wadsworth International Group, Belmont, CA
3. Clearwater S., and Stern E. A rule-learning program in high energy physics event classification. Comput. Phys. Commun. 67 (1991) 159-182
4. Domingos, P., 1999. MetaCost: A general method for making classifiers cost-sensitive. In: Proc. Fifth ACM SIGKDD Internat. Conf.on Knowledge Discovery and Data Mining, pp. 155-164.
5. Egan J.P. Signal detection theory and ROC analysis. Series in Cognition and Perception (1975), Academic Press, New York
6. Fawcett, T., 2001. Using rule sets to maximize ROC performance. In: Proc. IEEE Internat. Conf. on Data Mining (ICDM-2001), pp. 131-138.
7. Fawcett T., and Provost F. Combining data mining and machine learning for effective user profiling. In: Simoudis E., Han J., and Fayyad U. (Eds). Proc. Second Internat. Conf. on Knowledge Discovery and Data Mining (1996), AAAI Press, Menlo Park, CA 8-13
8. Fawcett T., and Provost F. Adaptive fraud detection. Data Mining and Knowledge Discovery 1 3 (1997) 291-316
9. Flach P., and Wu S. Repairing concavities in ROC curves. Proc. 2003 UK Workshop on Computational Intelligence (2003), University of Bristol 38-44
10. Forman, G., 2002. A method for discovering the insignificance of one's best classifier and the unlearnability of a classification task. In: Lavrac, N., Motoda, H., Fawcett, T. (Eds.), Proc. First Internat. Workshop on Data Mining Lessons Learned (DMLL-2002). Available from: http://www.purl.org/NET/tfawcett/DMLL-2002/Forman.pdf.
11. Hand D.J., and Till R.J. A simple generalization of the area under the ROC curve to multiple class classification problems. Mach. Learning 45 2 (2001) 171-186
12. Hanley J.A., and McNeil B.J. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143 (1982) 29-36
13. Holte, R., 2002. Personal communication.
14. Kubat M., Holte R.C., and Matwin S. Machine learning for the detection of oil spills in satellite radar images. Machine Learning 30 2-3 (1998) 195-215
15. Lane, T., 2000. Extensions of ROC analysis to multi-class domains. In: Dietterich, T., Margineantu, D., Provost, F., Turney, P. (Eds.), ICML-2000 Workshop on Cost-Sensitive Learning.
16. Lewis D. Representation quality in text classification: An introduction and experiment. Proc. Workshop on Speech and Natural Language (1990), Morgan Kaufmann, Hidden Valley, PA 288-295
17. Lewis D. Evaluating text categorization. Proc. Speech and Natural Language Workshop (1991), Morgan Kaufmann 312-318
18. Macskassy, S., Provost, F., 2004. Confidence bands for ROC curves: Methods and an empirical study. In: Proc. First Workshop on ROC Analysis in AI (ROCAI-04).
19. Provost, F., Domingos, P., 2001. Well-trained PETs: Improving probability estimation trees, CeDER Working Paper #IS-00-04, Stern School of Business, New York University, NY, NY 10012.
20. Provost F., and Fawcett T. Analysis and visualization of classifier performance: Comparison under imprecise class and cost distributions. Proc. Third Internat. Conf. on Knowledge Discovery and Data Mining (KDD-97) (1997), AAAI Press, Menlo Park, CA 43-48
21. Provost F., and Fawcett T. Robust classification systems for imprecise environments. Proc. AAAI-98 (1998), AAAI Press, Menlo Park, CA 706-713. Available from:
22. Provost F., and Fawcett T. Robust classification for imprecise environments. Mach. Learning 42 3 (2001) 203-231
23. Provost F., Fawcett T., and Kohavi R. The case against accuracy estimation for comparing induction algorithms. In: Shavlik J. (Ed). Proc. ICML-98 (1998), Morgan Kaufmann, San Francisco, CA 445-453. Available from:
24. Saitta L., and Neri F. Learning in the "real world". Mach. Learning 30 (1998) 133-163
25. Spackman K.A. Signal detection theory: Valuable tools for evaluating inductive learning. Proc. Sixth Internat. Workshop on Machine Learning (1989), Morgan Kaufman, San Mateo, CA 160-163
26. Srinivasan, A., 1999. Note on the location of optimal classifiers in n-dimensional ROC space. Technical Report PRG-TR-2-99, Oxford University Computing Laboratory, Oxford, England. Available from: .
27. Swets J. Measuring the accuracy of diagnostic systems. Science 240 (1988) 1285-1293
28. Swets J.A., Dawes R.M., and Monahan J. Better decisions through science. Scientific American 283 (2000) 82-87
29. van der Putten, P., van Someren, M., 2000. CoIL challenge 2000: The insurance company case. Technical Report 2000-09, Leiden Institute of Advanced Computer Science, Universiteit van Leiden. Available from: .
30. Zadrozny, B., Elkan, C., 2001. Obtaining calibrated probability estimates from decision trees and naive Bayesian classiers. In: Proc. Eighteenth Internat. Conf. on Machine Learning, pp. 609-616.
31. Zou, K.H., 2002. Receiver operating characteristic (ROC) literature research. On-line bibliography available from: .