An empirical evaluation of ranking measures with respect to robustness to noise

Journal of Artificial Intelligence Research

Volumn 49, Issue , 2014, Pages 241-267

  Berrar, Daniel ^a  

^a TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; ELECTRONICS ENGINEERING;

AREA UNDER THE ROC CURVE; CONVEX HULL; EMPIRICAL EVALUATIONS; KOLMOGOROV-SMIRNOV STATISTICS; MODEL EVALUATION; RANKING MEASURES; REAL-WORLD; ROBUSTNESS TO NOISE;

COMPUTATIONAL COMPLEXITY;

EID: 84894418956     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.4136     Document Type: Article

References (35)

1. Adams, N., & Hand, D. (1999). Comparing classifiers when the misallocation costs are uncertain. Pattern Recognition, 32 (7), 1139-1147.
2. Bache, K., & Lichman, M. (2013). UCI machine learning repository. [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.
3. Bamber, D. (1975). The area under the ordinal dominance graph and the area below the receiver operating characteristic curve. Journal of Mathematical Psychology, 12, 387-415.
4. Bengio, S., Mariéthoz, J., & Keller, M. (2005). The expected performance curve. Proceedings of the ICML 2005 workshop on ROC Analysis in Machine Learning, 9-16.
5. Berrar, D., & Flach, P. (2012). Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them). Briefings in Bioinformatics, 13 (1), 83-97.
6. Bradley, A. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30 (3), 1145-1159. (Pubitemid 127406521)
7. Bradley, A. (2013). ROC curve equivalence using the Kolmogorov-Smirnov test. Pattern Recognition Letters, 34 (5), 470-475.
8. Calders, T., & Jaroszewicz, S. (2007). Eficient AUC optimization for classification. In Kok, J., Koronacki, J., de Mántaras, R., Matwin, S., Mladenič, D., & Skowron, A. (Eds.), Proceedings of the 11th European Conference on Principles and Practice of Knowledge Discovery in Databases, pp. 42-53. Springer.
9. th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 69-78. ACM Press. (Pubitemid 40114917)
10. Drummond, C., & Holte, R. (2006). Cost curves: An improved method for visualizing classifier performance. Machine Learning, 65, 95-130. (Pubitemid 44451195)
11. Fawcett, T. (2004). ROC graphs: Notes and practical considerations for researchers. Kluwer Academic Publishers, 1-38.
12. Ferri, C., Flach, P., Hernández-Orallo, J., & Senad, A. (2005). Modifying ROC curves to incorporate predicted probabilities. In Proceedings of the 2nd Workshop on ROC Analysis in Machine Learning. Bonn, Germany.
13. Ferri, C., Hernández-Orallo, J., & Modroiu, R. (2009). An experimental comparison of performance measures for classification. Pattern Recognition Letters, 30, 27-38.
14. th International Conference on Machine Learning, pp. 194-201. AAAI Press.
15. Flach, P. (2010). ROC analysis. In Sammut, C., & Webb, G. (Eds.), Encyclopedia of Machine Learning, pp. 869-874. Springer.
16. th International Conference on Machine Learning, pp. 69-78.
17. Hanczar, B., Hua, J., Sima, C., Weinstein, J., Bittner, M., & Dougherty, E. (2010). Smallsample precision of ROC-related estimates. Bioinformatics, 26, 822-830.
18. Hand, D. (2006). Classifier technology and the illusion of progress. Statistical Science, 21 (1), 1-14. (Pubitemid 44046906)
19. Hand, D. (2009). Measuring classifier performance: a coherent alternative to the area under the ROC curve. Machine Learning, 77, 103-123.
20. Hand, D., & Anagnostopoulos, C. (2013). When is the area under the receiver operating characteristic curve an appropriate measure of classifier performance?. Pattern Recognition Letters, 34 (5), 492-495.
21. Hanley, J., & McNeil, B. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148 (3), 839-843. (Pubitemid 13033817)
22. Hernández-Orallo, J., Flach, P., & Ferri, C. (2012). A unified view of performance metrics: Translating threshold choice into expected classification loss. Journal of Machine Learning Research, 13, 2813-2869.
23. Hilden, J. (1991). The area under the ROC curve and its competitors. Medical Decision Making, 11 (2), 95-101.
24. Korb, K., Hope, L., & Hughes, M. (2001). The evaluation of predictive learners: Some theoretical and empirical results. In DeRaedt, L., & Flach, P. (Eds.), Lecture Notes in Artificial Intelligence, pp. 276-287. Springer. (Pubitemid 33331076)
25. Lobo, J., Jiménez-Valverde, A., & Real, R. (2008). AUC: a misleading measure of the performance of predictive distribution models. Global Ecology and Biogeography, 17, 145-151. (Pubitemid 351228538)
26. McClish, D. (1989). Analyzing a portion of the ROC curve. Medical Decision Making, 9 (3), 190-195. (Pubitemid 19181460)
27. Parker, C. (2013). On measuring the performance of binary classifiers. Knowledge and Information Systems, 35, 131-152.
28. Prati, R., Batista, G., & Monard, M. (2011). A survey on graphical methods for classification predictive performance evaluation. IEEE Transactions on Knowledge and Data Engineering, 23 (11), 1601-1618.
29. Provost, F., & Fawcett, T. (2001). Robust classification for imprecise environments. Machine Learning, 42 (3), 203-231. (Pubitemid 32188799)
30. R Development Core Team (2009). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0.
31. Sheskin, D. (2007). Handbook of Parametric and Nonparametric Statistical Procedures. 4th Edition, Chapman and Hall, London/New York.
32. Vanderlooy, S., & Hüllermeier, E. (2008). A critical analysis of variants of the AUC. Machine Learning, 72, 247-262.
33. Webb, G., & Ting, K. (2005). On the application of ROC analysis to predict classification performance under varying class distributions. Machine Learning, 58 (1), 25-32. (Pubitemid 40356737)
34. Wu, S., & Flach, P. (2005). A scored AUC metric for classifier evaluation and selection. Proceedings of the Second Workshop on ROC Analysis in Machine Learning.
35. Wu, S., Flach, P., & Ferri, C. (2007). An improved model selection heuristic for AUC. In Kok, J., Koronacki, J., de Mántaras, R., Matwin, S., Mladenič, D., & Skowron, A. (Eds.), Proceedings of the 18th European Conference on Machine Learning (ECML 2007), pp. 478-489. Springer.