A unified view of performance metrics: Translating threshold choice into expected classification loss

Journal of Machine Learning Research

Volumn 13, Issue , 2012, Pages 2813-2869

  Hernández Orallo, José ^a   Flach, Peter ^b,c   Ferri, César ^a  

^a UNIVERSITAT POLITÈCNICA DE VALÈNCIA   (Spain)

^b UNIVERSITY OF BRISTOL   (United Kingdom)

^c Merchant Venturers Building ^*  (United Kingdom)

Author keywords

Area under the ROC curve (AUC); Brier score; Calibration loss; Classification performance metrics; Cost sensitive evaluation; Operating condition; Refinement loss

Indexed keywords

AREA UNDER THE ROC CURVE; BRIER SCORE; CLASSIFICATION PERFORMANCE; COST-SENSITIVE EVALUATION; OPERATING CONDITION;

ARTIFICIAL INTELLIGENCE; SOFTWARE ENGINEERING;

CALIBRATION;

EID: 84869160181     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (59)

1. N. M. Adams and D. J. Hand. Comparing classifiers when the misallocation costs are uncertain. Pattern Recognition, 32(7):1139-1147, 1999.
2. T. A. Alonzo, M. S. Pepe, and T. Lumley. Estimating disease prevalence in two-phase studies. Biostatistics, 4(2):313-326, 2003.
3. M. Ayer, H.D. Brunk, G.M. Ewing,W.T. Reid, and E. Silverman. An empirical distribution function for sampling with incomplete information. Annals of Mathematical Statistics, 5:641-647, 1955.
4. A. Bella, C. Ferri, J. Hernandez-Orallo, and M.J. Ramirez-Quintana. Calibration of machine learning models. In Handbook of Research on Machine Learning Applications, pages 128-146. IGI Global, 2009.
5. A. Bella, C. Ferri, J. Hern'andez-Orallo, and M.J. Ram'irez-Quintana. Quantification via probability estimators. In 2010 IEEE International Conference on DataMining, pages 737-742. IEEE, 2010.
6. G. W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3, 1950.
7. N. Brümmer. Measuring, refining and calibrating speaker and language information extracted from speech. Ph.D. Dissertation, Department of Electrical and Electronic Engineering, University of Stellenbosch, 2010.
8. A. Buja, W. Stuetzle, and Y. Shen. Loss functions for binary class probability estimation: structure and applications. http://www-stat.wharton. upenn.edu/∼buja/PAPERS/paper-proper-scoring.pdf, 2005.
9. I. Cohen andM. Goldszmidt. Properties and benefits of calibrated classifiers. Knowledge Discovery in Databases: PKDD 2004, pages 125-136, 2004. (Pubitemid 39740397)
10. C. Drummond and R. C. Holte. Explicitly representing expected cost: an alternative to ROC representation. In Knowledge Discovery and Data Mining, pages 198-207, 2000.
11. C. Drummond and R. C. Holte. Cost curves: an improved method for visualizing classifier performance. Machine Learning, 65(1):95-130, 2006. (Pubitemid 44451195)
12. C. Elkan. The foundations of cost-sensitive searning. In Proceedings of the Seventeenth International Conference on Artificial Intelligence (IJCAI-01), pages 973-978, San Francisco, CA, 2001.
13. T. Fawcett. Using rule sets to maximize ROC performance. In 2001 IEEE International Conference on Data Mining (ICDM-01), pages 131-138, 2001.
14. T. Fawcett. An introduction to ROC analysis. Pattern Recognition Letters, 27(8):861-874, 2006.
15. T. Fawcett and A. Niculescu-Mizil. PAV and the ROC convex hull. Machine Learning, 68(1): 97-106, July 2007. (Pubitemid 46828728)
16. T. Fawcett and F. Provost. Adaptive fraud detection. Data Mining and Knowledge Discovery, 1(3): 291-316, 1997.
17. C. Ferri, P.A. Flach, J. Herńandez-Orallo, and A. Senad. Modifying ROC curves to incorporate predicted probabilities. In Second Workshop on ROC Analysis in ML, ROCML, pages 33-40, 2005.
18. C. Ferri, J. Herńandez-Orallo, and R. Modroiu. An experimental comparison of performance measures for classification. Pattern Recognition Letters, 30(1):27-38, 2009. ISSN 0167-8655.
19. P. A. Flach. The geometry of ROC space: understanding machine learning metrics through ROC isometrics. In Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), pages 194-201, 2003.
20. P. A. Flach. Machine Learning: The Art and Science of Algorithms That Make Sense of Data. Cambridge University Press, to appear, 2012.
21. P. A. Flach and E. T. Matsubara. A simple lexicographic ranker and probability estimator. In 18th European Conference on Machine Learning, pages 575-582. Springer, 2007.
22. P. A. Flach, J. Hern'andez-Orallo, and C. Ferri. A coherent interpretation of AUC as a measure of aggregated classification performance. In Proceedings of the 28th International Conference on Machine Learning, ICML2011, 2011.
23. G. Forman. Quantifying counts and costs via classification. DataMining and Knowledge Discovery, 17(2):164-206, 2008.
24. J. H. Friedman. On bias, variance, 0/1loss, and the curse-of- dimensionality. Data Mining and Knowledge Discovery, 1(1):55-77, 1997.
25. M. Gebel. Multivariate Calibration of Classifier Scores into the Probability Space. PhD thesis, University of Dortmund, 2009.
26. T. Gneiting and A.E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-378, 2007. ISSN 0162-1459.
27. I. J. Good. Rational decisions. Journal of the Royal Statistical Society, Series B, 14:107-114, 1952.
28. D. J. Hand. Construction and Assessment of Classification Rules. John Wiley & Sons Inc, 1997.
29. D. J. Hand. Measuring classifier performance: a coherent alternative to the area under the ROC curve. Machine Learning, 77(1):103-123, 2009.
30. D. J. Hand. Evaluating diagnostic tests: the area under the ROC curve and the balance of errors. Statistics in Medicine, 29(14):1502-1510, 2010.
31. D. J. Hand and C. Anagnostopoulos. When is the area under the ROC curve an appropriate measure of classifier performance? Technical report, Department of Mathematics, Imperial College, London, 2011.
32. D. J. Hand and C. Anagnostopoulos. A better Beta for the H measure of classification performance. arXiv:1202.2564v1 [stat] 12 Feb 2012, page 9, 2012.
33. J. Hern'andez-Orallo, P. A. Flach, and C. Ferri. Brier curves: a new cost-based visualisation of classifier performance. In Proceedings of the 28th International Conference on Machine Learning, ICML2011, 2011.
34. J. Hern'andez-Orallo, P. Flach, and C. Ferri. ROC curves in cost space. In Submitted, 2012.
35. N. Lachiche and P. Flach. Improving accuracy and cost of two-class and multi-class probabilistic classifiers using roc curves. In International Conference on Machine Learning, pages 416-423, 2003.
36. G. Lebanon and J. D. Lafferty. Cranking: combining rankings using conditional probability models on permutations. In Machine Learning, Proceedings of the Nineteenth International Conference (ICML 2002), pages 363-370, 2002.
37. A. Martin, G. Doddington, T. Kamm, M. Ordowski, and M. Przybocki. The DET curve in assessment of detection task performance. In Fifth European Conference on Speech Communication and Technology, 1997.
38. A. H. Murphy. A note on the utility of probabilistic predictions and the probability score in the cost-loss ratio decision situation. Journal of Applied Meteorology, 5:534-536, 1966. ISSN 0894-8763.
39. A. H. Murphy. Measures of the utility of probabilistic predictions in cost-loss ratio decision situations in which knowledge of the cost-loss ratios is incomplete. Journal of Applied Meteorology, 8:863-873, 1969. ISSN 0894-8763.
40. A. H. Murphy. A new vector partition of the probability score. Journal of Applied Meteorology, 12: 595-600, 1973.
41. A. H. Murphy and R. L. Winkler. Scoring rules in probability assessment and evaluation. Acta Psychologica, 34:273-286, 1970.
42. J. M. Murphy, D. M. Berwick, M. C. Weinstein, J. F. Borus, S. H. Budman, and G. L. Klerman. Performance of screening and diagnostic tests: application of receiver operating characteristic analysis. Archives of General Psychiatry, 44(6):550, 1987.
43. J. Neyman. Contribution to the theory of sampling human populations. Journal of the American Statistical Association, 33(201):101-116, 1938.
44. A. Niculescu-Mizil and R. Caruana. Obtaining calibrated probabilities from boosting. In The 21st Conference on Uncertainty in Artificial Intelligence (UAI 05), AUAI Press, pages 413-420. AUAI Press, 2005.
45. A. Niculescu-Mizil and R. Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine learning, pages 625-632. ACM, 2005. (Pubitemid 43183386)
46. G. Piatetsky-Shapiro and B. Masand. Estimating campaign benefits and modeling lift. In Proceedings of the fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 193. ACM, 1999.
47. J. C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, pages 61-74. MIT Press, Boston, 1999.
48. R. C. Prati, G. E. A. P. A. Batista, and M. C. Monard. A survey on graphical methods for classification predictive performance evaluation. IEEE Transactions on Knowledge and Data Engineering, 23:1601-1618, 2011.
49. F. Provost and T. Fawcett. Robust classification for imprecise environments. Machine Learning, 42 (3):203-231, 2001. (Pubitemid 32188799)
50. M. D. Reid and R. C. Williamson. Composite binary losses. The Journal of Machine Learning Research, 11:2387-2422, 2010.
51. M. D. Reid and R. C. Williamson. Information, divergence and risk for binary experiments. The Journal of Machine Learning Research, 12:731-817, 2011.
52. T. Robertson, F. Wright, and R. Dykstra. Order Restricted Statistical Inference. John Wiley and Sons, New York, 1988.
53. S. Rüping. Robust probabilistic calibration. Machine Learning: ECML 2006, pages 743-750, 2006.
54. J. A. Swets, R. M. Dawes, and J. Monahan. Better decisions through science. Scientific American, 283(4):82-87, October 2000.
55. A. Tenenbein. A double sampling scheme for estimating from binomial data with misclassifications. Journal of the American Statistical Association, pages 1350-1361, 1970.
56. P. Turney. Types of cost in inductive concept learning. Canada National Research Council Publications Archive, 2000.
57. S. Wieand, M.H. Gail, B.R. James, and K.L. James. A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika, 76(3):585-592, 1989.
58. B. Zadrozny and C. Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), pages 609-616, 2001.
59. B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. In The 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 694-699, 2002.