The problem of area under the curve

Proceedings of 2012 IEEE International Conference on Information Science and Technology, ICIST 2012

Volumn , Issue , 2012, Pages 567-573

  Powers, David M W ^a  

^a FLINDERS UNIVERSITY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

AREA UNDER THE CURVES; EXPERIMENTAL ALGORITHMS; F-MEASURE; KAPPA STATISTIC; MULTIPLE CLASS; MULTIPLE SOURCE; RECALL AND PRECISION; ROC ANALYSIS; ROC CURVES; TRADE OFF;

ALGORITHMS; TECHNOLOGY;

INFORMATION SCIENCE;

EID: 84864758814     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICIST.2012.6221710     Document Type: Conference Paper

References (32)

1. D. Janez, "Statistical comparisons of classifiers over multiple datasets," Journal of Machine Learning Research, vol. 7, pp. 1-30, 2006.
2. F. Provost and T. Fawcett, "Robust classification for imprecise environments," Machine Learning, vol. 44, pp. 203-231, 2001.
3. J.S. Uebersax, "Diversity of decision-making models and the measurement of interrater agreement," Psychological Bulletin, vol. 101, pp. 140-146, 1987.
4. A.P. Bradley, "The use of the area under the ROC curve in the evaluation of machinelearning algorithms," Pattern Recognition, vol. 30, pp. 1145-1159, 1997.
5. J. Carletta, "Assessing agreement on classification tasks: the kappa statistic," Computational Linguistics, vol. 22, pp. 249-254, 1996.
6. J. Cohen, "A coefficient of agreement for nominal scales," Educational and Psychological Measurement, pp. 37-46, 1960.
7. J. Cohen, "Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit," Psychological Bulletin, vol. 70, pp. 213-20, 1968.
8. T. Fawcett, "An introduction to ROC analysis," Pattern Recognition Letters, vol. 27, pp. 861-874, 2006.
9. P.A. Flach, "The Geometry of ROC Space: Understanding Machine Learning Metrics through ROC Isometrics," in Proc. Twentieth International Conference on Machine Learning (ICML-2003), Washington DC, 2003, pp. 226-233.
10. J. Fürnkranz and P.A. Flach, "ROC 'n' rule learning - towards a better understanding of covering algorithms," Machine Learning, vol. 58, pp. 39-77, 2005.
11. D.J. Hand, "Measuring classifier performance: a coherent alternative to the area under the ROC curve," Machine Learning, vol. 77, pp. 103-123, 2009.
12. D.J. Hand and R.J. Till, "A simple generalisation of the area under the ROC curve for multiple class classification problems," Machine Learning, vol. 45, pp. 171-186, 2001.
13. A. Ben-David, "About the relationship between ROC curves and Cohen's kappa," Engineering Applications of AI, vol. 21, pp. 874-882, 2008.
14. J.A. Cohen, "A coefficient of agreement for nominal scales, educational and psychological measurement," Psychological Measurement, vol. 20, pp. 37-46, 1960.
15. J.L. Fleiss, Statistical Methods for Rates and Proportions. John Wiley and Sons, 2nd edition, 1981.
16. B. Di Eugenio and M. Glass, "The Kappa statistic: a second look," Computational Linguistics, vol. 30, pp. 95-101, 2004.
17. A. Ben-David, "Comparison of classification accuracy using Cohen's Weighted Kappa," Expert Systems with Applications, vol. 34, pp. 825-832, 2008.
18. M.J. Warrens, "Inequalities between multi-rater kappas," Advances in Data Analysis and Classification, vol. 4, pp. 271-286, 2010.
19. M.J. Warrens, "A formal proof of a paradox associated with Cohen's kappa," Journal of Classificaiton, vol. 27, pp. 322-332, 2010.
20. M.J. Warrens, "A Kraemer-type rescaling that transforms the Odds Ratio into the Weighted Kappa Coefficient," Psychometrika, vol. 75, pp. 328-330, 2010.
21. M.J. Warrens, "Cohen's linearly wieghted Kappa is a weighted average of 2x2 Kappas," Psychometrika, vol. 76, pp. 471-486, 2011.
22. T.P. Hutchinson, "Focus on psychometrics. Kappa muddles together two sources of disagreement: tetrachoric correlation is preferable," Research in Nursing & Health, vol. 16, pp. 313-316, 1993.
23. D.G. Bonett and R.M. Price, "Inferential methods for the tetrachoric correlation coefficient," Journal of Educational and Behavioral Statistics, vol. 30, pp. 213-225, 2005.
24. P. Perruchet and R. Peereman, "The exploitation of distributional information in syllable processing," J. Neurolinguistics, vol. 17, pp. 97-119, 2004.
25. D.R. Shanks, "Is human learning rational?" Quarterly Journal of Experimental Psychology, vol. 48, pp. 57-279, 1995.
26. D.M.W. Powers, "Recall and Precision versus the Bookmaker," in Proc. of the International Conference on Cognitive Science (ICSC-2003), Sydney Australia, pp. 529-534, 2003.
27. D.M.W. Powers, "Evaluation: from precision, recall and F-measure to informedness, markedness & correlation," Journal of Machine Learning Technologies, vol. 2, pp. 37-63, 2011.
28. rd Edition, Morgan Kaufmann, 2011.
29. D.M.W. Powers, "The problem of Kappa," European Chapter of the Association for Computational Linguistics, to be published.
30. P.A. Flach and S. Wu, "Repairing Concavities in ROC Curves," in Proc. 19th International Joint Conference on Artificial Intelligence (IJCAI'05, 2005), pp. 702-707.
31. J. Entwisle and D.M.W. Powers, "The Present Use of Statistics in the Evaluation of NLP Parsers," in Proc. NeMLaP3/CoNLL98 Joint Conference, Sydney, 1998, pp. 215-224.
32. D.M.W. Powers, "ROC-ConCert ROC-based measurement of Consistency and Certainty," Spring World Congress on Science and Technology, to be published.