The ROC isometrics approach to construct reliable classifiers

Intelligent Data Analysis

Volumn 13, Issue 1, 2009, Pages 3-37

  Vanderlooy, Stijn ^a   Sprinkhuizen Kuyper, Ida G ^b   Smirnov, Evgueni N ^a   van den Herik, H Jaap ^a  

^a MAASTRICHT UNIVERSITY   (Netherlands)

^b RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

Author keywords

Abstaining classifiers; Cost sensitive classification; Isometrics; Reliable classifiers; ROC analysis

Indexed keywords

EID: 61449239251     PISSN: 1088467X     EISSN: 15714128     Source Type: Journal    
DOI: 10.3233/IDA-2009-0354     Document Type: Article

References (46)

1. S. Vanderlooy, I. Sprinkhuizen-Kuyper and E. Smirnov, Reliable classifiers in ROC space, in: Proceedings of the 15th Annual Machine Learning Conference of Belgium and the Netherlands (BENELEARN 2006), Y. Saeys, E. Tsiporkova, B.D. Baets and Y.V. de Peer, eds, Ghent, Belgium, 2006, pp. 113-120.
2. S. Vanderlooy, I. Sprinkhuizen-Kuyper and E. Smirnov, An analysis of reliable classifiers through ROC isometrics, in: Proceedings of the ICML 2006 Workshop on ROC Analysis (ROCML 2006), N. Lachiche, C. Ferri and S. Macskassy, eds, Pittsburgh, USA, 2006, pp. 55-62.
3. N. Lachiche and P. Flach, Improving accuracy and cost of two-class and multiclass probabilistic classifiers using ROC curves, in: Proceedings of the 20th International Conference on Machine Learning (ICML 2003), T. Fawcett and N. Mishra, eds, Washington, DC, USA, AAAI Press, 2003, pp. 416-423.
4. J. Swets, R. Dawes and J. Monahan, Better decisions through science, Scientific American 283(4) (2000), 82-87.
5. T. Fawcett, ROC graphs: Notes and practical considerations for researchers. Technical Report HPL-2003-4, HP Laboratories, 2003.
6. F. Provost and T. Fawcett, Robust classification for imprecise environments, Machine Learning 42(3) (2001), 203-231.
7. F. Provost and T. Fawcett, Analysis and visualization of classifier performance: Comparison under imprecise class and cost distributions, in: Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining (KDD 1997), D. Heckerman, H. Mannila and D. Pregibon, eds, AAAI Press, 1997, pp. 43-48.
8. C. Ling, J. Huang and H. Zhang, AUC: A statistically consistent and more discriminating measure than accuracy, in: Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI 2003) G. Gottlob and T. Walsh, eds, AAAI Press, 2003.
9. F. Provost, T. Fawcett and R. Kohavi, The case against accuracy estimation for comparing induction algorithms, in: Proceedings of the 15th International Conference on Machine Learning (ICML 1998), J. Shavlik, ed., Madison, Wisconsin, USA, Morgan Kaufmann, 1998, pp. 43-48.
10. P. Flach and S. Wu, Repairing concavities in ROC curves, in: Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI 2005), L. Kaelbling and A. Saffiotti, eds, Edinburgh, UK, Professional Book Center, 2005, pp. 702-707.
11. T. Fawcett and P. Flach, A response to Webb and Ting's on the application of ROC analysis to predict classification performance under varying class distributions, Machine Learning 58(1) (2005), 33-38.
12. K. Ting, A study of the effect of class distribution using cost-sensitive learning, in: Proceedings of the 5th International Conference on Discovery Science (DS 2002), S. Lange, K. Satoh and C. Smith, eds, Lübeck, Germany, Springer, 2002, pp. 98-112.
13. G. Weiss and F. Provost, Learning when training data are costly: The effect of class distribution on tree induction, Journal of Artificial Intelligence Research 19 (2003), 315-354.
14. G. Weiss, K. McCarthy and B. Zabar, Cost-sensitive learning vs. sampling: Which is best for handling unbalanced classes with unequal error costs? in: Proceedings of the 2007 International Conference on Data Mining (DMIN 2007), R. Stahlbock, S. Crone and S. Lessmann, eds, Las Vegas, NV, USA, CSREA Press, 2007, pp. 35-41.
15. C. Elkan, The foundations of cost-sensitive learning, in: Proceedings of the 17th International Joint Conference on Artificial Intelligence (IJCAI 2001), B. Nebel, ed., Seattle, WA, USA, Morgan Kaufmann, 2001, pp. 973-978.
16. P. Flach, The geometry of ROC space: Understanding machine learning metrics through ROC isometrics, in: Proceedings of the 20th International Conference on Machine Learning (ICML 2003), T. Fawcett and N. Mishra, eds, Washington, DC, USA, AAAI Press, 2003, pp. 194-201.
17. C. Friedel, U. Rückert and S. Kramer, Cost curves for abstaining classifiers, in: Proceedings of the ICML 2006 Workshop on ROC Analysis (ROCML 2006), N. Lachiche, C. Ferri and S. Macskassy, eds, Pittsburgh, PA, USA, 2006, pp. 33-40.
18. C. Chow, An optimum character recognition system using decision functions, IRE Transactions on Electronic Computers 6(4) (1957), 247-254.
19. C. Chow, On optimum recognition error and reject tradeoff, IEEE Transactions on Information Theory 16(1) (1970), 41-46.
20. M. Pazzani, P. Murphy, K. Ali and D. Schulenburg, Trading off coverage for accuracy in forecasts: Applications to clinical data analysis, in: Working Notes of the AAAI Symposium on AI in Medicine, Stanford, CA, USA, AAAI Press, 1994, pp. 106-110.
21. C. Ferri and J. Hernández-Orallo, Cautious classifiers, in: Proceedings of the 1st InternationalWorkshop on ROC Analysis in Artificial Intelligence (ROCAI 2004), J. Hernández-Orallo, C. Ferri, N. Lachiche and P. Flach, eds, Valencia, Spain, 2004, PP. 27-36.
22. F. Tortorella, A ROC-based reject rule for dichotomizers, Pattern Recognition Letters 26(2) (2005), 167-180.
23. T. Pietraszek, On the use of ROC analysis for the optimization of abstaining classifiers, Machine Learning 68(2) (2007), 137-169.
24. C. Santos-Pereira and A. Pires, On optimal reject rules and ROC curves, Pattern Recognition Letters 26(7) (2005), 943-952.
25. R. Muzzolini, Y.H. Yang and R. Pierson, Classifier design with incomplete knowledge, Pattern Recognition 31(4) (1998), 345-369.
26. C. Frélicot and L. Mascarilla, Reject strategies driven combination of pattern classifiers, Pattern Analysis and Applications 5 (2) (2002), 234-243.
27. C. Ferri, P. Flach and J. Herández-Orallo, Delegating classifiers, in: Proceedings of the 1st International Workshop on ROC Analysis in Artificial Intelligence (ROCAI 2004), J. Hernández-Orallo, C. Ferri, N. Lachiche and P. Flach, eds, Valencia, Spain, 2004, pp. 37-44.
28. J. Fürnkranz and P. Flach, Roc 'n' rule learning - towards a better understanding of covering algorithms, Machine Learning 58(1) (2005), 39-77.
29. C.V. Rijsbergen, Information Retrieval, 2nd edn, Department of Computer Science, University of Glasgow, 1979.
30. S. Džeroski and I. Bratko, Handling noise in inductive logic programming, in: Proceedings of the 2nd International Workshop on Inductive Logic Programming (ILP 1992), S. Muggleton and K. Furukawa, eds, Tokyo, Japan, Institute for New Generation Computer Technology, 1992, pp. 109-125.
31. N. Lavrac and S. Dzeroski, Inductive Logic Programming: Techniques and Applications, Ellis Horwood, New York, NY, USA, 1994.
32. S. Macskassy and F. Provost, Confidence bands for ROC curves: Methods and an empirical study, in: Proceedings of the 1st International Workshop on ROC Analysis in Artificial Intelligence (ROCAI 2004), J. Hernández-Orallo, C. Ferri, N. Lachiche and P. Flach, eds, Valencia, Spain, 2004, pp. 61-70.
33. S. Agarwal, T. Graepel, R. Herbrich, S. Har-Peled and D. Roth, Generalization error bounds for the area under the ROC curve, Journal of Machine Learning Research 6(Apr) (2005), 393-425.
34. D. Newman, S. Hettich, C. Blake and C. Merz, UCI repository of machine learning databases, 1998.
35. I. Guyon, Data representation discovery workshop of the 20th international joint conference on neural networks (IJCNN 2007), 2007.
36. J. Ye, R. Janardan and Q. Li, Two-dimensional linear discriminant analysis. In: Advances in Neural Information Processing Systems 17 (NIPS 2004), Vancouver, British Columbia, Canada, 2004, 1569-1576.
37. T. Cover and P. Hart, Nearest neighbor pattern classification, IEEE Transactions on Information Theory 13(1) (1967), 21-27.
38. K. Proedrou, I. Nouretdinov, V. Vovk and A. Gammerman, Transductive confidence machines for pattern recognition. Technical Report 01-02, Royal Holloway University of London, London, UK, 2001.
39. P. Domingos and M. Pazzani, On the optimality of the simple Bayesian classifier under zero-one loss, Machine Learning 29(2) (1997), 103-130.
40. J. Lu, Y. Yang and G. Webb, Incremental discretization for naive Bayes classifier, in: Proceedings of the 2nd International Conference on Advanced Data Mining and Applications (ADMA 2006), X. Li, O. Zaïane and Z. Li, eds, Xi'an, China, Springer, 2006, pp. 223-238.
41. C. Marrocco, M. Molinara and F. Tortorella, An empirical comparison of ideal and empirical ROC-based reject rules, in: Proceedings of the 5th International Conference on Machine Learning and Data Mining in Pattern Recognition (MLDM 2007), P. Perner, ed., Leipzig, Germany, Springer, 2007, pp. 47-60.
42. F. Tortorella, Reducing the classification cost of support vector classifiers through an ROC-based reject rule, Pattern Analysis and Applications 7(2) (2004), 128-143.
43. T. Pietraszek, Classification of intrusion detection alerts using abstaining classifiers, Intelligent Data Analysis 11(3) (2007), 293-316.
44. B. Cestnik, Estimating probabilities: A crucial task in machine learning, in: Proceedings of the 9th European Conference on Artificial Intelligence (ECAI 1990), L. Aiello, ed., Stockholm, Sweden, Pitman Publishing, 1990, pp. 147-149.
45. A. Atiya, Estimating the posterior probabilities using the k-nearest neighbor rule, Neural Computation 17(3) (2005), 731-740.
46. A. Niculescu-Mizil and R. Caruana, Predicting good probabilities with supervised learning. In: Proceedings of the 22nd International Conference on Machine Learning (ICML 2005), Bonn, Germany, ACM Press, 2005, 625-632.