Efficient multiclass ROC approximation by decomposition via confusion matrix perturbation analysis

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 30, Issue 5, 2008, Pages 810-822

  Landgrebe, Thomas C W ^a   Duin, Robert P W ^a  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

Author keywords

Classifier design and evaluation; Cost sensitive; Design methodology; Machine learning; Multiclass analysis; Pattern recognition; Receiver operator characteristic; Volume under the ROC

Indexed keywords

ALGORITHMS; COMPUTATIONAL COMPLEXITY; LEARNING SYSTEMS; OPTIMIZATION; PERTURBATION TECHNIQUES;

COST-SENSITIVE; MULTICLASS ANALYSIS; RECEIVER OPERATOR CHARACTERISTICS;

PATTERN RECOGNITION;

ALGORITHM; ARTICLE; ARTIFACT; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; CLUSTER ANALYSIS; INFORMATION RETRIEVAL; METHODOLOGY; ROC CURVE; STATISTICAL ANALYSIS;

ALGORITHMS; ARTIFACTS; ARTIFICIAL INTELLIGENCE; CLUSTER ANALYSIS; DATA INTERPRETATION, STATISTICAL; INFORMATION STORAGE AND RETRIEVAL; PATTERN RECOGNITION, AUTOMATED; ROC CURVE;

EID: 41749097006     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2007.70740     Document Type: Article

References (30)

1. C. Metz, "Basic Principles of ROC Analysis," Seminars in Nuclear Medicine, vol. 3, no. 4, 1978.
2. T. Fawcett, "An Introduction to ROC analysis," Pattern Recognition Letters, special issue on ROC analysis, vol. 27, pp. 861-874, 2005.
3. F. Provost and T. Fawcett, "Robust Classification for Imprecise Environments," Machine Learning, vol. 42, pp. 203-231, 2001.
4. R. Duda, P. Hart, and D. Stork, Pattern Classification, second ed. Wiley-Interscience, 2001.
5. A. Bradley, "The Use of the Area under the ROC Curve in the Evaluation of Machine Learning Algorithms," Pattern Recognition, vol. 30, no. 7, pp. 1145-1159, 1997.
6. T. Landgrebe and R. Duin, "Combining Accuracy and Prior Sensitivity for Classifier Design under Prior Uncertainty," Proc. IAPR Int'l Workshops Structural and Syntactic Pattern Recognition, pp. 512-521, Aug. 2006.
7. D. Mossman, "Three-Way ROC's," Medical Decision Making, vol. 19, pp. 78-89, 1999.
8. S. Dreisetl, S. Ohno-Machado, and M. Binder, "Comparing Trichotomous Tests by Three-Way ROC Analysis," Medical Decision Making, vol. 20, no. 3, pp. 323-331, 2000.
9. T. Landgrebe and R. Duin, "Approximating the Multiclass ROC by Pairwise Analysis," Pattern Recognition Letters, 2007.
10. A. Srinivasan, "Note on the Location of Optimal Classifiers in N Dimensional ROC Space," Technical Report PRG-TR-2-99, Computing Laboratory, Oxford Univ., Oct. 1999.
11. F. Provost and T. Fawcett, "Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions," Proc. Third Int'l Conf. Knowledge Discovery and Data Mining, pp. 43-48, 2001.
12. C. Ferri, J. Hernandez-Orallo, and M. Salido, "Volume Under the ROC Surface for Multi-Class Problems," Proc. 14th European Conf. Machine Learning, pp. 108-120, 2003.
13. D. Edwards, C. Metz, and R. Nishikawa, "The Hypervolume under the ROC Hypersurface of 'Near-Guessing' and 'Near-Perfect' Observers in N Class Classification Tasks," IEEE Trans. Medical Imaging, vol. 24, no. 3, pp. 293-299, Mar. 2005.
14. T. Landgrebe and R. Duin, "A Simplified Extension of the Area under the ROC to the Multiclass Domain," Proc. 17th Ann. Symp. Pattern Recognition Assoc. South Africa, Nov. 2006.
15. F. Provost and P. Domingos, Well Trained PETs: Improving Probability Estimation Trees, CeDER Working Paper IS-00-04, Stern School of Business, New York Univ., 2001.
16. D. Hand and R. Till, "A Simple Generalisation of the Area under the ROC Curve for Multiple Class Classification Problems," Machine Learning vol. 45, pp. 171-186, 2001.
17. N. Lachiche and P. Flach, "Improving Accuracy and Cost of Two-Class and Multi-Class Probabilistic Classifiers Using ROC Curves," Proc. 20th Int'l Conf. Machine Learning, pp. 416-423, 2003.
18. D. O'Brien and R. Gray, "Improving Classification Performance by Exploring the Role of Cost Matrices in Partitioning the Estimated Class Probability Space," Proc. 22nd Int'l Conf. Machine Learning Workshop ROC Analysis in Machine Learning, 2005.
19. R. Everson and J. Fieldsend, "Multi-Class ROC Analysis from a Multi-Objective Optimisation Perspective," Pattern Recognition Letters, special issue on ROC analysis, vol. 27, 2005.
20. P. Domingos, "MetaCost: A General Method for Making Classifiers Cost-Sensitive," Proc. Fifth ACM Int'l Conf. Knowledge Discovery and Data Mining, pp. 155-164, 1999.
21. N. Abe, B. Zadrozny, and J. Langford, "An Iterative Method for Multi-Class Cost-Sensitive Learning," Proc. 10th ACM Int'l Conf. Knowledge Discovery and Data Mining, pp. 3-11, Aug. 2004.
22. D. Edwards, C. Metz, and M. Kupinski, "Ideal Observers and Optimal ROC Hypersurfaces in N-Class Classification," IEEE Trans. Medical Imaging, vol. 23, no. 7, pp. 891-895, July 2004.
23. T. Landgrebe and R. Duin, "On Neyman-Pearson Optimisation for Multiclass Classifiers," Proc. 16th Ann. Symp. Pattern Recognition Assoc. South Africa, Nov. 2005.
24. M. Li and I. Sethi, "Confidence-Based Classifier Design," Pattern Recognition, vol. 39, pp. 1230-1240, 2006.
25. P. Murphy and D. Aha, "CI Repository of Machine Learning Databases," Dept. of Information and Computer Science, Univ. of California, ftp://ftp.ics.uci.edu/pub/machine-learning-data bases, 1992.
26. M. Loog, R. Duin, and H.-U.R., "Multiclass Linear Dimension Reduction by Weighted Pairwise Fisher Criteria," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 7, pp. 762-766, July 2001.
27. "Delve Datasets, Image Segmentation," Univ. of Toronto, http:// www.cs.toronto.edu, 2008.
28. ELENA Project, "European ESPRIT 5516 Project," Satimage Dataset, ftp://ftp.dice.ucl.ac.be/pub/neural-nets/ELENA/databases, 2004.
29. C. Wilson and M.M.D. Garris, "Handprinted Character Database 3," Advanced Systems Division, Nat'l Inst. of Standards and Technology, 1992.
30. R. McDonald, "The Mean Subjective Utility Score, a Novel Metric for Cost-Sensitive Classifier Evaluation," Pattern Recognition Letters vol. 27, no. 13, pp. 1472-1477, Oct. 2006.