A framework for monitoring classifiers' performance: When and why failure occurs?

Knowledge and Information Systems

Volumn 18, Issue 1, 2009, Pages 83-108

  Cieslak, David A ^a   Chawla, Nitesh V ^a  

^a University of Notre Dame   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

LEARNING ALGORITHMS;

BREAK-POINTS; CLASSIFIER PERFORMANCE; COMPREHENSIVE EVALUATION; COVARIATE SHIFTS; DATA CHANGE; DATA DISTRIBUTION; DATA VARIANCE; SAMPLE SELECTION BIAS;

CLASSIFICATION (OF INFORMATION);

EID: 58549104329     PISSN: 02191377     EISSN: 02193116     Source Type: Journal    
DOI: 10.1007/s10115-008-0139-1     Document Type: Article

References (27)

1. SVM light Support Vector Machine. http://www.cs.cornell.edu/People/tj/ svm_light/
2. Basu A, Harris IR, Basu S (1997) Minimum distance estimation: The approach using density-based distances. Handb Stat 15:21-48
3. Caruana R, Niculescu-Mizil A (2004) Data mining in metric space: An empirical analysis of suppervised learning performance criteria. In: Proceedings of the Tenth International Conference on Knowledge Discovery and Data Mining (KDD'04), pp 69-78
4. Chawla NV, Karakoulas G (2005) Learning from labeled and unlabeled data: an empirical study across techniques and domains. JAIR 23:331-366
5. Fan W, Davidson I (2006) ReverseTesting: An efficient framework to select amongst classifiers under sample selection bias. In: Proceedings of KDD
6. Fan W, Davidson I., Zadrozny B., Yu P. (2005) An improved categorization of classifier's sensitivity on sample selection bias. In: 5th IEEE International Conference on Data Mining
7. Gibbons JD (1985) Nonparametric statistical inference, 2nd edn. M. Dekker, New York
8. Groot P, ten Teije A, van Harmelen F (2004) A quantitative analysis of the robustness of knowledge-based systems through degradation studies. Knowl Inform Syst 7(2):224-245
9. Hall L, Mohney B, Kier L (1991) The electrotopological state: Structure information at the atomic level for molecular graphs. J Chem Inform Comput Sci 31(76)
10. Heckman JJ Sample selection bias as a specification error. Econometrica 47(1):153-161
11. Kailath T (1967) The divergence and bhattacharyya distance measures in signal selection. IEEE Trans Commun 15(1):52-60
12. Kolmogorov AN (1933) On the empirical determination of a distribution function. (Italian) Giornale dell'Instituto Italiano degli Attuari 4:83-91
13. Kubat M, Holte R, Matwin S (1998) Machine learning for the detection of oil spills in satellite radar images. Mach Learn 30:195-215
14. Kukar M (2006) Quality assessment of individual classifications in machine learning and data mining. Knowl Inform Sys 9(3):364-384
15. Legendre P, Gallagher ED (2001) Ecologically meaningful transformations for ordination of species data. Oecologia 129:271-280
16. Lindman HR (1974) Analysis of variance in complex experimental designs. W. H. Freeman, San Francisco
17. Little R, Rubin D (1987) Statistical analysis with missing data. Wiley, New York
18. Newman D, Hettich S, Blake C, Merz C (1998) UCI repository of machine learning databases
19. Provost F, Domingos P (2003) Tree induction for probability-based ranking. Mach Learn 52(3):199-215
20. Rao C (1995) A review of canonical coordinates and an alternative to corresponence analysis using hellinger distance. Questiio (Quaderns d'Estadistica i Investigacio Operativa) 19:23-63
21. Shawe-Taylor J, Bartlett P, Williamson R, Anthony M (1996) A framework for structural risk minimisation. In: Proceedings of the 9th Annual Conference on Computational Learning Theory
22. Smirnov N (1939) On the estimation of the discrepancy between empirical curves of distribution for two independent samples. (Russ) Bull Mosc Univ 2:3-16
23. Vapnik V (1996) The nature of statistical learning. Springer, New York
24. Woods K, Doss C, Bowyer K, Solka J, Priebe C, Kegelmeyer WP (1993) Comparative evaluation of pattern recognition techniques for detection of microcalcifications in mammography. IJPRAI 7(6):1417-1436
25. Wu X, Kumar V, Quinlan JR, Ghosh J, Yang Q, Motoda H, McLachlan GJ, Ng A, Liu B, Yu PS, Zhou Z-H, Steinbach M, Hand DJ, Steinberg D (2008) Top 10 algorithms in data mining. Knowl Inform Syst 14(1):1-37
26. Yamanishi K, ichi Takeuchi J, Williams GJ, Milne P (2004) On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms. In: Knowledge Discovery and Data Mining, pp 275-300
27. Zadrozny B (2004) Learning and evaluating under sample selection bias. In: Proceedings of the 21st International Conference on Machine Learning