On the importance of the validation technique for classification with imbalanced datasets: Addressing covariate shift when data is skewed

Information Sciences

Volumn 257, Issue , 2014, Pages 1-13

  López, Victoria ^a   Fernández, Alberto ^b   Herrera, Francisco ^a  

^a UNIVERSITY OF GRANADA   (Spain)

^b UNIVERSITY OF JAÉN   (Spain)

Author keywords

Classification; Covariate shift; Dataset shift; Imbalanced dataset; Partitioning; Validation technique

Indexed keywords

COVARIATE SHIFTS; DATASET SHIFTS; IMBALANCED DATASET; PARTITIONING; VALIDATION TECHNIQUE;

STATISTICAL TESTS;

CLASSIFICATION (OF INFORMATION);

EID: 84888645340     PISSN: 00200255     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ins.2013.09.038     Document Type: Article

References (52)

1. D.W. Aha, D. Kibler, and M.K. Albert Instance-based learning algorithms Machine Learning 6 1991 37 66
2. R. Alaiz-Rodríguez, and N. Japkowicz Assessing the impact of changing environments on classifier performance Proceedings of the 21st Canadian Conference on Advances in Artificial Intelligence (CCAI'08) 2008 Springer-Verlag Berlin, Heidelberg
3. J. Alcalá-Fdez, A. Fernández, J. Luengo, J. Derrac, S. García, L. Sánchez, and F. Herrera KEEL data-mining software tool: data set repository, integration of algorithms and experimental analysis framework Journal of Multi-Valued Logic and Soft Computing 17 2-3 2011 255 287
4. J. Alcalá-Fdez, L. Sánchez, S. García, M.J. del Jesus, S. Ventura, J.M. Garrell, J. Otero, C. Romero, J. Bacardit, V.M. Rivas, J.C. Fernández, and F. Herrera KEEL: a software tool to assess evolutionary algorithms for data mining problems Soft Computing 13 2009 307 318
5. R. Barandela, J.S. Sánchez, V. García, and E. Rangel Strategies for learning in class imbalance problems Pattern Recognition 36 3 2003 849 851
6. G.E.A.P.A. Batista, R.C. Prati, and M.C. Monard A study of the behaviour of several methods for balancing machine learning training data SIGKDD Explorations 6 1 2004 20 29
7. A.P. Bradley The use of the area under the ROC curve in the evaluation of machine learning algorithms Pattern Recognition 30 7 1997 1145 1159
8. J.Q. Candela, M. Sugiyama, A. Schwaighofer, and N.D. Lawrence Dataset Shift in Machine Learning 2009 The MIT Press
9. J.R. Cano, F. Herrera, and M. Lozano Evolutionary stratified training set selection for extracting classification rules with trade off precision-interpretability Data and Knowledge Engineering 60 2007 90 108
10. N.V. Chawla, K.W. Bowyer, L.O. Hall, and W.P. Kegelmeyer SMOTE: synthetic minority over-sampling technique Journal of Artificial Intelligent Research 16 2002 321 357
11. N.V. Chawla, N. Japkowicz, and A. Kotcz Editorial: special issue on learning from imbalanced data sets SIGKDD Explorations 6 1 2004 1 6
12. Y. Chen, and J. Wang Support vector learning for fuzzy rule-based classification systems IEEE Transactions on Fuzzy Systems 11 6 2003 716 728
13. Z. Chi, H. Yan, T. Pham, Fuzzy Algorithms with Applications to Image Processing and Pattern Recognition, World Scientific, 1996.
14. O. Cordón, M.J. del Jesus, and F. Herrera A proposal on reasoning methods in fuzzy rule-based classification systems International Journal of Approximate Reasoning 20 1 1999 21 45
15. C. Cortes, and V. Vapnik Support vector networks Machine Learning 20 1995 273 297
16. T. Cover, and P. Hart Nearest neighbor pattern classification IEEE Transactions on Information Theory 13 1967 21 27
17. J. Demšar Statistical comparisons of classifiers over multiple data sets Journal of Machine Learning Research 7 2006 1 30
18. P. Domingos, Metacost: a general method for making classifiers cost-sensitive, in: Proceedings of the 5th International Conference on Knowledge Discovery and Data Mining (KDD'99), 1999.
19. S. García, A. Fernández, J. Luengo, and F. Herrera A study of statistical techniques and performance measures for genetics-based machine learning: accuracy and interpretability Soft Computing 13 10 2009 959 977
20. S. García, and F. Herrera An extension on "statistical comparisons of classifiers over multiple data sets" for all pairwise comparisons Journal of Machine Learning Research 9 2008 2607 2624
21. H. He, and E.A. Garcia Learning from imbalanced data IEEE Transactions on Knowledge and Data Engineering 21 9 2009 1263 1284
22. J. Huang, and C.X. Ling Using AUC and accuracy in evaluating learning algorithms IEEE Transactions on Knowledge and Data Engineering 17 3 2005 299 310
23. Y.-M. Huang, C.-M. Hung, and H.C. Jiau Evaluation of neural networks and data mining methods on a credit assessment task for class imbalance problem Nonlinear Analysis: Real World Applications 7 4 2006 720 747
24. H. Ishibuchi, T. Nakashima, and M. Nii Classification and Modeling with Linguistic Information Granules: Advanced Approaches to Linguistic Data Mining 2004 Springer-Verlag
25. H. Ishibuchi, and T. Yamamoto Rule weight specification in fuzzy rule-based classification systems IEEE Transactions on Fuzzy Systems 13 2005 428 435
26. N. Japkowicz, and S. Stephen The class imbalance problem: a systematic study Intelligent Data Analysis Journal 6 5 2002 429 450
27. T. Lane, C.E. Brodley, Approaches to online learning and concept drift for user identification in computer security, in: KDD, 1998.
28. V. López, A. Fernández, J.G. Moreno-Torres, and F. Herrera Analysis of preprocessing vs. cost-sensitive learning for imbalanced classification. Open problems on intrinsic data characteristics Expert Systems with Applications 39 7 2012 6585 6608
29. J.G. Moreno-Torres, F. Herrera, A preliminary study on overlapping and data fracture in imbalanced domains by means of genetic programming-based feature extraction, in: Proceedings of the 10th International Conference on Intelligent Systems Design and Applications (ISDA'10), 2010.
30. J.G. Moreno-Torres, T. Raeder, R. Aláiz-Rodríguez, N.V. Chawla, and F. Herrera A unifying view on dataset shift in classification Pattern Recognition 45 1 2012 521 530
31. J.G. Moreno-Torres, J.A. Sáez, and F. Herrera Study on the impact of partition-induced dataset shift on k-fold cross-validation IEEE Transactions on Neural Networks and Learning Systems 23 8 2012 1304 1313
32. A. Orriols-Puig, and E. Bernadó-Mansilla Evolutionary rule-based systems for imbalanced datasets Soft Computing 13 3 2009 213 225
33. A. Orriols-Puig, E. Bernadó-Mansilla, D.E. Goldberg, K. Sastry, and P.L. Lanzi Facetwise analysis of XCS for problems with class imbalances IEEE Transactions on Evolutionary Computation 13 2009 260 283
34. J. Quinlan C4.5: Programs for Machine Learning 1993 Morgan Kauffman
35. D. Sheskin Handbook of Parametric and Nonparametric Statistical Procedures 2006 Chapman & Hall/CRC
36. H. Shimodaira Improving predictive inference under covariate shift by weighting the log-likelihood function Journal of Statistical Planning and Inference 90 2 2000 227 244
37. A. Storkey When training and test sets are different: characterizing learning transfer J.Q. Candela, M. Sugiyama, A. Schwaighofer, N.D. Lawrence, Dataset Shift in Machine Learning 2009 MIT Press 3 28
38. Y. Sun, A.K.C. Wong, and M.S. Kamel Classification of imbalanced data: a review International Journal of Pattern Recognition and Artificial Intelligence 23 4 2009 687 719
39. Y. Tang, Y.-Q. Zhang, N.V. Chawla, and S. Kresser SVMs modeling for highly imbalanced classification IEEE Transactions on Systems, Man and Cybernetics, Part B 39 1 2009 281 288
40. K.M. Ting An instance-weighting method to induce cost-sensitive trees IEEE Transactions on Knowledge and Data Engineering 14 3 2002 659 665
41. M. Wasikowski, and X.-W. Chen Combating the small sample class imbalance problem using feature selection IEEE Transactions on Knowledge and Data Engineering 22 10 2010 1388 1400
42. G.M. Weiss Mining with rarity: a unifying framework SIGKDD Explorations 6 1 2004 7 19
43. G.M. Weiss, and F.J. Provost Learning when training data are costly: the effect of class distribution on tree induction Journal of Artificial Intelligence Research 19 2003 315 354
44. G.M. Weiss, and Y. Tian Maximizing classifier utility when there are data acquisition and modeling costs Data Mining and Knowledge Discovery 17 2 2008 253 282
45. G. Widmer, and M. Kubat Learning in the presence of concept drift and hidden contexts Machine Learning 23 1 1996 69 101
46. I.H. Witten, and E. Frank Data Mining: Practical Machine Learning Tools and Techniques second ed. 2005 Morgan Kaufmann San Mateo, CA
47. K. Yamazaki, M. Kawanabe, S. Watanabe, M. Sugiyama, and K.-R. Müller Asymptotic bayesian generalization error when training and test distributions are different Z. Ghahramani, ICML ACM International Conference Proceeding Series vol. 227 2007 ACM
48. Q. Yang, and X. Wu 10 Challenging problems in data mining research International Journal of Information Technology and Decision Making 5 4 2006 597 604
49. B. Zadrozny, C. Elkan, Learning and making decisions when costs and probabilities are both unknown, in: Proceedings of the 7th International Conference on Knowledge Discovery and Data Mining (KDD'01), 2001.
50. B. Zadrozny, J. Langford, N. Abe, Cost-sensitive learning by cost-proportionate example weighting, in: Proceedings of the 3rd IEEE International Conference on Data Mining (ICDM'03), 2003.
51. J.H. Zar Biostatistical Analysis 1999 Prentice Hall Upper Saddle River, New Jersey
52. X. Zeng, and T.R. Martinez Distribution-balanced stratified cross validation for accuracy estimation Journal of Experimental and Theoretical Artificial Intelligence 12 1 2000 1 12