Analysis of preprocessing vs. cost-sensitive learning for imbalanced classification. Open problems on intrinsic data characteristics

Expert Systems with Applications

Volumn 39, Issue 7, 2012, Pages 6585-6608

  López, Victoria ^a   Fernández, Alberto ^b   Moreno Torres, Jose G ^a   Herrera, Francisco ^a  

^a UNIVERSITY OF GRANADA   (Spain)

^b UNIVERSITY OF JAÉN   (Spain)

Author keywords

Class overlap; Classification; Cost sensitive learning; Dataset shift; Imbalanced datasets; Preprocessing

Indexed keywords

CLASS OVERLAP; COST-SENSITIVE LEARNING; DATASET SHIFT; IMBALANCED DATA-SETS; PREPROCESSING;

COSTS; LEARNING ALGORITHMS;

CLASSIFICATION (OF INFORMATION);

EID: 84856964446     PISSN: 09574174     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.eswa.2011.12.043     Document Type: Article

References (86)

1. Akbani, R.; Kwek, S.; & Japkowicz, N. (2004). Applying support vector machines to imbalanced datasets. In ECML (pp. 39-50). (Pubitemid 41050079)
2. R. Alaiz-Rodríguez, A. Guerrero-Curieses, and J. Cid-Sueiro Improving classification under changes in class and within-class distributions IWANN'09: Proceedings of the 10th international work-conference on artificial neural networks 2009 Springer Berlin, Heidelberg 122 130
3. R. Alaiz-Rodríguez, and N. Japkowicz Assessing the impact of changing environments on classifier performance Canadian AI'08: Proceedings of the Canadian society for computational studies of intelligence, 21st conference on advances in artificial intelligence 2008 Springer-Verlag Berlin, Heidelberg 13 24 (Pubitemid 351777508)
4. J. Alcalá-Fdez, A. Fernández, J. Luengo, J. Derrac, S. García, and L. Sánchez KEEL data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework Journal of Multi-Valued Logic and Soft Computing 17 2011 255 287
5. J. Alcalá-Fdez, L. Sánchez, S. García, M.J. del Jesus, S. Ventura, and J.M. Garrell Keel: A software tool to assess evolutionary algorithms for data mining problems Soft Computing 13 2008 307 318
6. R. Barandela, J.S. Sánchez, V. García, and E. Rangel Strategies for learning in class imbalance problems Pattern Recognition 36 2003 849 851
7. G.E.A.P.A. Batista, R.C. Prati, and M.C. Monard A study of the behaviour of several methods for balancing machine learning training data SIGKDD Explorations 6 2004 20 29
8. P. Bermejo, J. Gámez, and J. Puerta Improving the performance of Naive Bayes multinomial in e-mail foldering by introducing distribution-based balance of datasets Expert Systems with Applications 38 2011 2072 2080
9. S. Bickel, M. Brückner, and T. Scheffer Discriminative learning under covariate shift Journal of Machine Learning Research 10 2009 2137 2155
10. Bradford, J. P.; Kunz, C.; Kohavi, R.; Brunk, C.; & Brodley, C. E. (1998). Pruning decision trees with misclassification costs. In ECML (pp. 131-136). (Pubitemid 128067177)
11. A.P. Bradley The use of the area under the roc curve in the evaluation of machine learning algorithms Pattern Recognition 30 1997 1145 1159 (Pubitemid 127406521)
12. Bunkhumpornpat, C.; Sinapiromsaran, K.; & Lursinsap, C. (2009). Safe-level-smote: Safe-level-synthetic minority over-sampling technique for handling the class imbalanced problem. In Proceedings of the 13th Pacific-Asia conference on advances in knowledge discovery and data mining PAKDD'09 (pp. 475-482).
13. J. Quiñonero Candela, M. Sugiyama, A. Schwaighofer, and N.D. Lawrence Dataset shift in machine learning 2009 The MIT Press
14. G.A. Carpenter, S. Grossberg, and D.B. Rosen Fuzzy art: Fast stable learning and categorization of analog patterns by an adaptive resonance system Neural Networks 4 1991 759 771
15. N.V. Chawla, K.W. Bowyer, L.O. Hall, and W.P. Kegelmeyer SMOTE: Synthetic minority over-sampling technique Journal of Artificial Intelligent Research 16 2002 321 357 (Pubitemid 43057176)
16. N.V. Chawla, D.A. Cieslak, L.O. Hall, and A. Joshi Automatically countering imbalance and its empirical relationship to cost Data Mining and Knowledge Discovery 17 2008 225 252
17. N.V. Chawla, N. Japkowicz, and A. Kotcz Editorial: Special issue on learning from imbalanced data sets SIGKDD Explorations 6 2004 1 6
18. M.-C. Chen, L.-S. Chen, C.-C. Hsu, and W.-R. Zeng An information granulation based data mining approach for classifying imbalanced data Information Sciences 178 2008 3214 3227
19. X. Chen, T. Fang, H. Huo, and D. Li Graph-based feature selection for object-oriented classification in vhr airborne imagery IEEE Transactions on Geoscience and Remote Sensing 49 2011 353 365
20. D.A. Cieslak, and N.V. Chawla A framework for monitoring classifiers performance: When and why failure occurs? Knowledge and Information Systems 18 2009 83 108
21. C. Cortes, and V. Vapnik Support vector networks Machine Learning 20 1995 273 297
22. T. Cover, and P. Hart Nearest neighbor pattern classification IEEE Transactions on Information Theory 13 1967 21 27
23. J. Demšar Statistical comparisons of classifiers over multiple data sets Journal of Machine Learning Research 7 2006 1 30 (Pubitemid 43022939)
24. Denil, M.; & Trappenberg, T. (2010). Overlap versus imbalance. In Canadian AI 2010, LNAI (Vol. 6085, pp. 220-231).
25. Domingos, P. (1999). Metacost: A general method for making classifiers cost-sensitive. In Proceedings of the 5th international conference on knowledge discovery and data mining (KDD'99) (pp. 155-164).
26. P. Ducange, B. Lazzerini, and F. Marcelloni Multi-objective genetic fuzzy classifiers for imbalanced and cost-sensitive datasets Soft Computing 14 2010 713 728
27. Elkan, C. (2001). The foundations of cost-sensitive learning. In Proceedings of the 17th IEEE international joint conference on artificial intelligence (IJCAI'01) (pp. 973-978).
28. A. Estabrooks, T. Jo, and N. Japkowicz A multiple resampling method for learning from imbalanced data sets Computational Intelligence 20 2004 18 36
29. A. Fernández, S. García, M.J. del Jesus, and F. Herrera A study of the behaviour of linguistic fuzzy rule based classification systems in the framework of imbalanced data-sets Fuzzy Sets and Systems 159 2008 2378 2398
30. A. Fernández, M.J. del Jesus, and F. Herrera Hierarchical fuzzy rule based classification systems with genetic rule selection for imbalanced data-sets International Journal of Approximate Reasoning 50 2009 561 577
31. A. Fernández, M.J. del Jesús, and F. Herrera On the 2-tuples based genetic tuning performance for fuzzy rule based classification systems in imbalanced data-sets Information Sciences 180 2010 1268 1291
32. M. Friedman The use of ranks to avoid the assumption of normality implicit in the analysis of variance Journal of the American Statistical Association 32 1937 675 701
33. J. Gama Functional trees Machine Learning 55 2004 219 250
34. S. García, A. Fernández, J. Luengo, and F. Herrera A study of statistical techniques and performance measures for genetics-based machine learning: Accuracy and interpretability Soft Computing 13 2009 959 977
35. S. García, A. Fernández, J. Luengo, and F. Herrera Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power Information Sciences 180 2010 2044 2064
36. S. García, and F. Herrera An extension on "statistical comparisons of classifiers over multiple data sets" for all pairwise comparisons Journal of Machine Learning Research 9 2008 2607 2624
37. V. García, R. Mollineda, and J.S. Sánchez On the k-NN performance in a challenging scenario of imbalance and overlapping Pattern Analysis Applications 11 2008 269 280
38. A. Globerson, C.H. Teo, A. Smola, and S. Roweis An adversarial view of covariate shift and a minimax approach J. Quiñonero Candela, M. Sugiyama, A. Schwaighofer, N.D. Lawrence, Dataset shift in machine learning 2009 The MIT Press 179 198
39. A. Gretton, A. Smola, J. Huang, M. Schmittfull, K. Borgwardt, and B. Schölkopf Covariate shift by kernel mean matching J. Quiñonero Candela, M. Sugiyama, A. Schwaighofer, N.D. Lawrence, Dataset shift in machine learning 2009 The MIT Press 131 160
40. Han, H.; Wang, W.; & Mao, B. (2005). Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning. In Proceedings of the international conference on intelligent computing (pp. 878-887). (Pubitemid 41491129)
41. D.J. Hand, and V. Vinciotti Choosing k for two-class nearest neighbour classifiers with unbalanced classes Pattern Recognition Letters 24 2003 1555 1562 (Pubitemid 36225860)
42. He, H.; Bai, Y.; Garcia, E.; & Li, S. (2008). ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In Proceedings of international joint conference on neural networks (pp. 1322-1328).
43. H. He, and E.A. Garcia Learning from imbalanced data IEEE Transactions on Knowledge and Data Engineering 21 2009 1263 1284
44. T. Ho, and M. Basu Complexity measures of supervised classification problems IEEE Transactions on Pattern Analysis and Machine Intelligence 24 2002 289 300 (Pubitemid 35289347)
45. J. Huang, and C.X. Ling Using auc and accuracy in evaluating learning algorithms - appendices IEEE Transactions on Knowledge and Data Engineering 2005 17
46. H. Ishibuchi, and T. Nakashima Effect of rule weights in fuzzy rule-based classification systems IEEE Transactions on Fuzzy Systems 9 2001 506 515 (Pubitemid 32935694)
47. H. Ishibuchi, and T. Yamamoto Rule weight specification in fuzzy rule-based classification systems IEEE Transactions on Fuzzy Systems 13 2005 428 435 (Pubitemid 41461439)
48. H. Ishibuchi, T. Yamamoto, and T. Nakashima Hybridization of fuzzy GBML approaches for pattern classification problems IEEE Transactions on System, Man and Cybernetics B 35 2005 359 365 (Pubitemid 40535918)
49. N. Japkowicz, and S. Stephen The class imbalance problem: A systematic study Intelligent Data Analysis Journal 6 2002 429 450
50. Kaburlasos, V. G. (2007). Granular enhancement of fuzzy art/som neural classifiers based on lattice theory. In Computational intelligence based on lattice theory (pp. 3-23).
51. Kubat, M.; & Matwin, S. (1997). Addressing the curse of imbalanced training sets: One-sided selection. In International conference on machine learning (pp. 179-186).
52. Laurikkala, J. (2001). Improving identification of difficult small classes by balancing class distribution. In Proceedings of the conference on AI in medicine in Europe: Artificial intelligence medicine (pp. 63-66). (Pubitemid 33301585)
53. Ling, C. X.; Yang, Q.; Wang, J.; & Zhang, S. (2004). Decision trees with minimal costs. In ICML.
54. López, V.; Fernández, A.; & Herrera, F. (2010). A first approach for cost-sensitive classification with linguistic genetic fuzzy systems in imbalanced data-sets. In 10th International conference on intelligent systems design and applications, ISDA 2010 (pp. 676-681).
55. J. Luengo, A. Fernández, S. García, and F. Herrera Addressing data complexity for imbalanced data sets: Analysis of SMOTE-based oversampling and evolutionary undersampling Soft Computing 15 2011 1909 1936
56. Martín-Félez, R.; & Mollineda, R. (2010). On the suitability of combining feature selection and resampling to manage data complexity. In CAEPIA 2009, LNAI (Vol. 5988, pp. 141-150).
57. Moreno-Torres, J. G.; & Herrera, F. (2010). A preliminary study on overlapping and data fracture in imbalanced domains by means of genetic programming-based feature extraction. In Proceedings of the 10th international conference on intelligent systems design and applications (ISDA 2010) (pp. 501-506).
58. Moreno-Torres, J. G.; Llorà, X.; Goldberg, D. E.; & Bhargava, R. (in press). Repairing fractures between data using genetic programming-based feature extraction: A case study in cancer diagnosis. Information Sciences, doi:10.1016/j.ins.2010.09.018.
59. A. Orriols-Puig, and E. Bernadó-Mansilla Evolutionary rule-based systems for imbalanced datasets Soft Computing 13 2009 213 225
60. Prati, R. C.; & Batista, G. E. A. P. A. (2004). Class imbalances versus class overlapping: An analysis of a learning system behavior. In Proceedings of Mexican international conference on artificial intelligence (MICAI) (pp. 312-321). (Pubitemid 38716795)
61. J. Quinlan C4.5: Programs for machine learning 1993 Morgan Kauffman
62. P. Riddle, R. Segal, and O. Etzioni Representation design and brute-force induction in a boeing manufacturing domain Applied Artificial Intelligence 8 1994 125 147
63. J. Shaffer Modified sequentially rejective multiple test procedures Journal of the American Statistical Association 81 1986 826 831
64. D. Sheskin Handbook of parametric and nonparametric statistical procedures 2006 Chapman & Hall
65. H. Shimodaira Improving predictive inference under covariate shift by weighting the log-likelihood function Journal of Statistical Planning and Inference 90 2000 227 244
66. M. Sikora Induction and pruning of classification rules for prediction of microseismic hazards in coal mines Expert Systems with Applications 38 2011 6748 6758
67. M. Sugiyama, M. Krauledat, and K.-R. Müller Covariate shift adaptation by importance weighted cross validation Journal of Machine Learning Research 8 2007 985 1005 (Pubitemid 46798411)
68. Y. Sun, M.S. Kamel, A.K.C. Wong, and Y. Wang Cost-sensitive boosting for classification of imbalanced data Pattern Recognition 40 2007 3358 3378 (Pubitemid 47223287)
69. Y. Sun, A.K.C. Wong, and M.S. Kamel Classification of imbalanced data: A review International Journal of Pattern Recognition and Artificial Intelligence 23 2009 687 719
70. K.M. Ting An instance-weighting method to induce cost-sensitive trees IEEE Transactions on Knowledge and Data Engineering 14 2002 659 665 (Pubitemid 34669622)
71. V. Vapnik Statistical learning theory 1998 Wiley New York, USA
72. V. Vapnik, and A. Lerner Pattern recognition using generalized portrait method Automation and Remote Control 24 1963
73. Veropoulos, K.; Campbell, C.; & Cristianini, N. (1999). Controlling the sensitivity of support vector machines. In Proceedings of the international joint conference on AI (pp. 55-60).
74. Villar, P.; Fernández, A.; & Herrera, F. (2010). A genetic algorithm for feature selection and granularity learning in fuzzy rule-based classification systems for highly imbalanced data-sets. In 13th International conference on information processing and management of uncertainty in knowledge-based systems (IPMU2010), CCIS (Vol. 80, pp. 741-750).
75. Villar, P.; Fernández, A.; Sánchez, A.; & Herrera, F. (2009). Un algoritmo genético para selección de características en sistemas de clasificación basados en reglas difusas para conjuntos de datos altamente no balanceados. In Actas de la XIII Conferencia de la Asociación Española para la Inteligencia Artificial (CAEPIA09) (pp. 519-528).
76. Wang, B.; & Japkowicz, N. (2004). Imbalanced data set learning with synthetic samples. In Proceedings of IRIS machine learning workshop.
77. Wang, K.; Zhou, S.; Fu, C. A.; Yu, J. X.; Jeffrey, F.; & Yu, X. (2003). Mining changes of classification by correspondence tracing. In Proceedings of the 2003 SIAM international conference on data mining (SDM 2003).
78. G.M. Weiss, and F.J. Provost Learning when training data are costly: The effect of class distribution on tree induction Journal of Artificial Intelligence Research 19 2003 315 354 (Pubitemid 41525924)
79. D.L. Wilson Asymptotic properties of nearest neighbor rules using edited data IEEE Transactions on Systems, Man and Cybernetics 2 1972 408 421
80. G. Wu, and E.Y. Chang Kba: Kernel boundary alignment considering imbalanced data distribution IEEE Transactions on Knowledge and Data Engineering 17 2005 786 795 (Pubitemid 40860458)
81. L. Xu, M.-Y. Chow, and L.S. Taylor Power distribution fault cause identification with imbalanced data using the data mining-based fuzzy classification e-algorithm IEEE Transactions on Power Systems 22 2007 164 171 (Pubitemid 46437725)
82. Q. Yang, and X. Wu 10 Challenging problems in data mining research International Journal of Information Technology and Decision Making 5 2006 597 604
83. Y. Yang, X. Wu, and X. Zhu Conceptual equivalence for contrast mining in classification learning Data and Knowledge Engineering 67 2008 413 429
84. Zadrozny, B.; & Elkan, C. (2001). Learning and making decisions when costs and probabilities are both unknown. In Proceedings of the 7th international conference on knowledge discovery and data mining (KDD'01) (pp. 204-213).
85. Zadrozny, B.; Langford, J.; & Abe, N. (2003). Cost-sensitive learning by cost-proportionate example weighting. In Proceedings of the 3rd IEEE international conference on data mining (ICDM'03) (pp. 435-442).
86. Z.-H. Zhou, and X.-Y. Liu Training cost-sensitive neural networks with methods addressing the class imbalance problem IEEE Transactions on Knowledge and Data Engineering 18 2006 63 77 (Pubitemid 43145089)