A study on combining dynamic selection and data preprocessing for imbalance learning

Neurocomputing

Volumn 286, Issue , 2018, Pages 179-192

  Roy, Anandarup ^a   Cruz, Rafael M O ^a   Sabourin, Robert ^a   Cavalcanti, George D C ^b  

^a ÉCOLE DE TECHNOLOGIE SUPÉRIEURE   (Canada)

^b FEDERAL UNIVERSITY OF PERNAMBUCO   (Brazil)

Author keywords

Dynamic ensemble selection; Ensemble of classifiers; Imbalanced learning; Multi class imbalance; Preprocessing; SMOTE

Indexed keywords

COMPUTER APPLICATIONS; NEURAL NETWORKS;

CLASS IMBALANCE; DYNAMIC ENSEMBLE SELECTIONS; ENSEMBLE OF CLASSIFIERS; IMBALANCED LEARNING; PREPROCESSING; SMOTE;

CLASSIFICATION (OF INFORMATION);

ARTICLE; CLASSIFIER; CONTROLLED STUDY; DATA PROCESSING; DYNAMIC CLASSIFIER SELECTION; DYNAMIC ENSEMBLE SELECTION; DYNAMIC SELECTION; IMBALANCE LEARNING; INTERMETHOD COMPARISON; LEARNING; MULTICLASS IMBALANCED CLASSIFICATION; PRIORITY JOURNAL; RANDOM BALANCE; RANKED MINORITY OVER SAMPLING; SYNTHETIC MINORITY OVER SAMPLING TECHNIQUE;

EID: 85042400496     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2018.01.060     Document Type: Article

References (74)

1. He, H., Garcia, E., Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 21:9 (2009), 1263–1284.
2. Akbani, R., Kwek, S., Japkowicz, N., Applying support vector machines to imbalanced datasets. Proceedings of the European Conference on Machine Learning (ECML), Pisa, Italy, 2004, Springer Berlin Heidelberg, Berlin, Heidelberg, 39–50.
3. Wei, W., Li, J., Cao, L., Ou, Y., Chen, J., Effective detection of sophisticated online banking fraud on extremely imbalanced data. World Wide Web 16:4 (2013), 449–475.
4. Zheng, Z., Wu, X., Srihari, R., Feature selection for text categorization on imbalanced data. SIGKDD Explor. Newsl. 6:1 (2004), 80–89.
5. Piras, L., Giacinto, G., Synthetic pattern generation for imbalanced learning in image retrieval. Pattern Recognit. Lett. 33:16 (2012), 2198–2205.
6. Thammasiri, D., Delen, D., Meesad, P., Kasap, N., A critical assessment of imbalanced class distribution problem: the case of predicting freshmen student attrition. Expert Syst. Appl. 41:2 (2014), 321–330.
7. Wu, K., Edwards, A., Fan, W., Gao, J., Zhang, K., Classifying imbalanced data streams via dynamic feature group weighting with importance sampling. Proceedings of the 2014 SIAM International Conference on Data Mining, Philadelphia, Pennsylvania, USA, 2014, 722–730.
8. Wang, S., Minku, L.L., Yao, X., Dealing with multiple classes in online class imbalance learning. Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, (IJCAI), New York, NY, USA, 2016, 2118–2124.
9. Xioufis, E.S., Spiliopoulou, M., Tsoumakas, G., Vlahavas, I.P., Dealing with concept drift and class imbalance in multi-label stream classification. Proceedings of the 22nd International Joint Conference on Artificial Intelligence, (IJCAI), Barcelona, Catalonia, Spain, 2011, 1583–1588.
10. Chen, S., He, H., SERA: selectively recursive approach towards nonstationary imbalanced stream data mining. Proceedings of the International Joint Conference on Neural Networks, (IJCNN), Atlanta, Georgia, USA, 2009, 522–529.
11. He, H., Ma, Y., Imbalanced Learning: Foundations, Algorithms, and Applications. first edn., 2013, Wiley-IEEE Press.
12. Branco, P., Torgo, L., Ribeiro, R.P., A survey of predictive modeling on imbalanced domains. ACM Comput. Surv. 49:2 (2016), 31:1–31:50.
13. Galar, M., Fernández, A., Barrenechea, E., Bustince, H., Herrera, F., A review on ensembles for the class imbalance problem: Bagging-, boosting-, and hybrid-based approaches. Trans. Sys. Man Cyber Part C 42:4 (2012), 463–484.
14. Krawczyk, B., Learning from imbalanced data: open challenges and future directions. Prog. Artif. Intell. 5:4 (2016), 221–232.
15. Prati, R.C., Batista, G.E.A.P.A., Silva, D.F., Class imbalance revisited: a new experimental setup to assess the performance of treatment methods. Knowl. Inf. Syst. 45:1 (2015), 247–270.
16. García, V., Mollineda, R.A., Sánchez, J.S., On the k-nn performance in a challenging scenario of imbalance and overlapping. Pattern Anal. Appl. 11:3–4 (2008), 269–280.
17. Hu, J., Li, Y., Yan, W.-X., Yang, J.-Y., Shen, H.-B., Yu, D.-J., Knn-based dynamic query-driven sample rescaling strategy for class imbalance learning. Neurocomputing 191 (2016), 363–373 https://doi.org/10.1016/j.neucom.2016.01.043.
18. Batuwita, R., Palade, V., FSVM-CIL: fuzzy support vector machines for class imbalance learning. IEEE Trans. Fuzzy Syst. 18:3 (2010), 558–571, 10.1109/TFUZZ.2010.2042721.
19. Cieslak, D.A., Hoens, T.R., Chawla, N.V., Kegelmeyer, W.P., Hellinger distance decision trees are robust and skew-insensitive. Data Mining Knowl. Discov. 24:1 (2012), 136–158.
20. García, V., Sánchez, J.S., Mollineda, R.A., On the effectiveness of preprocessing methods when dealing with different levels of class imbalance. Know. Based Syst. 25:1 (2012), 13–21.
21. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P., Smote: Synthetic minority over-sampling technique. J. Artif. Int. Res. 16:1 (2002), 321–357.
22. Batista, G.E.A.P.A., Prati, R.C., Monard, M.C., A study of the behavior of several methods for balancing machine learning training data. SIGKDD Explor. Newsl. 6:1 (2004), 20–29.
23. Castro, C.L., Braga, A.P., Novel cost-sensitive approach to improve the multilayer perceptron performance on imbalanced data. IEEE Trans. Neural Netw. Learn. Syst. 24:6 (2013), 888–899.
24. Alejo, R., García, V., Sotoca, J.M., Mollineda, R.A., Sánchez, J.S., Improving the performance of the rbf neural networks trained with imbalanced samples. Proceedings of the International Work-Conference on Artificial Neural Networks, (IWANN), 2007, Springer Berlin, Heidelberg, San Sebastián, Spain, 162–169, 10.1007/978-3-540-73007-1_20.
25. Sun, Y., Kamel, M.S., Wong, A.K., Wang, Y., Cost-sensitive boosting for classification of imbalanced data. Pattern Recognit. 40:12 (2007), 3358–3378.
26. Kuncheva, L.I., Combining Pattern Classifiers: Methods and Algorithms. second edn., 2014, Wiley Publishing.
27. Chawla, N.V., Lazarevic, A., Hall, L.O., Bowyer, K.W., SMOTEBoost: improving prediction of the minority class in boosting. Proceedings of the European Conference on Principles and Practice of Knowledge Discovery in Databases, 2003, Springer, Berlin, Heidelberg, 107–119.
28. Freund, Y., Schapire, R.E., Experiments with a new boosting algorithm. Proceedings of the Thirteenth International Conference on Machine Learning (ICML), 1996, Morgan Kaufmann, 148–156.
29. Seiffert, C., Khoshgoftaar, T.M., Van Hulse, J., Napolitano, A., RUSBoost: a hybrid approach to alleviating class imbalance. Trans. Sys. Man Cyber. Part A 40:1 (2010), 185–197.
30. Chen, S., He, H., Garcia, E.A., RAMOBoost: ranked minority oversampling in boosting. Trans. Neur. Netw. 21:10 (2010), 1624–1642.
31. Breiman, L., Bagging predictors. Mach. Learn. 24:2 (1996), 123–140.
32. Wang, S., Yao, X., Diversity analysis on imbalanced data sets by using ensemble models. Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining, 2009, 324–331.
33. Díez-Pastor, J.F., Rodríguez, J.J., García-Osorio, C.I., Kuncheva, L.I., Diversity techniques improve the performance of the best imbalance learning ensembles. Inf. Sci. 325 (2015), 98–117.
34. Błaszczyński, J., Stefanowski, J., Neighbourhood sampling in bagging for imbalanced data. Neurocomputing 150, Part B (2015), 529–542.
35. Roy, A., Cruz, R.M.O., Sabourin, R., Cavalcanti, G.D.C., Meta-learning recommendation of default size of classifier pool for meta-des. Neurocomputing 216 (2016), 351–362.
36. Britto, A.S., Sabourin, R., Oliveira, L.E.S., Dynamic selection of classifiers - a comprehensive review. Pattern Recognit. 47:11 (2014), 3665–3680.
37. Cruz, R.M., Sabourin, R., Cavalcanti, G.D., Dynamic classifier selection: Recent advances and perspectives. Inf. Fus. 41 (2018), 195–216.
38. Woods, K., Kegelmeyer, W.P., Bowyer, K., Combination of multiple classifiers using local accuracy estimates. IEEE Trans. Pattern Anal. Mach. Intell. 19:4 (1997), 405–410.
39. Ko, A., Sabourin, R., Britto, J.A., From dynamic classifier selection to dynamic ensemble selection. Pattern Recognit. 41:5 (2008), 1718–1731.
40. Cavalin, P.R., Sabourin, R., Suen, C.Y., Dynamic selection approaches for multiple classifier systems. Neural Comput. Appl. 22:3–4 (2013), 673–688.
41. Cruz, R.M.O., Sabourin, R., Cavalcanti, G.D.C., Ren, T.I., META-DES: a dynamic ensemble selection framework using meta-learning. Pattern Recognit. 48:5 (2015), 1925–1935.
42. Xiao, J., Xie, L., He, C., Jiang, X., Dynamic classifier ensemble model for customer classification with imbalanced class distribution. Expert Syst. Appl. 39:3 (2012), 3668–3675.
43. Sabourin, M., Mitiche, A., Thomas, D., Nagy, G., Classifier combination for hand-printed digit recognition. Proceedings of the 2nd International Conference on Document Analysis and Recognition, 1993, 163–166.
44. Cavalin, P.R., Sabourin, R., Suen, C.Y., Logid: An adaptive framework combining local and global incremental learning for dynamic selection of ensembles of HMMs. Pattern Recognit. 45:9 (2012), 3544–3556.
45. Cruz, R.M.O., Sabourin, R., Cavalcanti, G.D.C., META-DES.H: A dynamic ensemble selection technique using meta-learning and a dynamic weighting approach. Proceedings of the International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland, 2015, 1–8.
46. Roy, A., Cruz, R.M.O., Sabourin, R., Cavalcanti, G.D.C., Meta-regression based pool size prediction scheme for dynamic selection of classifiers. Proceedings of the 23rd International Conference on Pattern Recognition (ICPR), 2016, IEEE, 211–216.
47. Kuncheva, L.I., A theoretical study on six classifier fusion strategies. IEEE Trans. Pattern Anal. Mach. Intell. 24:2 (2002), 281–286.
48. Cruz, R.M.O., Sabourin, R., Cavalcanti, G.D.C., META-DES.ORACLE: Meta-learning and feature selection for dynamic ensemble selection. Inf. Fusion 38 (2017), 84–103.
49. Cruz, R.M.O., Sabourin, R., Cavalcanti, G.D.C., A DEEP analysis of the META-DES framework for dynamic selection of ensemble of classifiers. CoRR, 2015 abs/1509.00825.
50. Cruz, R.M.O., Sabourin, R., Cavalcanti, G.D.C., Prototype selection for dynamic classifier and ensemble selection. Neural Comput. Appl. 29:2 (2018), 447–457.
51. Zadrozny, B., Langford, J., Abe, N., Cost-sensitive learning by cost-proportionate example weighting. Proceedings of the Third IEEE International Conference on Data Mining (ICDM), 2003, IEEE Computer Society, Washington, DC, USA, 435–442.
52. Barandela, R., Valdovinos, R., Sánchez, J., New applications of ensembles of classifiers. Pattern Anal. Appl. 6:3 (2003), 245–256.
53. Han, H., Wang, W.-Y., Mao, B.-H., Borderline-smote: A new over-sampling method in imbalanced data sets learning. Proceedings of the International Conference on Advances in Intelligent Computing - Volume Part I (ICIC), 2005, Springer-Verlag, Berlin, Heidelberg, 878–887.
54. Bunkhumpornpat, C., Sinapiromsaran, K., Lursinsap, C., Safe-level-smote: Safe-level-synthetic minority over-sampling technique for handling the class imbalanced problem. Proceedings of the Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining (PAKDD), 2009, Springer-Verlag, Berlin, Heidelberg, 475–482.
55. He, H., Bai, Y., Garcia, E., Li, S., Adasyn: adaptive synthetic sampling approach for imbalanced learning. Proceedings of the International Joint Conference on Neural Networks, 2008, 1322–1328, 10.1109/IJCNN.2008.4633969.
56. Díez-Pastor, J.F., Rodríguez, J.J., García-Osorio, C.I., Kuncheva, L.I., Random balance: ensembles of variable priors classifiers for imbalanced data. Knowl. Based Syst. 85 (2015), 96–111.
57. Witten, I.H., Frank, E., Hall, M.A., Pal, C.J., Data Mining: Practical Machine Learning Tools and Techniques. fourth, 2016, Morgan Kaufmann, Burlington, MA.
58. Rodriguez, J.J., Kuncheva, L.I., Alonso, C.J., Rotation forest: A new classifier ensemble method. IEEE Trans. Pattern Anal. Mach. Intell. 28:10 (2006), 1619–1630.
59. Quinlan, J.R., C4.5: Programs for Machine Learning. 1993, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
60. Chawla, N.V., C4.5 and imbalanced data sets: investigating the effect of sampling method, probabilistic estimate, and decision tree structure. Proceedings of the ICML Workshop on Class Imbalances, 2003.
61. Fawcett, T., An introduction to ROC analysis. Pattern Recognit. Lett. 27:8 (2006), 861–874.
62. Rijsbergen, C.J.V., Information Retrieval. second, 1979, Butterworth-Heinemann, Newton, MA, USA.
63. Kubat, M., Matwin, S., Addressing the curse of imbalanced training sets: one-sided selection. Proceedings of the 14th International Conference on Machine Learning (ICML), 1997, 179–186.
64. Alcalá-Fdez, J., Fernández, A., Luengo, J., Derrac, J., García, S., Sánchez, F., Herrera, L., Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework. J. Multiple Valued Logic Soft Comput. 17:2–3 (2011), 255–287.
65. Fernández, A., García, S., del Jesus, M.J., Herrera, F., A study of the behaviour of linguistic fuzzy rule based classification systems in the framework of imbalanced data-sets. Fuzzy Sets Syst. 159:18 (2008), 2378–2398.
66. Iman, R.L., Davenport, J.M., Approximations of the critical region of the fbietkan statistic. Commun. Stat. Theory Methods 9:6 (1980), 571–595.
67. García, S., Fernández, A., Luengo, J., Herrera, F., Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power. Inf. Sci. 180:10 (2010), 2044–2064.
68. Finner, H., On a monotonicity problem in step-down multiple test procedures. J. Am. Stat. Assoc. 88:423 (1993), 920–923.
69. Valentini, G., An experimental bias-variance analysis of svm ensembles based on resampling techniques. IEEE Trans. Sys. Man Cybern. Part B Cybern. 35:6 (2005), 1252–1271.
70. Abdi, L., Hashemi, S., To combat multi-class imbalanced problems by means of over-sampling techniques. IEEE Trans. Knowl. Data Eng. 28:1 (2016), 238–251.
71. Abdi, L., Hashemi, S., To combat multi-class imbalanced problems by means of over-sampling and boosting techniques. Soft Comput. 19:12 (2015), 3369–3385.
72. Fernández, A., López, V., Galar, M., Del Jesus, M.J., Herrera, F., Analysing the classification of imbalanced data-sets with multiple classes: Binarization techniques and ad-hoc approaches. Knowl. Based Syst. 42 (2013), 97–110.
73. Fernández-Navarro, F., Hervás-Martínez, C., Gutiérrez, P.A., A dynamic over-sampling procedure based on sensitivity for multi-class problems. Pattern Recognit. 44:8 (2011), 1821–1833.
74. Sun, Y., Kamel, M.S., Wang, Y., Boosting for learning multiple classes with imbalanced class distribution. Proceedings of the International Conference on Data Mining (ICDM), 2006, 592–602.