NMC: nearest matrix classification – A new combination model for pruning One-vs-One ensembles by transforming the aggregation problem

Information Fusion

Volumn 36, Issue , 2017, Pages 26-51

  Galar, Mikel ^a   Fernández, Alberto ^b   Barrenechea, Edurne ^a   Bustince, Humberto ^a   Herrera, Francisco ^c,d  

^a PUBLIC UNIVERSITY OF NAVARRE   (Spain)

^b UNIVERSITY OF JAÉN   (Spain)

^c UNIVERSITY OF GRANADA   (Spain)

^d KING ABDULAZIZ UNIVERSITY   (Saudi Arabia)

Author keywords

Classifier selection; Decomposition strategies; Ensemble pruning; Multi class classification; One vs One

Indexed keywords

CLASSIFICATION (OF INFORMATION); LINEAR TRANSFORMATIONS; MACHINE LEARNING;

CLASSIFIER SELECTION; DECOMPOSITION STRATEGY; ENSEMBLE PRUNING; MULTI-CLASS CLASSIFICATION; ONE VS ONES;

MATRIX ALGEBRA;

EID: 84994890364     PISSN: 15662535     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.inffus.2016.11.004     Document Type: Article

References (66)

1. [1] Sanz, J.A., Galar, M., Jurío, A., Brugos, A., Pagola, M., Bustince, H., Medical diagnosis of cardiovascular diseases using an interval-valued fuzzy rule-based classification system. Appl. Soft Comput. 20 (2014), 103–111.
2. [2] Galar, M., Derrac, J., Peralta, D., Triguero, I., Paternain, D., Lopez-Molina, C., García, S., Ben’ıtez, J.M., Pagola, M., Barrenechea, E., Bustince, H., Herrera, F., A survey of fingerprint classification part II: experimental analysis and ensemble proposal. Knowl.-based Syst. 81 (2015), 98–116.
3. [3] Bolón-Canedo, V., no, N.S.-M., Alonso-Betanzos, A., An ensemble of filters and classifiers for microarray data classification. Pattern Recogn. 45:1 (2012), 531–539.
4. [4] Katakis, I., Tsoumakas, G., Vlahavas, I., Tracking recurring contexts using ensemble classifiers: an application to email filtering. Knowl. Inf. Syst. 22 (2010), 371–391.
5. [5] Vapnik, V., Statistical Learning Theory. 1998, New York: Wiley.
6. [6] Lorena, A.C., Carvalho, A.C., Gama, J.M., A review on the combination of binary classifiers in multiclass problems. Artif. Intell. Rev. 30:1–4 (2008), 19–37.
7. [7] Kuncheva, L.I., Combining Pattern Classifiers: Methods and Algorithms. Wiley-Interscience, 2004.
8. [8] Knerr, S., Personnaz, L., Dreyfus, G., Single-layer learning revisited: a stepwise procedure for building and training a neural network. Soulié, F.F., Hérault, J., (eds.) Neurocomputing: Algorithms, Architectures and Applications, Vol. F68 of NATO ASI Series, 1990, Springer-Verlag, 41–50.
9. [9] Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H., The weka data mining software: an update. SIGKDD Explor. Newsl. 11 (2009), 10–18.
10. [10] Chang, C.-C., Lin, C.-J., LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol. 2 (2011), 27:1–27:27 Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
11. [11] Alcalá-Fdez, J., Sánchez, L., García, S., del Jesus, M.J., Ventura, S., Garrell, J.M., Otero, J., Romero, C., Bacardit, J., Rivas, V.M., Fernández, J., Herrera, F., KEEL: A software tool to assess evolutionary algorithms for data mining problems. Soft Comput. 13:3 (2009), 307–318.
12. [12] Dietterich, T.G., Bakiri, G., Solving multiclass learning problems via error-correcting output codes. J. Artif. Intell. Res. 2 (1995), 263–286.
13. [13] Allwein, E.L., Schapire, R.E., Singer, Y., Reducing multiclass to binary: a unifying approach for margin classifiers. J. Mach. Learn. Res. 1 (2000), 113–141.
14. [14] Hüllermeier, E., Vanderlooy, S., Combining predictions in pairwise classification: an optimal adaptive voting strategy and its relation to weighted voting. Pattern Recogn. 43:1 (2010), 128–142.
15. [15] Elkano, M., Galar, M., Sanz, J., Fernandez, A., Barrenechea, E., Herrera, F., Bustince, H., Enhancing multi-class classification in farc-hd fuzzy classifier: on the synergy between n-dimensional overlap functions and decomposition strategies. IEEE Trans. Fuzzy Syst. 23:5 (2015), 1562–1580.
16. [16] Fürnkranz, J., Round robin classification. J. Mach. Learn. Res. 2 (2002), 721–747.
17. [17] Fürnkranz, J., Round robin ensembles. Intell. Data Anal. 7:5 (2003), 385–403.
18. [18] Galar, M., Fernández, A., Barrenechea, E., Bustince, H., Herrera, F., An overview of ensemble methods for binary classifiers in multi-class problems: experimental study on one-vs-one and one-vs-all schemes. Pattern Recogn. 44:8 (2011), 1761–1776.
19. [19] Sáez, J.A., Galar, M., Luengo, J., Herrera, F., Analyzing the presence of noise in multi-class problems: alleviating its influence with the one-vs-one decomposition. Knowl. Inf. Syst. 38:1 (2014), 179–206.
20. [20] Fei, B., Liu, J., Binary tree of SVM: a new fast multiclass training and classification algorithm. IEEE Trans. Neural Networks 17:3 (2006), 696–704.
21. [21] Liu, B., Hao, Z., Tsang, E.C.C., Nesting one-against-one algorithm based on SVMs for pattern classification. IEEE Trans. Neural Networks 19:12 (2008), 2044–2052.
22. [22] Platt, J.C., Cristianini, N., Shawe-taylor, J., Large margin DAGs for multiclass classification. Advances in Neural Information Processing Systems, 2000, MIT Press, 547–553.
23. [23] Wu, T.F., Lin, C.J., Weng, R.C., Probability estimates for multi-class classification by pairwise coupling. J. Mach. Learn. Res. 5 (2004), 975–1005.
24. [24] Galar, M., Fernández, A., Barrenechea, E., Bustince, H., Herrera, F., Dynamic classifier selection for one-vs-one strategy: avoiding non-competent classifiers. Pattern Recogn. 46:12 (2013), 3412–3424.
25. [25] Galar, M., Fernández, A., Barrenechea, E., Herrera, F., DRCW-OVO: distance-based relative competence weighting combination for one-vs-one strategy in multi-class problems. Pattern Recogn. 48:1 (2015), 28–42.
26. [26] Wolpert, D.H., Stacked generalization. Neural Netw. 5:2 (1992), 241–260.
27. [27] Kang, S., Cho, S., Optimal construction of one-against-one classifier based on meta-learning. Neurocomputing 167 (2015), 459–466.
28. [28] Menahem, E., Rokach, L., Elovici, Y., Troika - an improved stacking schema for classification tasks. Inform. Sci. 179:24 (2009), 4097–4122.
29. [29] Aha, D.W., Kibler, D., Albert, M.K., Instance-based learning algorithms. Mach. Learn. 6 (1991), 37–66.
30. [30] García, S., Derrac, J., Cano, J., Herrera, F., Prototype selection for nearest neighbor classification: taxonomy and empirical study. IEEE Trans. Pattern Anal. Machine Intell. 34:3 (2012), 417–435.
31. [31] noz, G.M.-M., Hernández-Lobato, D., Suárez, A., An analysis of ensemble pruning techniques based on ordered aggregation. IEEE Trans. Pattern Anal. Machine Intell. 31:2 (2009), 245–259.
32. [32] Guyon, I., Elisseeff, A., An introduction to variable and feature selection. J. Mach. Learn. Res. 3 (2003), 1157–1182.
33. [33] Derrac, J., García, S., Herrera, F., Ifs-coco:instance and feature selection based on cooperative coevolution with nearest neighbor rule. Pattern Recogn. 43:6 (2010), 2082–2105.
34. [34] García-Pedrajas, N., de Haro-García, A., Pérez-Rodríguez, J., A scalable approach to simultaneous evolutionary instance and feature selection. Inform. Sci. 228 (2013), 150–174.
35. [35] García-Pedrajas, N., del Castillo, J.A.R., Ortiz-Boyer, D., A cooperative coevolutionary algorithm for instance selection for instance-based learning. Mach. Learn. 78:3 (2010), 381–420.
36. [36] Alcalá-Fdez, J., Fernández, A., Luengo, J., Derrac, J., García, S., Sánchez, L., Herrera, F., KEEL data-mining software tool: data set repository, integration of algorithms and experimental analysis framework. J. Mult.-Valued Logic Soft Comput. 17 (2011), 255–287.
37. [37] Quinlan, J.R., C4.5: programs for Machine Learning. 1st edition, 1993, San Mateo-California: Morgan Kaufmann Publishers.
38. [38] Rokach, L., Decision forest: twenty years of research. Inform. Fusion 27 (2016), 111–125.
39. [39] Cohen, W.W., Fast effective rule induction. Proceedings of Twelfth International Conference Machine Learning, ICML, 1995, 1–10.
40. [40] Demšar, J., Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7 (2006), 1–30.
41. [41] García, S., Herrera, F., An extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons. J. Mach. Learn. Res. 9 (2008), 2677–2694.
42. [42] Fernández, A., Calderón, M., Barrenechea, E., Bustince, H., Herrera, F., Solving mult-class problems with linguistic fuzzy rule based classification systems based on pairwise learning and preference relations. Fuzzy Sets Syst. 161:23 (2010), 3064–3080.
43. [43] Galar, M., Fernández, A., Barrenechea, E., Bustince, H., Herrera, F., Aggregation schemes for binarization techniques. methods’ description, 2011, Research Group on Soft Computing and Intelligent Information Systems Tech. rep. http://sci2s.ugr.es/ovo-ova/AggregationMethodsDescription.pdf.
44. [44] Kuncheva, L.I., Bezdek, J.C., Duin, R.P.W., Decision templates for multiple classifier fusion: an experimental comparison. Pattern Recogn. 34:2 (2001), 299–314.
45. [45] Margineantu, D.D., Dietterich, T.G., Pruning adaptive boosting. Proceedings of the Fourteenth International Conference on Machine Learning, ICML ’97, 1997, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 211–218.
46. [46] Omari, A., Figueiras-Vidal, A.R., Post-aggregation of classifier ensembles. Inform. Fusion 26 (2015), 96–102.
47. [47] Kull, M., Flach, P.A., Reliability maps: a tool to enhance probability estimates and improve classification accuracy. Calders, T., Esposito, F., Hüllermeier, E., Meo, R., (eds.) Machine Learning and Knowledge Discovery in Databases, Vol. 8725 of LNSC, 2014, Springer, 18–33.
48. [48] Garcia-Pedrajas, N., Ortiz-Boyer, D., Improving multiclass pattern recognition by the combination of two strategies. IEEE Trans. Pattern Anal. Machine Intell. 28:6 (2006), 1001–1006.
49. [49] Arruti, A., Mendialdua, I., Sierra, B., Lazkano, E., Jauregi, E., New one versus all-one method: Nov@. Expert Syst. Appl. 41:14 (2014), 6251–6260.
50. [50] Park, S.-H., Fürnkranz, J., Efficient prediction algorithms for binary decomposition techniques. Data Min. Knowl. Discov. 24:1 (2012), 40–77.
51. [51] Rocha, A., Goldenstein, S.K., Multiclass from binary:expanding one-versus-all, one-versus-one and ecoc-based approaches. IEEE Trans. Neural Netw. Learn. Syst. 25:2 (2014), 289–302.
52. [52] Ou, G., Murphey, Y.L., Multi-class pattern classification using neural networks. Pattern Recogn. 40:1 (2007), 4–18.
53. [53] nés, E.M., Barranquero, J., Díez, J., del Coz, J.J., Enhancing directed binary trees for multi-class classification. Inform. Sci. 223 (2013), 42–55.
54. [54] Eshelman, J.L., The CHC adaptive search algorithm: how to have safe search when engaging in nontraditional genetic recombination. Rawlins, G.J.E., (eds.) Foundations of Genetic Algorithms, San Francisco, 1991, CA: Morgan Kaufmann, 265–283.
55. [55] Galar, M., Fernández, A., Barrenechea, E., Herrera, F., EUSBoost: enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling. Pattern Recogn. 46:12 (2013), 3460–3471.
56. [56] Galar, M., Fernández, A., Barrenechea, E., Herrera, F., Empowering difficult classes with a similarity-based aggregation in multi-class classification problems. Inform. Sci. 264 (2014), 135–157.
57. [57] Hsu, C.W., Lin, C.J., A comparison of methods for multiclass support vector machines. IEEE Trans. Neural Netw. 13:2 (2002), 415–425.
58. [58] Platt, J.C., Smola, A., Bartlett, P., Schölkopf, B., Schuurmans, D., Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. Advances in Large Margin Classifiers, 2000, MIT Press Cambridge, MA., 61–74.
59. [59] Moreno-Torres, J.G., Saez, J.A., Herrera, F., Study on the impact of partition-induced dataset shift on k-fold cross-validation. IEEE Trans. Neural Netw. Learn. Syst. 23:8 (2012), 1304–1312.
60. [60] Hodges, J.L., Lehmann, E.L., Rank methods for combination of independent experiments in analysis of variance. Ann. Math. Statist 33 (1962), 482–497.
61. [61] Holm, S., A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6 (1979), 65–70.
62. [62] Wilcoxon, F., Individual comparisons by ranking methods. Biometrics Bull. 1:6 (1945), 80–83.
63. [63] Galar, M., Fernández, A., Barrenechea, E., Bustince, H., Herrera, F., A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE Trans. Syst. Man Cybern Part C 42:4 (2012), 463–484.
64. [64] Fernández, A., López, V., Galar, M., del Jesus, M.J., Herrera, F., Analysing the classification of imbalanced data-sets with multiple classes: binarization techniques and ad-hoc approaches. Knowl.-based Syst. 42 (2013), 97–110.
65. [65] Ho, T.K., Basu, M., Measuring the complexity of classifier problems. Proceedings of 15th International Conference on Pattern Recognition (ICPR), Vol. 2, Barcelona, Spain, 2000, 43–47.
66. [66] Sáez, J.A., Luengo, J., Herrera, F., Predicting noise filtering efficacy with data complexity measures for nearest neighbor classification. Pattern Recogn. 46:1 (2013), 355–364.