An up-to-date comparison of state-of-the-art classification algorithms

Expert Systems with Applications

Volumn 82, Issue , 2017, Pages 128-150

  Zhang, Chongsheng ^a   Liu, Changchang ^a   Zhang, Xiangliang ^b   Almpanidis, George ^a  

^a HENAN UNIVERSITY   (China)

^b KING ABDULLAH UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Saudi Arabia)

Author keywords

Classification benchmarking; Classifier comparison; Classifier evaluation

Indexed keywords

DECISION TREES; EFFICIENCY; FORECASTING; LEARNING SYSTEMS; STOCHASTIC SYSTEMS; SUPPORT VECTOR MACHINES;

CLASSIFICATION ALGORITHM; CLASSIFIER EVALUATION; COMPARATIVE STUDIES; EXTREME LEARNING MACHINE; PREDICTION PERFORMANCE; SPARSE REPRESENTATION BASED CLASSIFICATIONS; STOCHASTIC GRADIENT BOOSTING; TRAINING AND TESTING;

CLASSIFICATION (OF INFORMATION);

EID: 85017304883     PISSN: 09574174     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.eswa.2017.04.003     Document Type: Article

References (86)

1. Alcalá-Fdez, J., Fernández, A., Luengo, J., Derrac, J., García, S., Sánchez, L., Herrera, F., Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework. Journal of Multiple-Valued Logic and Soft Computing 17:2–3 (2010), 255–287.
2. Alcalá-Fdez, J., Sánchez, L., García, S., del Jesus, M., Ventura, S., Garrell, J.,.. Herrera, F., Keel: A software tool to assess evolutionary algorithms for data mining problems. Soft Computing 13:3 (2009), 307–318.
3. Altman, N.S., An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician 46:3 (1992), 175–185.
4. Arauzo-Azofra, A., Aznarte, J.L., Benitez, J.M., Empirical study of feature selection methods based on individual feature evaluation for classification problems. Expert Systems with Applications 38:7 (2011), 8170–8177.
5. Ayaki, T., Yanagimoto, H., Yoshioka, M., Recommendation from access logs with ensemble learning. Artificial Life and Robotics, 2016, 1–5.
6. Baesens, B., Van Gestel, T., Viaene, S., Stepanova, M., Suykens, J., Vanthienen, J., Benchmarking state-of-the-art classification algorithms for credit scoring. Journal of the Operational Research Society 54:6 (2003), 627–635.
7. Ballings, M., den Poel, D.V., Hespeels, N., Gryp, R., Evaluating multiple classifiers for stock price direction prediction. Expert Systems with Applications 42:20 (2015), 7046–7056.
8. Bengio, Y., Learning deep architectures for ai. Foundation and Trends in Machine Learning 2:1 (2009), 1–127.
9. Blake, C. L., & Merz, C. J. (1998). UCI repository of machine learning databases.
10. Bradley, A.P., The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern Recognition 30:7 (1997), 1145–1159.
11. Breiman, L., Random forests. Mach. Learn. 45 (2001), 5–32.
12. Brown, I., Mues, C., An experimental comparison of classification algorithms for imbalanced credit scoring data sets. Expert Systems with Applications 39:3 (2012), 3446–3453.
13. Carletta, J., Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics 22:2 (1996), 249–254.
14. Caruana, R., Niculescu-Mizil, A., An empirical comparison of supervised learning algorithms. Proceedings of the 23rd international conference on machine learning, 2006, ACM, 161–168.
15. Chang, C.-C., Lin, C.-J., LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2 (2011), 27:1–27:27.
16. Chapelle, O., Chang, Y., Yahoo! learning to rank challenge overview. Chapelle, O., Chang, Y., Liu, T.-Y., (eds.) Yahoo! learning to rank challenge JMLR Proceedings, 14, 2011, JMLR.org, 1–24.
17. Chen, T., Guestrin, C., Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining KDD ‘16, 2016, 785–794.
18. Cortes, C., Vapnik, V., Support vector networks. Machine Learning 20 (1995), 273–297.
19. Cox, D.R., The regression analysis of binary sequences. Journal of the Royal Statistical Society. Series B (Methodological), 1958, 215–242.
20. Daniel, W.W., Applied nonparametric statistics. 2nd ed., 2000, Cengage Learning.
21. Dash, M., Liu, H., Feature selection for classification. Intelligent Data Analysis, 1, 1997.
22. Demšar, J., Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research 7 (2006), 1–30.
23. Deng, L., Yu, D., Deep learning: Methods and applications. Foundations and Trends in Signal Processing 7:3–4 (2014), 197–387.
24. Duda, R.O., Stork, D.G., Hart, P.E., Pattern classification. 2000, Wiley.
25. Dunn, O.J., Multiple comparisons using rank sums. Technometrics 6:3 (1964), 241–252.
26. Fernández-Delgado, M., Cernadas, E., Barro, S., Amorim, D., Do we need hundreds of classifiers to solve real world classification problems?. Journal of Machine Learning Research 15:1 (2014), 3133–3181.
27. Freund, Y., Schapire, R.E., Experiments with a new boosting algorithm. International conference on machine learning, 1996, 148–156.
28. Friedman, J.H., Stochastic gradient boosting. Computational Statistics and Data Analysis 38:4 (2002), 367–378.
29. Friedman, M., The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association 32 (1937), 675–701.
30. Garcia, S., Herrera, F., An extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons. Journal of Machine Learning Research 9 (2009), 2677–2694.
31. Giudici, P., Applied data mining: Statistical methods for business and industry. 2005, John Wiley & Sons.
32. Goodfellow, I. J., Warde-Farley, D., Lamblin, P., Dumoulin, V., Mirza, M., Pascanu, R., Bergstra, J., Bastien, F., & Bengio, Y. (2013). Pylearn2: A machine learning research library. arXiv:1308.4214.
33. Guyon, I., Elisseeff, A., An introduction to variable and feature selection. Journal of Machine Learning Research 3 (2003), 1157–1182.
34. Hand, D.J., Measuring classifier performance: A coherent alternative to the area under the roc curve. Machine Learning 77:1 (2009), 103–123.
35. Hand, D.J., Till, R.J., A simple generalisation of the area under the roc curve for multiple class classification problems. Machine Learning 45:2 (2001), 171–186.
36. He, H., Garcia, E.A., Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering 21:9 (2009), 1263–1284.
37. Hernández-Orallo, J., Flach, P., Ferri, C., A unified view of performance metrics: Translating threshold choice into expected classification loss. Journal of Machine Learning Research 13:Oct (2012), 2813–2869.
38. Hinton, G.E., Osindero, S., Teh, Y.W., A fast learning algorithm for deep belief nets. Neural Computation 18:7 (2006), 1527–1554.
39. Hsu, C.-W., Chang, C.-C., Lin, C.-J., et al. A practical guide to support vector classification. Technical Report, 2003, Department of Computer Science, National Taiwan University.
40. Hu, H., Li, J., Plank, A.W., Wang, H., Daggard, G., A comparative study of classification methods for microarray data analysis. Christen, P., Kennedy, P.J., Li, J., Simoff, S.J., Williams, G.J., (eds.) AusDM, 61, 2006, Australian Computer Society, 33–37.
41. Huang, G.-B., Zhou, H., Ding, X., Zhang, R., Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B 42:2 (2012), 513–529.
42. Jones, S., Johnstone, D., Wilson, R., An empirical evaluation of the performance of binary classifiers in the prediction of credit ratings changes. Journal of Banking & Finance 56:C (2015), 72–85.
43. Keerthi, S.S., Lin, C.-J., Asymptotic behaviors of support vector machines with gaussian kernel. Neural Computation 15:7 (2003), 1667–1689.
44. Khoshgoftaar, T.M., Golawala, M., Hulse, J.V., An empirical study of learning from imbalanced data using random forest. ICTAI (2), 2007, IEEE Computer Society, 310–317.
45. King, R.D., Feng, C., Sutherland, A., Statlog: Comparison of classification algorithms on large real-world problems. Applied Artificial Intelligence 9 (1995), 289–333.
46. Kononenko, I., Bratko, I., Information-based evaluation criterion for classifier's performance. Machine Learning 6:1 (1991), 67–80.
47. Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 2012.
48. Kuncheva, L.I., Switching between selection and fusion in combining classifiers: An experiment. IEEE Transactions on Systems, Man, and Cybernetics, Part B 32:2 (2002), 146–156.
49. Kuncheva, L.I., Rodríguez, J.J., Classifier ensembles for fmri data analysis: An experiment. Magnetic Resonance Imaging 28:4 (2010), 583–593.
50. Landgrebe, T.C.W., Duin, R.P.W., Approximating the multiclass roc by pairwise analysis. Pattern Recognition Letters 28:13 (2007), 1747–1758.
51. Le Cessie, S., Van Houwelingen, J.C., Ridge estimators in logistic regression. Applied Statistics 41:1 (1992), 191–201.
52. Lessmann, S., Baesens, B., Seow, H.-V., Thomas, L.C., Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research. European Journal of Operational Research 247:1 (2015), 124–136.
53. Li, T., Zhang, C., Ogihara, M., A comparative study of feature selection and multiclass classification methods for tissue classification based on gene expression. Bioinformatics 20 (2004), 2429–2437.
54. Lim, T.-S., Loh, W.-Y., Shih, Y.-S., A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Machine Learning 40:3 (2000), 203–228.
55. Lin, H.-T., Lin, C.-J., A study on sigmoid kernels for svm and the training of non-psd kernels by smo-type methods. Neural Computation, 2003, 1–32 submitted to.
56. Ling, C.X., Huang, J., Zhang, H., Auc: a better measure than accuracy in comparing learning algorithms. Conference of the canadian society for computational studies of intelligence, 2003, 329–341.
57. Liu, H., Yu, L., Toward integrating feature selection algorithms for classification and clustering. IEEE Transactions on Knowledge and Data Engineering 17 (2005), 491–502.
58. Liu, Y., A comparative study on feature selection methods for drug discovery. Journal of Chemical Information and Modeling 44:5 (2004), 1823–1828.
59. Liu, Y., Loh, H.T., Tor, S.B., Comparison of extreme learning machine with support vector machine for text classification. Ali, M., Esposito, F., (eds.) IEA/AIEaie Lecture Notes in Computer Science, 3533, 2005, Springer, 390–399.
60. Lorena, A.C., Jacintho, L.F.O., de Siqueira, M.F., Giovanni, R.D., Lohmann, L.G., de Carvalho, A.C.P.L.F., Yamamoto, M., Comparing machine learning classifiers in potential distribution modelling. Expert Systems with Applications 38:5 (2011), 5268–5275.
61. Macia, N., Bernado-Mansilla, E., Towards uci+: A mindful repository design. Information Science 261 (2014), 237–262.
62. Nanni, L., Brahnam, S., Ghidoni, S., Lumini, A., Toward a general-purpose heterogeneous ensemble for pattern classification. Computational Intelligence and Neuroscience, 2015, 2015, 85.
63. Nanni, L., Brahnam, S., Lumini, A., High performance set of pseaac and sequence based descriptors for protein classification. Journal of Theoretical Biology 266:1 (2010), 1–10.
64. Nanni, L., Brahnam, S., Lumini, A., Matrix representation in pattern classification. Expert Systems with Applications 39:3 (2012), 3031–3036.
65. Nanni, L., Fantozzi, C., Lazzarini, N., Coupling different methods for overcoming the class imbalance problem. Neurocomputing 158:C (2015), 48–61.
66. Ngai, E., Xiu, L., Chau, D., Application of data mining techniques in customer relationship management: A literature review and classification. Expert Systems with Applications 36:2, Part 2 (2009), 2592–2602.
67. Pal, M., Random forest classifier for remote sensing classification. International Journal of Remote Sensing 26:1 (2005), 217–222.
68. Palm, R.B., Prediction as a candidate for learning deep hierarchical models of data, 5, 2012, Technical University of Denmark.
69. Quinlan, J.R., C4.5: Programs for machine learning. 1993, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
70. Rijsbergen, C.J.V., Information retrieval. 2nd, 1979, Butterworth-Heinemann.
71. Saeys, Y., Abeel, T., de Peer, Y.V., Robust feature selection using ensemble feature selection techniques. Machine learning and knowledge discovery in databases, european conference, ECML/PKDD 2008, 2008, 313–325.
72. Saeys, Y., Inza, I.n., Larrañaga, P., A review of feature selection techniques in bioinformatics. Bioinformatics 23:19 (2007), 2507–2517.
73. Tukey, J.W., Exploratory data analysis. 1977, Addison-Wesley.
74. Tuszynski, J. (2008). Catools: Tools: Moving window statistics, gif, base64, roc auc, etc. R package version, 1.
75. Van Rijn, J.N., Bischl, B., Torgo, L., Gao, B., Umaashankar, V., Fischer, S.,.. Vanschoren, J., Openml: A collaborative science platform. Joint european conference on machine learning and knowledge discovery in databases, 2013, Springer, 645–649.
76. Vanschoren, J., Blockeel, H., Pfahringer, B., Holmes, G., Experiment databases - a new way to share, organize and learn from experiments. Machine Learning 87:2 (2012), 127–158.
77. Wei, B., Yang, M., Shen, Y., Rana, R.K., Chou, C.T., Hu, W., Real-time classification via sparse representation in acoustic sensor networks. The 11th ACM conference on embedded network sensor systems, SenSys ‘13, Roma, Italy, 2013, 1–14.
78. Wilcoxon, F., Individual comparisons by ranking methods. Biometrics Bulletin 1:6 (1945), 80–83.
79. Wolpert, D.H., The lack of a priori distinctions between learning algorithms. Neural Computing 8:7 (1996), 1341–1390.
80. Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Ma, Y., Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 31:2 (2009), 210–227.
81. Wu, X., Kumar, V., Quinlan, J.R., Ghosh, J., Yang, Q., Motoda, H.,.. Philip, S.Y., et al. Top 10 algorithms in data mining. Knowledge and Information Systems 14:1 (2008), 1–37.
82. Xia, Y., Liu, C., Li, Y., Liu, N., A boosted decision tree approach using bayesian hyper-parameter optimization for credit scoring. Expert Systems with Applications 78 (2017), 225–241.
83. Ye, J., Chow, J.-H., Chen, J., Zheng, Z., Stochastic gradient boosted distributed decision trees. Cheung, D.W.-L., Song, I.-Y., Chu, W.W., Hu, X., Lin, J.J., (eds.) CIKM, 2009, ACM, 2061–2064.
84. Yule, G.U., On the association of attributes in statistics: With illustrations from the material of the childhood society, &c. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character 194 (1900), 257–319.
85. Zheng, Z., A benchmark for classifier learning. Technical Report, 1993, University of Sydney.
86. Zhu, X., Wu, X., Yang, Y., Error detection and impact-sensitive instance ranking in noisy datasets. AAAI, 2004, 378–384.