Review and performance comparison of SVM- and ELM-based classifiers

Neurocomputing

Volumn 128, Issue , 2014, Pages 507-516

  Chorowski, Jan ^a,b   Wang, Jian ^a,c,d   Zurada, Jacek M ^a,e  

^a University of Louisville   (United States)

^b UNIVERSITY OF WROC AW   (Poland)

^c DALIAN UNIVERSITY OF TECHNOLOGY   (China)

^d CHINA UNIVERSITY OF PETROLEUM   (China)

^e UNIVERSITY OF SOCIAL SCIENCES   (Poland)

Author keywords

Classification; Convex quadratic programming; ELM; Machine learning; Randomization; SVM

Indexed keywords

CLASSIFICATION ACCURACY; CONVEX OPTIMIZATION PROBLEMS; CONVEX QUADRATIC PROGRAMMING; ELM; EXTREME LEARNING MACHINE; NESTED CROSS VALIDATIONS; RANDOMIZATION; SVM;

CLASSIFICATION (OF INFORMATION); CONVEX OPTIMIZATION; DATA PROCESSING; LEARNING SYSTEMS;

SUPPORT VECTOR MACHINES;

ACCURACY; ALGORITHM; ARTICLE; CLASSIFICATION; CLASSIFIER; EXTREME LEARNING MACHINE; LEAST SQUARE SUPPORT VECTOR MACHINE; LINEAR SYSTEM; MACHINE LEARNING; MARGIN LOSS EXTREME LEARNING MACHINE; MATHEMATICAL COMPUTING; MATHEMATICAL MODEL; NONLINEAR SYSTEM; PRIORITY JOURNAL; REGRESSION ANALYSIS; SUPPORT VECTOR MACHINE;

EID: 84893662789     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2013.08.009     Document Type: Article

References (74)

1. Cortes C., Vapnik V. Support-vector networks. Mach. Learn. 1995, 20(3):273-297.
2. Vapnik V. The Nature of Statistical Learning Theory 1995, Springer-Verlag, New York Inc.
3. Cherkassky V., Mulier F. Learning from Data. Concepts, Theory, and Methods 2007, Wiley-IEEE Press.
4. Boyd S., Vandenberghe L. Convex Optimization 2004, Cambridge Univ Press.
5. M. Grant, S. Boyd, CVX: Matlab Software For Disciplined Convex Programming, version 1.21, 2011. URL http://cvxr.com/cvx.
6. Bishop C. Pattern Recognition and Machine Learning 2006, Springer, New York.
7. Y. Grandvalet, J. Mariéthoz, S. Bengio, A probabilistic interpretation of SVMS with an application to unbalanced classification, in: Advances in Neural Information Processing Systems, vol. 18, 2006, pp. 467-474.
8. Sollich P. Probabilistic methods for support vector machines. Adv. Neural Inf. Process. Syst. 2000, 12:349-355.
9. Huang G., Zhu Q., Siew C. Extreme learning machine. theory and applications. Neurocomputing 2006, 70(1):489-501.
10. B. Boser, I. Guyon, V. Vapnik, A training algorithm for optimal margin classifiers, in: Proceedings of the Fifth Annual Workshop on Computational Learning Theory, 1992, pp. 144-152.
11. Wahba G., et al. Support vector machines, reproducing Kernel Hilbert spaces and the randomized GACV. Adv. Kernel Methods Support Vector Learn. 1999, 6:69-87.
12. Tan P., Steinbach M., Kumar V., et al. Introduction to Data Mining 2006, Pearson Addison Wesley, Boston.
13. Nocedal J., Wright S. Numerical Optimization 1999, Springer Verlag.
14. Zhu J., Hastie T. Kernel logistic regression and the import vector machine. J. Comput. Graph. Stat. 2005, 14(1):185-205.
15. Suykens J., Vandewalle J. Least squares support vector machine classifiers. Neural Process. Lett. 1999, 9(3):293-300.
16. J. Suykens, J. Vandewalle, Multiclass least squares support vector machines, in: International Joint Conference on Neural Networks, IJCNN'99, vol. 2, IEEE, 1999, pp. 900-903.
17. Vapnik V. Statistical Learning Theory 1998, Wiley, New York.
18. Smola A., Schölkopf B., Müller K. The connection between regularization operators and support vector kernels. Neural Netw. 1998, 11(4):637-649.
19. Schölkopf B., Burges C., Smola A.E. Advances in Kernel Methods. Support Vector Learning 1999, The MIT Press.
20. Cristianini N., Shawe-Taylor J. An Introduction to Support Vector Machines. and Other Kernel-based Learning Methods 2000, Cambridge Univ Press.
21. Burges C. A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Dis. 1998, 2(2):121-167.
22. Golub G., Van Loan C. Matrix Computations 1996, vol. 3. Johns Hopkins University Press.
23. J. Suykens, L. Lukas, P. V. Dooren, B. D. Moor, J. Vandewalle, Least squares support vector machine classifiers: a large scale algorithm, in: European Conference on Circuit Theory and Design, 1999, pp. 839-842.
24. Van Gestel T., Suykens J., Baesens B., Viaene S., Vanthienen J., Dedene G., De Moor B., Vandewalle J. Benchmarking least squares support vector machine classifiers. Mach. Learn. 2004, 54(1):5-32.
25. Suykens J., De Brabanter J., Lukas L., Vandewalle J. Weighted least squares support vector machines. robustness and sparse approximation. Neurocomputing 2002, 48(1-4):85-105.
26. Cawley G., Talbot N. Improved sparse least-squares support vector machines. Neurocomputing 2002, 48(1-4):1025-1031.
27. L. Wei, Z. Chen, J. Li, W. Xu, Sparse and robust least squares support vector machine: a linear programming formulation, in: IEEE International Conference on Grey Systems and Intelligent Services, GSIS 2007, IEEE, 2007, pp. 1134-1138.
28. Liu J., Li J., Xu W., Shi Y. A weighted Lq adaptive least squares support vector machine classifiers-robust and sparse approximation. Expert Syst. Appl. 2011, 38(3):2253-2259.
29. Wei L., Chen Z., Li J. Evolution strategies based adaptive Lp LS-SVM. Inf. Sci. 2011, 181:3000-3016.
30. Huang G., Chen L., Siew C. Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans. Neural Netw. 2006, 17(4):879-892.
31. Haykin S.O. Neural Networks. A Comprehensive Foundation, 2E 1999, Prentice Hall.
32. Huang G., Babri H. Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions. IEEE Trans. Neural Netw. 1998, 9(1):224-229.
33. Sartori M., Antsaklis P. A simple method to derive bounds on the size and to train multilayer neural networks. IEEE Trans. Neural Netw. 1991, 2(4):467-471.
34. Huang S., Huang Y. Bounds on the number of hidden neurons in multilayer perceptrons. IEEE Trans. Neural Netw. 1991, 2(1):47-55.
35. Leshno M., Lin V., Pinkus A., Schocken S. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Netw. 1993, 6(6):861-867.
36. Gao B., Xu Y. Univariant approximation by superpositions of a sigmoidal function. J. Math. Anal. Appl. 1993, 178(1):221-226.
37. Hornik K. Approximation capabilities of multilayer feedforward networks. Neural Netw. 1991, 4(2):251-257.
38. Hornik K., Stinchcombe M., White H. Multilayer feedforward networks are universal approximators. Neural Netw. 1989, 2(5):359-366.
39. Funahashi K. On the approximate realization of continuous mappings by neural networks. Neural Netw. 1989, 2(3):183-192.
40. Huang G., Chen L. Convex incremental extreme learning machine. Neurocomputing 2007, 70(16-18):3056-3062.
41. Huang G., Chen L. Enhanced random search based incremental extreme learning machine. Neurocomputing 2008, 71(16):3460-3468.
42. Li M., Huang G., Saratchandran P., Sundararajan N. Fully complex extreme learning machine. Neurocomputing 2005, 68:306-314.
43. Huang G., Li M., Chen L., Siew C. Incremental extreme learning machine with fully complex hidden nodes. Neurocomputing 2008, 71(4):576-583.
44. Rong H., Huang G., Sundararajan N., Saratchandran P. Online sequential fuzzy extreme learning machine for function approximation and classification problems. IEEE Trans. Syst. Man. Cybern. Part B. Cybern. 2009, 39(4):1067-1072.
45. Liang N., Huang G., Saratchandran P., Sundararajan N. A fast and accurate online sequential learning algorithm for feedforward networks. IEEE Trans. Neural Netw. 2006, 17(6):1411-1423.
46. Lan Y., Soh Y., Huang G. Ensemble of online sequential extreme learning machine. Neurocomputing 2009, 72(13-15):3391-3395.
47. Sun Z., Choi T., Au K., Yu Y. Sales forecasting using extreme learning machine with applications in fashion retailing. Decis. Support Syst. 2008, 46(1):411-419.
48. Van Heeswijk M., Miche Y., Lindh-Knuutila T., Hilbers P., Honkela T., Oja E., Lendasse A. Adaptive ensemble models of extreme learning machines for time series prediction. Lect. Notes Comput. Sci. 2009, 5769:305-314.
49. Huang G., Wang D., Lan Y. Extreme learning machines. a survey. Int. J. Mach. Learn. Cybern. 2011, 2:107-122.
50. Huang G.-B., Zhou H., Ding X., Zhang R. Extreme learning machine for regression and multiclass classification. IEEE Trans. Syst. Man Cybern. Part B. Cybern. 2012, 42(2):513-529.
51. W. Deng, Q. Zheng, L. Chen, Regularized extreme learning machine, in: IEEE Symposium on Computational Intelligence and Data Mining, CIDM'09, IEEE, 2009, pp. 389-395.
52. Deng W., Chen L. Color image watermarking using regularized extreme learning machine. Neural Netw. World 2010, 20(Sp. Iss. {SI} 3):317-330.
53. Q. Liu, Q. He, Z. Shi, Extreme support vector machine classifier, in: Proceedings of the 12th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, Springer-Verlag, 2008, pp. 222-233.
54. B. Frénay, M. Verleysen, Using svms with randomised feature spaces: an extreme learning approach, in: Proceedings of the 18th European Symposium on Artificial Neural Networks (ESANN), Bruges, Belgium, vol. 28, 2010, p. 30.
55. Huang G., Ding X., Zhou H. Optimization method based extreme learning machine for classification. Neurocomputing 2010, 74(1):155-163.
56. Williams C. Computation with infinite neural networks. Neural Comput. 1998, 10(5):1203-1216.
57. E. Parviainen, J. Riihimäki, Y. Miche, A. Lendasse, Interpreting extreme learning machine as an approximation to an infinite neural network, in: KDIR 2010: Proceedings of the International Conference on Knowledge Discovery and Information Retrieval, Valencia, Spain, 2010.
58. Frénay B., Verleysen M. Parameter-insensitive kernel in extreme learning for non-linear support vector regression. Neurocomputing 2011, 74:2526-2531.
59. Tibshirani R. Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B (Meth.) 1996, 58:267-288.
60. Martínez-Martínez J., Escandell-Montero P., Soria-Olivas E., Martín-Guerrero J., Magdalena-Benedito R., Gómez-Sanchis J. Regularized extreme learning machine for regression problems. Neurocomputing 2011, 74(17):3716-3721.
61. Y. Miche, A. Sorjamaa, A. Lendasse, Op-elm: Theory, experiments and a toolbox, in: Proceedings of the 18th International Conference on Artificial Neural Networks, Part I, ICANN '08, Springer-Verlag, Berlin, Heidelberg, 2008, pp. 145-154.
62. Zhu J., Rosset S., Hastie T., Tibshirani R. 1-norm support vector machines. Adv. Neural Inf. Process. Syst. 2004, 16(1):49-56.
63. Mangasarian O. Exact 1-norm support vector machines via unconstrained convex differentiable minimization. J. Mach. Learn. Res. 2006, 7:1517-1530.
64. 1-norm multiclass support vector machines. J. Am. Stat. Assoc. 2007, 102(478):583-594.
65. Chang C., Lin C. Libsvm. a library for support vector machines. ACM Trans. Intell. Syst. Technol. (TIST) 2011, 2(3):27.
66. Hsu C., Lin C. A comparison of methods for multiclass support vector machines. IEEE Trans. Neural Netw. 2002, 13(2):415-425.
67. Duan K., Keerthi S. Which is the best multiclass SVM methodα An empirical study. Mult. Classif. Syst. 2005, 732-760.
68. Official ELM Toolbox (accessed 6/9/2012). http://www.ntu.edu.sg/home/egbhuang/ELM_Codes.htm.
69. Keerthi S., Shevade S. SMO algorithm for least-squares SVM formulations. Neural Comput. 2003, 15:487-507.
70. Platt J.C. Fast training of support vector machines using sequential minimal optimization. Advances in Kernel Methods 1999, 185-208. MIT Press, Cambridge, MA, USA. B. Schölkopf, C.J.C. Burges, A.J. Smola (Eds.).
71. D.N.A. Asuncion, UCI Machine Learning Repository, 2007. URL http://www.ics.uci.edu/~mlearn/MLRepository.html.
72. K.D. Brabanter, P. Karsmakers, C.A.F. Ojeda, J.D. Brabanter, K. Pelckmans, B.D. Moor, J. Vandewalle, J. Suykens, Ls-svmlab toolbox. URL (accessed 6/9/2012). http://www.esat.kuleuven.be/sista/lssvmlab/.
73. M. Grant, S. Boyd, Graph implementations for nonsmooth convex programs, in: V. Blondel, S. Boyd, H. Kimura (Eds.), Recent Advances in Learning and Control (tribute to M. Vidyasagar), Lecture Notes in Control and Information Sciences, Springer, 2008, pp. 95-110.
74. S. Canu, Y. Grandvalet, V. Guigue, A. Rakotomamonjy, SVM and kernel Methods Matlab Toolbox, Perception Systemes et Information, INSA de Rouen, Rouen, France, vol. 2, 2005.