Feature vector regression with efficient hyperparameters tuning and geometric interpretation

Neurocomputing

Volumn 218, Issue , 2016, Pages 411-422

  Liu, Jie ^a   Zio, Enrico ^a,b  

^a UNIVERSITÉ PARIS SACLAY   (France)

^b POLITECNICO DI MILANO   (Italy)

Author keywords

Computational complexity; Feature Vector Regression; Feature vector selection; Hyperparameters tuning; Kernel method; Prediction; Regression

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTATIONAL COMPLEXITY; FORECASTING; LEARNING SYSTEMS; OPTIMIZATION; REGRESSION ANALYSIS; VECTOR SPACES; VECTORS;

FEATURE VECTOR SELECTION; FEATURE VECTORS; HYPERPARAMETERS; KERNEL METHODS; REGRESSION;

FEATURE EXTRACTION;

ARTICLE; CONTROLLED STUDY; FEATURE VECTOR SELECTION; HYPERPARAMETER TUNING; KERNEL BASED RECURSIVE LEAST SQUARES TRACKER; KERNEL GAUSSIAN PROCESS; KERNEL METHOD; KERNEL PARTIAL LEAST SQUARE; LEAST SQUARE ANALYSIS; MEASUREMENT ACCURACY; PREDICTION; PRIORITY JOURNAL; PROCESS OPTIMIZATION; QUANTIZED KERNEL LEAST MEAN SQUARE; RELEVANCE VECTOR MACHINE; SUPPORT VECTOR MACHINE; TUNING CURVE;

EID: 84994128794     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2016.08.093     Document Type: Article

References (58)

1. [1] Amari, S., Wu, S., Improving support vector machine classifiers by modifying kernel functions. Neural Netw. 12 (1999), 783–789, 10.1016/S0893-6080(99)00032-5.
2. [2] Baudat, G., Anouar, F., Feature vector selection and projection using kernels. Neurocomputing 55 (2003), 21–38, 10.1016/S0925-2312(03)00429-6.
3. [3] Brooks, T., Pope, D., Marcolini, M., Airfoil Self-noise Prediction. 1989.
4. [4] Csató, L., Opper, M., Sparse on-line gaussian processes. Neural Comput. 14 (2002), 641–668, 10.1162/089976602317250933.
5. [5] Cawley, G.C., Talbot, N.L.C., Reduced rank kernel ridge regression. Neural Process. Lett. 16 (2002), 293–302, 10.1023/A:1021798002258.
6. [6] Chapelle, O., Training a support vector machine in the primal. Neural Comput. 19 (2007), 1155–1178, 10.1162/neco.2007.19.5.1155.
7. [7] Chen, S., Local regularization assisted orthogonal least squares regression. Neurocomputing 69 (2006), 559–585, 10.1016/j.neucom.2004.12.011.
8. [8] Chen, S., Cowan, C.N., Grant, P.M., Orthogonal least squares learning algorithm for radial basis function networks. IEEE Trans. Neural Netw. 2 (1991), 302–309, 10.1109/72.80341.
9. [9] Cherkassky, V., Ma, Y., Practical selection of SVM parameters and noise estimation for SVM regression. Neural Netw. 17 (2004), 113–126, 10.1016/S0893-6080(03)00169-2.
10. [10] Braga, I., Monard, M.C., Improving the kernel regularized least squares method for small-sample regression. Neurocomputing, 2014, 10.1016/j.neucom.2014.12.097.
11. [11] G. Fung, O.L. Mangasarian, Proximal support vector machine classifiers, in: Proc. Seventh ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. - KDD ’01, 2001: pp. 77–86. doi:10.1145/502512.502527.
12. [12] Furey, T.S., Cristianini, N., Duffy, N., Bednarski, D.W., Schummer, M., Haussler, D., Support vector machine classification and validation of cancer tissue samples using microarray expression data. Bioinformatics 16 (2000), 906–914, 10.1093/bioinformatics/16.10.906.
13. [13] Gao, J., Shi, D., Liu, X., Significant vector learning to construct sparse kernel regression models. Neural Netw. 20 (2007), 791–798, 10.1016/j.neunet.2007.03.001.
14. [14] Gold, C., Holub, A., Sollich, P., Bayesian approach to feature selection and parameter tuning for support vector machine classifiers. Neural Netw. 18 (2005), 693–701, 10.1016/j.neunet.2005.06.044.
15. [15] Yin, J.B., Li, T., Shen, H. Bin, Gaussian kernel optimization: complex problem and a simple solution. Neurocomputing 74 (2011), 3816–3822, 10.1016/j.neucom.2011.07.017.
16. [16] Hofmann, T., Schölkopf, B., Smola, A.J., Kernel methods in machine learning. Ann. Stat. 36 (2008), 1171–1220, 10.1214/009053607000000677.
17. [17] Zhou, X., Jiang, W., Tian, Y., Shi, Y., Kernel subclass convex hull sample selection method for SVM on face recognition. Neurocomputing 73 (2010), 2234–2246, 10.1016/j.neucom.2010.01.008.
18. [18] Jeng, J.-T., Hybrid approach of selecting hyperparameters of support vector machine for regression. IEEE Trans. Syst. Man. Cybern. B Cybern. 36 (2006), 699–709.
19. [19] Krishnapuram, B., Carin, L., Figueiredo, M.A.T., Hartemink, A.J., Sparse multinomial logistic regression: Fast algorithms and generalization bounds. IEEE Trans. Pattern Anal. Mach. Intell. 27 (2005), 957–968, 10.1109/TPAMI.2005.127.
20. [20] Izquierdo-Verdiguier, E., Jenssen, R., Gomez-Chova, L., Camps-Valls, G., Spectral clustering with the probabilistic cluster kernel. Neurocomputing 149 (2015), 1299–1304, 10.1016/j.neucom.2014.08.068.
21. [21] Chih-Wei Hsu, C.-J.L., Chang, Chih-Chung, A Practical Guide to Support Vector Classification. BJU Int. 101 (2008), 1396–1400, 10.1177/02632760022050997.
22. [22] Lin, S.-W., Ying, K.-C., Chen, S.-C., Lee, Z.-J., Particle swarm optimization for parameter determination and feature selection of support vector machines. Expert Syst. Appl. 35 (2008), 1817–1824, 10.1016/j.eswa.2007.08.088.
23. [23] Liu, S., Feng, L., Xiao, Y., Wang, H., Robust activation function and its application: Semi-supervised kernel extreme learning method. Neurocomputing 144 (2014), 318–328, 10.1016/j.neucom.2014.04.041.
24. [24] Lopez, R., Balsa-Canto, E., Oñate, E., Neural networks for variational problems in engineering. Int. J. Numer. Methods Eng. 75 (2008), 1341–1360, 10.1002/nme.2304.
25. [25] Guo, J., Yi, P., Wang, R., Ye, Q., Zhao, C., Feature selection for least squares projection twin support vector machine. Neurocomputing 144 (2014), 174–183, 10.1016/j.neucom.2014.05.040.
26. [26] Müller, K., Smola, A., Rätsch, G., Schölkopf, B., J, Predicting time series with support vector machines, Artif. Neural Networks—ICANN’97 1327 (1997), 999–1004, 10.1007/BFb0020283.
27. [27] Müller, K.R., Mika, S., Rätsch, G., Tsuda, K., Schölkopf, B., An introduction to kernel-based learning algorithms. IEEE Trans. Neural Netw. 12 (2001), 181–201, 10.1109/72.914517.
28. [28] Polat, K., Güneş, S., A new feature selection method on classification of medical datasets: Kernel F-score feature selection. Expert Syst. Appl. 36 (2009), 10367–10373, 10.1016/j.eswa.2009.01.041.
29. [29] Rasmussen, C.E., Ghahramani, Z., Infinite Mixtures of Gaussian Process Experts. Adv. Neural Inf. Process. Syst. 2 (2002), 881–888 〈http://books.google.com/books?hl=en&lr=&id=GbC8cqxGR7YC&oi=fnd&pg=PA881&dq=Infinite+Mixtures+of+Gaussian+Process+Experts&ots=ZvK1D34Avb&sig=7Ayuhz3bGQDumESnS_szzHhqCF8〉.
30. [30] Rosipal, R., Trejo, L.J., Kernel Partial Least Squares Regression in Reproducing Kernel Hilbert Space. J. Mach. Learn. Res. 2 (2002), 97–123, 10.1162/15324430260185556.
31. [31] Pontil, M., Verri, A., Support vector machines for 3D object recognition. IEEE Trans. Pattern Anal. Mach. Intell., 20, 1998, 10.1109/34.683777.
32. [32] Lu, C., Zhang, C., Zhang, T., Zhang, W., Kernel based symmetrical principal component analysis for face classification. Neurocomputing 70 (2007), 904–911, 10.1016/j.neucom.2006.10.019.
33. [33] Schölkopf, B., Smola, A., Müller, K.-R., Nonlinear Component Analysis as a Kernel Eigenvalue Problem. Neural Comput. 10 (1998), 1299–1319, 10.1162/089976698300017467.
34. [34] S. Mika, G. Ratsch, J. Weston, B. Scholkopf, Fisher discriminant analysis with kernels, Neural Networks Signal Process. IX, 1999. Proc. 1999 IEEE Signal Process. Soc. Work., 1999, pp. 41–48. doi:10.1109/NNSP.1999.788121.
35. [35] Schölkopf, B., Mika, S., Burges, C.J.C., Knirsch, P., Müller, K.R., Rätsch, G., et al. Input space versus feature space in kernel-based methods. IEEE Trans. Neural Netw. 10 (1999), 1000–1017, 10.1109/72.788641.
36. [36] Schölkopf, B., Herbrich, R., Smola, A.J., A Generalized Representer Theorem. Most 2111 (2001), 416–426, 10.1007/3-540-44581-1_27.
37. [37] Zhang, L., Hu, X., Locally adaptive multiple kernel clustering. Neurocomputing, 2014, 10.1016/j.neucom.2013.05.064.
38. [38] Smola, A.J., Schölkopf, B., A tutorial on support vector regression. Stat. Comput. 14 (2004), 199–222, 10.1023/B:STCO.0000035301.49549.88.
39. [39] Steinwart, I., On the optimal parameter choice for nu-support vector machines. IEEE Trans. Pattern Anal. Mach. Intell. 25 (2002), 1274–1284, 10.1109/TPAMI.2003.1233901.
40. [40] Vandewalle, J., Least squares support vector machine classifiers. Neural Process. Lett. 9 (1999), 293–300, 10.1023/A:1018628609742.
41. [41] Yang, S., Yang, L., Feng, Z., Wang, M., Jiao, L., Sparse Ridgelet Kernel Regressor and its online sequential extreme learning. Neurocomputing 134 (2014), 173–180, 10.1016/j.neucom.2012.12.066.
42. [42] Tsang, I.W.H., Kwok, J.T.Y., Zurada, J.M., Generalized core vector machines. IEEE Trans. Neural Netw. 17 (2006), 1126–1140, 10.1109/TNN.2006.878123.
43. [43] Tüfekci, P., Prediction of full load electrical power output of a base load operated combined cycle power plant using machine learning methods. Int. J. Electr. Power Energy Syst. 60 (2014), 126–140, 10.1016/j.ijepes.2014.02.027.
44. [44] Vijayakumar, S., D'Souza, A., Schaal, S., Incremental online learning in high dimensions. Neural Comput. 17 (2005), 2602–2634, 10.1162/089976605774320557.
45. [45] Wang, J., Fleet, D., Hertzmann, A., Gaussian process dynamical models. Adv. Neural Inf. Process. Syst., 2006, 1441–1448, 10.1109/TPAMI.2007.1167.
46. [46] Williams, P.M., Using neural networks to model conditional multivariate densities. Neural Comput. 8 (1996), 843–854, 10.1162/neco.1996.8.4.731.
47. [47] Soh, W., Kim, H., Yum, B.-J., Application of kernel principal component analysis to multi-characteristic parameter design problems. Ann. Oper. Res., 2015, 10.1007/s10479-015-1889-2.
48. [48] Pai, P.F., System reliability forecasting by support vector machines with genetic algorithms. Math. Comput. Model. 43 (2006), 262–274, 10.1016/j.mcm.2005.02.008.
49. [49] Wei, Z., Tao, T., ZhuoShu, D., Zio, E., A dynamic particle filter-support vector regression method for reliability prediction. Reliab. Eng. Syst. Saf. 119 (2013), 109–116, 10.1016/j.ress.2013.05.021.
50. [50] Zhu, J., Hastie, T., Kernel Logistic Regression and the Import Vector Machine. J. Comput. Graph. Stat. 14 (2005), 185–205, 10.1198/106186005×25619.
51. [51] Choi, Y., Ozawa, S., Lee, M., Incremental two-dimensional kernel principal component analysis. Neurocomputing 134 (2014), 280–288, 10.1016/j.neucom.2013.08.045.
52. [52] Nandan, M., Khargonekar, P.P., Talathi, S.S., Fast SVM Training Using Approximate Extreme Points. J. Mach. Learn. Res. 15 (2014), 59–98 〈http://jmlr.org/papers/v15/nandan14a.html〉.
53. [53] Cho, Y., Saul, L.K., Kernel methods for deep learning. NIPS 9 (2009), 342–350.
54. [54] Rahimi, A., Recht, B., Random features for large-scale kernel machines. Adv. Neural Inf., 2007, 1–8 doi:10.1.1.145.8736.
55. [55] Lu, Z., May, A., Liu, K., Garakani, A.B., Guo, D., Bellet, A., et al. How to scale up kernel methods to be as good as deep neural nets. Arxiv, 2014, 1–21.
56. [56] Van Vaerenbergh, S., Lazaro-Gredilla, M., Santamaria, I., Kernel recursive least-squares tracker for time-varying regression. IEEE Trans. Neural Netw. Learn. Syst. 23 (2012), 1313–1326, 10.1109/TNNLS.2012.2200500.
57. [57] Chen, B., Zhao, S., Zhu, P., Principe, J.C., Quantized kernel least mean square algorithm. IEEE Trans. Neural Netw. Learn. Syst. 23 (2012), 22–32, 10.1109/TNNLS.2011.2178446.
58. [58] Demšar, J., Statistical Comparisons of Classifiers over Multiple Data Sets. J. Mach. Learn. Res. 7 (2006), 1–30, 10.1016/j.jecp.2010.03.005.