Towards ultrahigh dimensional feature selection for big data

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 1371-1429

  Tan, Mingkui ^a   Tsang, Ivor W ^b   Wang, Li ^c  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^b UNIVERSITY OF TECHNOLOGY SYDNEY   (Australia)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Big data; Feature generation; Feature selection; Multiple kernel learning; Nonlinear feature selection; Ultrahigh dimensionality

Indexed keywords

FEATURE EXTRACTION; ITERATIVE METHODS; MATHEMATICAL PROGRAMMING; VIRTUAL REALITY;

COMPETITIVE PERFORMANCE; FEATURE GENERATION; FEATURE SELECTION METHODS; GENERALIZATION PERFORMANCE; MULTIPLE KERNEL LEARNING; NONLINEAR FEATURES; SEMI-INFINITE PROGRAMMING PROBLEMS; ULTRAHIGH DIMENSIONALITY;

BIG DATA;

EID: 84901642365     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (91)

1. F. R. Bach, R. Jenatton, J. Mairal, and G. Obozinski. Convex optimization with sparsityinducing norms. In Optimization for Machine Learning. S. Sra, S. Nowozin, S. J. Wright., 2011.
2. F. R. Bach. High-dimensional non-linear variable selection through hierarchical kernel learning. Technical report, 2009.
3. F. R. Bach, G. R. G. Lanckriet, and M. I. Jordan. Multiple kernel learning, conic duality, and the SMO algorithm. In ICML, 2004.
4. A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. on Imaging Sciences, 2(1):183-202, 2009.
5. B . Blum, M. I. Jordan, D. E. Kim, R. Das, P. Bradley, and D. Baker. Feature selection methods for improving protein structure prediction with rosetta. In NIPS, 2007.
6. P. S. Bradley and O. L. Mangasarian. Feature selection via concave minimization and support vector machines. In ICML, 1998.
7. A. B. Chan, N. Vasconcelos, and G. R. G. Lanckriet. Direct convex relaxations of sparse SVM. In ICML, 2007.
8. Y. W. Chang, C. J. Hsieh, K. W. Chang, M. Ringgaard, and C. J. Lin. Training and testing low-degree polynomial data mappings via linear SVM. JMLR, 11:1471-1490, 2010.
9. O. Chapelle and S. S. Keerthi. Multi-class feature selection with support vector machines. In Proceedings of the American Statistical Association, 2008.
10. O. Chapelle, V. Vapnik, O. Bousquet, and S. Mukherjee. Choosing multiple parameters for support vector machines. Mach. Learn., 46(1):131-159, 2002. (Pubitemid 34129966)
11. J. Chen and J. Ye. Training SVM with indenite kernels. In ICML, 2008.
12. A. Dasgupta, P. Drineas, and B. Harb. Feature selection methods for text classication. In KDD, 2007.
13. J. Deng, A. C. Berg, and F. Li. Hierarchical semantic indexing for large scale image retrieval. In CVPR, pages 785-792. IEEE, 2011.
14. J. Duchi and Y. Singer. Ecient online and batch learning using forward backward splitting. JMLR, 10:2899-2934, 2009.
15. R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classication (2nd ed.). Hoboken, NJ: Wiley-Interscience, 2000.
16. R. Fan, P. Chen, and C.-J. Lin. Working set selection using second order information for training SVM. JMLR, 6:1889-1918, 2005. (Pubitemid 41798130)
17. M Figueiredo, R. Nowak, and S. Wright. Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Sign. Proces.: Special Issue on Convex Optimization Methods for Signal Processing, 1(4):586-597, 2007. (Pubitemid 351276802)
18. G. M. Fung and O. L. Mangasarian. A feature selection newton method for support vector machine classication. Comput. Optim. Appl., 28:185-202, 2004.
19. P. Gehler and S. Nowozin. Innite kernel learning. Technical report, 2008.
20. T. R. Golub, D. K. Slonim, and P. Tamayo. Molecular classication of cancer: class discovery and class prediction by gene expression monitoring. Science, 7:286-531, 1999.
21. Y. Grandvalet and S. Canu. Adaptive scaling for feature selection in svms. In NIPS, 2002.
22. Q. Gu, Z. Li, and J. Han. Correlated multi-label feature selection. In CIKM, 2011a.
23. Q. Gu, Z. Li, and J. Han. Generalizedsher score for feature selection. In UAI, 2011b.
24. I. Guyon and A. Elissee. An introduction to variable and feature selection. JMLR, 3: 1157-1182, 2003.
25. I. Guyon, J. Weston, S. Barnhill, and V. Vapnik. Gene selection for cancer classication using support vector machines. Mach. Learn., 46:389-422, 2002. (Pubitemid 34129977)
26. C. J. Hsieh, K. W. Chang, C. J. Lin, S. S. Keerthi, and S. Sundararajan. A dual coordinate descent method for large-scale linear svm. In ICML, 2008.
27. A. Jain, S.V.N. Vishwanathan, and M. Varma. SPF-GMKL: generalized multiple kernel learning with a million kernels. In KDD, 2012.
28. R. Jenatton, J. Audibert, and F. Bach. Structured variable selection with sparsity-inducing norms. Technical report, 2011a.
29. R. Jenatton, J. Mairal, G. Obozinski, and F. Bach. Proximal methods for hierarchical sparse coding. JMLR, 12:2297-2334, 2011b.
30. T. Joachims. Training linear SVMs in linear time. In KDD, 2006.
31. J. E. Kelley. The cutting plane method for solving convex programs. J. Soc. Ind. Appl. Math., 8(4):703 -712, 1960.
32. S. Kim and E. P. Xing. Tree-guided group lasso for multi-response regression with structured sparsity, with applications to eQTL mapping. Ann. Statist., Forthcoming, 2012.
33. S. Kim and E. P. Xing. Tree-guided group lasso for multi-task regression with structured sparsity. In ICML 2010, 2010.
34. S. Kim and S. Boyd. On general minimax theorems. Pacic J. Math., 1958, 8(1):171-176, 1958.
35. M. Kloft and G. Blanchard. On the convergence rate of p-norm multiple kernel learning. JMLR, 13:2465-2501, 2012.
36. M. Kloft, U. Brefeld, S. Sonnenburg, P. Laskov, K. M-uller, and A. Zien. Ecient and accurate p-norm multiple kernel learning. NIPS, 22(22):997-1005, 2009.
37. M. Kloft, U. Brefeld, S. Sonnenburg, and A. Zien. Lp-norm multiple kernel learning. JMLR, 12:953-997, 2011.
38. R. Kohavi and G. John. Wrappers for feature subset selection. Artif. Intell., 97:273-324, 1997. (Pubitemid 127401107)
39. G. R. G. Lanckriet, N. Cristianini, P. Bartlett, L. E. Ghaoui, and M. I. Jordan. Learning the kernel matrix with semidenite programming. JMLR, 5:27-72, 2004.
40. J. Langford, L. Li, and T. Zhang. Sparse online learning via truncated gradient. JMLR, 10:777-801, 2009.
41. P. Li, A. Shrivastava, J. Moore, and A. C. Konig. Hashing algorithms for large-scale learning. In NIPS, 2011.
42. P. Li, A. Owen, and C. H. Zhang. One permutation hashing. In NIPS, 2012.
43. D. Lin, D. P. Foster, and L. H. Ungar. A risk ratio comparison of '0 and '1 penalized regressions. Technical report, University of Pennsylvania, 2010.
44. J. Liu and J. Ye. Ecient l1/lq norm regularization. Technical report, 2010.
45. A. C. Lozano, G. Swirszcz, and N. Abe. Group orthogonal matching pursuit for logistic regression. In AISTATS, 2011.
46. S. Maji and A. C. Berg. Max-margin additive classiers for detection. In ICCV, 2009.
47. Q. Mao and I. W. Tsang. A feature selection method for multivariate performance measures. IEEE Trans. Pattern Anal. Mach., 35(9):2051-2063, 2013.
48. A. F. T. Martins, M. A. T. Figueiredo, P. M. Q. Aguiar, N. A. Smith, and E. P. Xing. Online multiple kernel learning for structured prediction. Technical report, 2010.
49. L. Meier, S. Van De Geer, and P. B-uhlmann. The group lasso for logistic regression. J. Roy. Stat. Soc. B., 70(1):53-71, 2008.
50. A. Mutapcic and S. Boyd. Cutting-set methods for robust convex optimization with pessimizing oracles. Optim. Method Softw., 24(3):381-06, 2009.
51. A. Nedic and A. Ozdaglar. Subgradient methods for saddle-point problems. J. Optimiz. Theory App., 142(1):205-228, 2009.
52. A. Nemirovski. Prox-method with rate of convergence o(1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM J. Opt., 15:229-251, 2005. (Pubitemid 40360669)
53. Y. Nesterov. Gradient methods for minimizing composite objective function. Technical report, 2007.
54. A. Y. Ng. On feature selection: Learning with exponentially many irrelevant features as training examples. In ICML, 1998.
55. J. Nocedal and S. J. Wright. Numerical Optimization. Springer, New York, 2nd edition, 2006.
56. F. Orabona and L. Jie. Ultra-fast optimization algorithm for sparse multi kernel learning. In ICML, 2011.
57. E. Y. Pee and J. O. Royset. On solving large-scalenite minimax problems using exponential smoothing. J. Optimiz. Theory App., online, 2010.
58. Z. Qin, K. Scheinberg, and D. Goldfarb. Ecient block-coordinate descent algorithms for the group lasso. Technical report, 2010.
59. A. Rakotomamonjy. Variable selection using svm-based criteria. JMLR, 3:1357-1370, 2003.
60. A. Rakotomamonjy, F. R. Bach, Y. Grandvalet, and S. Canu SimpleMKL. JMLR, 9: 2491-2521, 2008.
61. M. Rastegari, C. Fang, and L. Torresani. Scalable object-class retrieval with approximate and top-ranking. In ICCV, 2011.
62. V. Roth and B. Fischer. The group-lasso for generalized linear models: uniqueness of solutions and ecient algorithms. In ICML, 2008.
63. D. Sculley, G. M.Wachman, and C. E. Brodley. Spamltering using inexact string matching in explicit feature space with on-line linear classiers. In The Fifteenth Text REtrieval Conference (TREC) Proceedings, 2006.
64. S. Shalev-Shwartz and T. Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. JMLR, 14:567-599, 2013.
65. S. Sonnenburg, G. R-atsch, C. Sch-afer, and B. Sch-olkopf. Large Scale Multiple Kernel Learning. JMLR, 7:1531-1565, 2006. (Pubitemid 44373694)
66. S. Sonnenburg, G. R-atsch, and K. Rieck. Large scale learning with string kernels. MIT Press, 2007.
67. M. Tan, I. W. Tsang, and L. Wang. Learning sparse svm for feature selection on very high dimensional data sets. In ICML, 2010.
68. M. Tan, I. W. Tsang, and L. Wang. Minimax sparse logistic regression for very highdimensional feature selection. IEEE Trans. Neural Netw. Learning Syst., 24(10):1609-1622, 2013.
69. A. Tewari, P. Ravikumar, and I. S. Dhillon. Greedy algorithms for structurally constrained high dimensional problems. In NIPS, 2011.
70. K. C. Toh and S. Yun. An accelerated proximal gradient algorithm for nuclear norm regularized least squares problems. Technical report, 2009.
71. P. Tseng. On accelerated proximal gradient methods for convex-concave optimization. Technical report, University of Washington, 2008.
72. P. Tseng. Convergence of a block coordinate descent method for nondierentiable minimization. J. Optimiz. Theory App., 109(3):475-494, 2001.
73. M. Varma and B. R. Babu. More generality in ecient multiple kernel learning. In ICML, 2009.
74. A. Vedaldi and A. Zisserman. Ecient additive kernels via explicit feature maps. In CVPR, 2010.
75. S. V. N. Vishwanathan, Z. Sun, N. Theera-Ampornpunt, and M. Varma. Multiple kernel learning and the SMO algorithm. In NIPS, December 2010.
76. K. Weinberger, A. Dasgupta, J. Langford, A. Smola, and J. Attenberg. Feature hashing for large scale multitask learning. In ICML, 2009.
77. J. Weston, S. Mukherjee, O. Chapelle, M. Pontil, T. Poggio, and V. Vapnik. Feature selection for svms. In NIPS, 2000.
78. J. Wu. Ecient hik svm learning for image classication. IEEE Trans. Image Process, 21(10):4442-4453, 2012.
79. L. Xiao. Dual averaging methods for regularized stochastic learning and online optimization. In NIPS, 2009.
80. Z. Xu, R. Jin, Ye J., Michael R. Lyu, and King I. Non-monotonic feature selection. In ICML, 2009a.
81. Z. Xu, R. Jin, I. King, and M.R. Lyu. An extended level method for ecient multiple kernel learning. In NIPS. 2009b.
82. Z. Xu, R. Jin, H. Yang, I. King, and M. R. Lyu. Simple and ecient multiple kernel learning by group lasso. In ICML, 2010.
83. G. X. Yuan, K. W. Chang, C. J. Hsieh, and C. J. Lin. A comparison of optimization methods and software for large-scale l1-regularized linear classication. JMLR, 11:3183-3236, 2010.
84. G. X. Yuan, C. H. Ho, and C. J. Lin. An improved GLMNET for l1-regularized logistic regression and support vector machines. JMLR, 13:1999-2030, 2012.
85. M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. J. Roy. Stat. Soc. B., 68(1):49-67, 2006. (Pubitemid 43415335)
86. C. H. Zhang. Nearly unbiased variable selection under minimax concave penalty. Ann. Statist., 38(2):894-942, 2010a.
87. C. H. Zhang and J. Huang. The sparsity and bias of the lasso selection in high-dimensional linear regression. Ann. Statist., 36(4):1567-1594, 2008.
88. D. Zhang and W. S. Lee. Extracting key-substring-group features for text classication. In KDD, 2006.
89. T. Zhang. Analysis of multi-stage convex relaxation for sparse regularization. JMLR, 11: 1081-1107, 2010b.
90. Z. Zhao and H. Liu. Spectral feature selection for supervised and unsupervised learning. In ICML, 2007.
91. J. Zhu, S. Rossett, T. Hastie, and R. Tibshirani. 1-norm support vector machines. In NIPS, 2003.