Feature selection in machine learning: an exact penalty approach using a Difference of Convex function Algorithm

Machine Learning

Volumn 101, Issue 1-3, 2015, Pages 163-186

  Le Thi, Hoai An ^a   Le, Hoai Minh ^a   Pham Dinh, Tao ^b  

^a UNIVERSITÉ DE LORRAINE   (France)

^b Avenue de l Université 76801 Saint Etienne du Rouvray   (France)

Author keywords

DC programming; DCA; Exact penalty; Feature selection; Zero norm

Indexed keywords

ALGORITHMS; APPROXIMATION ALGORITHMS; ARTIFICIAL INTELLIGENCE; CLASSIFICATION (OF INFORMATION); FUNCTIONS; ITERATIVE METHODS; LEARNING SYSTEMS; LINEAR PROGRAMMING;

D-C PROGRAMMING; DC PROGRAMMING AND DCA; DCA; DIFFERENCE OF CONVEX FUNCTIONS; EMPIRICAL - COMPARISONS; EXACT PENALTY; FEATURE SELECTION AND CLASSIFICATION; ZERO NORMS;

FEATURE EXTRACTION;

EID: 84942368519     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-014-5455-y     Document Type: Article

References (54)

1. Amaldi, E., & Kann, V. (1998). On the approximability of minimizing non zero variables or unsatisfied relations in linear systems. Theoretical Computer Science, 209, 237–260.
2. Bach, F., Jenatton, R., Mairal, J., & Obzinski, G. (2012). Optimization with sparsity-inducing penalties foundations and trends. Foundations and Trends in Machine Learning, 4(1), 1–106.
3. Bradley, P. S., & Mangasarian, O. L. (1998). Feature selection via concave minimization and support vector machines. In Proceeding of international conference on machine learning ICML’98.
4. Candes, E., Wakin, M., & Boyd, S. (2008). Enhancing sparsity by reweighted (Formula Presented.) minimization. Journal of Mathematical Analysis and Applications, 14, 877–905.
5. Chartrand, R., & Yin, W. (2008). Iteratively reweighted algorithms for compressive sensing. Acoustics, speech and signal processing, IEEE international conference ICASSP, 2008, 3869–3872.
6. Chen, X., Xu, F. M., & Ye, Y. (2010). Lower bound theory of nonzero entries in solutions of l2-lp minimization. SIAM Journal on Scientific Computing, 32(5), 2832–2852.
7. Chen, Y., Li, Y., Cheng, X.-Q., & Guo, L. (2006). Survey and taxonomy of feature selection algorithms in intrusion detection system. In Proceedings of inscrypt, 2006. LNCS (Vol. 4318, 153–167).
8. Collober, R., Sinz F., Weston, J., & Bottou, L. (2006). Trading convexity for scalability. In Proceedings of the 23rd international conference on machine learning ICML 2006 (pp. 201–208). Pittsburgh, PA. ISBN:1-59593-383-2.
9. Cristianini, N., & Shawe-Taylor, N. (2000). Introduction to support vector machines. Cambridge: Cambridge University Press.
10. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1997). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39, 1–38.
11. Fan, J., & Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96(456), 1348–1360.
12. Fu, W. J. (1998). Penalized regression: The bridge versus the lasso. Journal of Computational and Graphical Statistics, 7, 397–416.
13. Gasso, G., Rakotomamonjy, A., & Canu, S. (2009). Recovering sparse signals with a certain family of nonconvex penalties and dc programming. IEEE Transactions on Signal Processing, 57, 4686–4698.
14. Gorodnitsky, I. F., & Rao, B. D. (1997). Sparse signal reconstructions from limited data using FOCUSS: A re-weighted minimum norm algorithm. IEEE Transactions on Signal Processing, 45, 600–616.
15. Guan, W., & Gray, A. (2013). Sparse high-dimensional fractional-norm support vector machine via DC programming. Computational Statistics and Data Analysis, 67, 136–148.
16. Gribonval, R., & Nielsen, M. (2003). Sparse representation in union of bases. IEEE Transactions on Information Theory, 49, 3320–3325.
17. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning (2nd ed.). Heidelberg: Springer.
18. Huang, J., Horowitz, J., & Ma, S. (2008). Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Annals of Statistics, 36, 587–613.
19. Kim, Y., Choi, H., & Oh, H. S. (2008). Smoothly clipped absolute deviation on high dimensions. Journal of the American Statistical Association, 103(484), 1665–1673.
20. Knight, K., & Fu, W. (2000). Asymptotics for lasso-type estimators. Annals of Statistics, 28, 1356–1378.
21. Krause, N., & Singer, Y. (2004). Leveraging the margin more carefully. In Proceedings of the 21 international conference on Machine learning ICML 2004. Banff, Alberta, Canada, 63.ISBN:1-58113-828-5.
22. Le Thi, H.A. DC Programming and DCA. http://lita.sciences.univ-metz.fr/~lethi.
23. Le Thi, H. A. (1997). Contribution à l’optimisation non convexe et l’optimisation globale: Théorie. Algorithmes et Applications: Habilitation à Diriger des Recherches, Université de Rouen.
24. Le Thi, H. A., & Pham Dinh, T. (1997). Solving a class of linearly constrained indefinite quadratic problems by DC algorithms. Journal of Global Optimization, 11(3), 253–285.
25. Le Thi, H. A., & Pham Dinh, T. (2005). The DC (difference of convex functions) programming and DCA revisited with DC models of real-world nonconvex optimization problems. Annals of Operations Research, 133, 23–46.
26. Le Thi, H. A., Belghiti, T., Pham Dinh, T. (2007) A new efficient algorithm based on DC programming and DCA for clustering. Journal of Global Optimization, 37, 593–608.
27. Le Thi, H. A., Le, H. M. & Pham Dinh, T. (2006). Optimization based DC programming and DCA for hierarchical clustering. European Journal of Operational Research, 183(3), 1067–1085.
28. Le Thi, H. A., Le, H. M., Nguyen, V. V., & Pham Dinh, T. (2008). A dc programming approach for feature selection in support vector machines learning. Journal of Advances in Data Analysis and Classification, 2, 259–278.
29. Le Thi, H. A., Nguyen, V. V., & Ouchani, S. (2009). Gene selection for cancer classification using DCA. Journal of Fonctiers of Computer Science and Technology, 3(6), 62–72.
30. Le Thi, H. A., Huynh, V. N., & Pham Dinh, T. (2012). Exact penalty and error bounds in DC programming. Journal of Global Optimization dedicated to Reiner Horst, 52(3), 509–535.
31. Liu, Y., Shen, X., & Doss, H. (2005). Multicategory ψ-learning and support vector machine: Computational tools. Journal of Computational and Graphical Statistics, 14, 219–236.
32. Liu, Y., & Shen, X. (2006). Multicategory ψ-learning. Journal of the American Statistical Association, 101, 500–509.
33. Mangasarian, O. L. (1996). Machine learning via polyhedral concave minimization. In H. Fischer, B. Riedmueller, & S. Schaeffler (Eds.), Applied mathematics and parallel computing—Festschrift for Klaus Ritter (pp. 175–188). Heidelberg: Physica.
34. Mallat, S., & Zhang, Z. (1993). Matching pursuit in a time-frequency dictionary. IEEE Transactions on Signal Processing, 41(12), 3397–3415.
35. Meinshausen, N. (2007). Relaxed Lasso. Computational Statistics and Data Analysis, 52(1), 374–393.
36. Natarajan, B. K. (1995). Sparse approximate solutions to linear systems. SIAM Journal on Computing, 24, 227–234.
37. Neumann, J., Schnörr, C., & Steidl, G. (2005). Combined SVM-based feature selection and classification. Machine Learning, 61(1–3), 129–150.
38. Ong, C. S., & Le Thi, H. A. (2013). Learning sparse classifiers with Difference of Convex functions algorithms. Optimization Methods and Software, 28(4), 830–854.
39. Peleg, D., & Meir, R. (2008). A bilinear formulation for vector sparsity optimization. Signal Processing, 8(2), 375–389.
40. Pham Dinh, T., & Le Thi, H. A. (1998). DC optimization algorithms for solving the trust region subproblem. SIAM Journal on Optimization, 8, 476–505.
41. Pham Dinh, T., & Le Thi, H. A (2014). Recent advances in DC programming and DCA. Transactions on Computational Collective. Intelligence., 8342, 1–37.
42. Rakotomamonjy, A., Flamary, R., Gasso, G., & Canu, S. (2011). (Formula Presented.)-ℓq penalty for sparse linear and sparse multiple kernel multi-task learning. IEEE Transactions on Neural Networks, 22(8), 13071320.
43. Rao, B. D., & Kreutz-Delgado, K. (1999). An affine scaling methodology for best basis selection. IEEE Transactions on Signal Processing, 47, 187–200.
44. Rao, B. D., Engan, K., Cotter, S. F., Palmer, J., & KreutzDelgado, K. (2003). Subset selection in noise based on diversity measure minimization. IEEE Transactions on Signal Processing, 51(3), 760–770.
45. Rinaldi, F. (2000). Mathematical Programming Methods for minimizing the zero-norm over polyhedral sets, PhD thesis, Sapienza, University of Rome (2009)
46. Thiao, M., Pham Dinh, T., & Le Thi, H. A. (2010). A DC programming approach for sparse eigenvalue problem. Proceeding of ICML, 2010, 1063–1070.
47. Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, 46, 431–439.
48. Yuille, A. L., & Rangarajan, A. (2003). The convex concave procedure. Neural Computation, 15(4), 915–936.
49. Wang, L., Zhu, J., & Zou, H. (2006). The doubly regularized support vector machine. Statistica Sinica, 16, 589–615.
50. Weston, J., Elisseeff, A., Scholkopf, B., & Tipping, M. (2003). Use of the zero-norm with linear models and kernel methods. Journal of Machine Learning Research., 3, 1439–1461.
51. Zhang, H. H., Ahn, J., Lin, X., & Park, C. (2006). Gene selection using support vector machines with non-convex penalty. Bioinformatics, 2(1), 88–95.
52. Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B, 67, 301–320.
53. Zou, H. (2006). The adaptive lasso and its oracle properties. Journal of the American Statistical Association, 101, 1418–1429.
54. Zou, H., & Li, R. (2008). One-step sparse estimates in nonconcave penalized likelihood models. Annals of Statistics, 36(4), 1509–1533.