Structured sparsity through convex optimization

Statistical Science

Volumn 27, Issue 4, 2012, Pages 450-468

  Bach, Francis ^a   Jenatton, Rodolphe ^b   Mairal, Julien ^c   Obozinski, Guillaume ^a  

^a ECOLE NORMALE SUPÉRIEURE   (France)

^b CENTRE DE MATHÉMATIQUES APPLIQUÉES   (France)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Convex optimization; Sparsity

Indexed keywords

EID: 84871604261     PISSN: 08834237     EISSN: None     Source Type: Journal    
DOI: 10.1214/12-STS394     Document Type: Article

References (105)

1. ADAMS, R., GHAHRAMANI, Z. and JORDAN, M. (2010). Treestructured stick breaking for hierarchical data. In Advances in Neural Information Processing Systems 23 (J. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel and A. Culotta, eds.) 19-27.
2. AHARON, M., ELAD, M. and BRUCKSTEIN, A. (2006). K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Processing 54 4311-4322.
3. BACH, F. R. (2008). Consistency of the group lasso and multiple kernel learning. J. Mach. Learn. Res. 9 1179-1225.
4. BACH, F. (2009). Exploring large feature spaces with hierarchical multiple kernel learning. In Neural Information Processing Systems 21.
5. BACH, F. (2010). Structured sparsity-inducing norms through submodular functions. In Advances in Neural Information Processing Systems 23.
6. BACH, F. (2011a). Learning with submodular functions: A convex optimization perspective. Technical Report No. 00645271, HAL.
7. BACH, F. (2011b). Shaping level sets with submodular functions. In Advances in Neural Information Processing Systems 24.
8. BACH, F., MAIRAL, J. and PONCE, J. (2008). Convex sparse matrix factorizations. Technical report. Preprint. Available at arXiv:0812.1869.
9. BACH, F., JENATTON, R., MAIRAL, J. and OBOZINSKI, G. (2012). Optimization with sparsity-inducing penalties. Foundations and Trends in Machine Learning 4 1-106.
10. BARANIUK, R. G., CEVHER, V., DUARTE, M. F. and HEGDE, C. (2010). Model-based compressive sensing. IEEE Trans. Inform. Theory 56 1982-2001.
11. BECK, A. and TEBOULLE, M. (2009). A fast iterative shrinkagethresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2 183-202.
12. BECKER, S.,BOBIN, J. and CANDÈS, E. J. (2011). NESTA: A fast and accurate first-order method for sparse recovery. SIAM J. Imaging Sci. 4 1-39.
13. BICKEL, P. J., RITOV, Y. and TSYBAKOV, A. B. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705-1732.
14. BLEI, D. M., GRIFFITHS, T. L. and JORDAN, M. I. (2010). The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. J. ACM 57 1-30.
15. BLEI, D., NG, A. and JORDAN, M. (2003). Latent Dirichlet allocation. J. Mach. Learn. Res. 3 993-1022.
16. BONDELL, H. D. and REICH, B. J. (2008). Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with OSCAR. Biometrics 64 115-123.
17. BORWEIN, J. M. and LEWIS, A. S. (2006). Convex Analysis and Nonlinear Optimization. Theory and Examples, 2nd ed. CMS Books in Mathematics/Ouvrages de Mathématiques de la SMC 3. Springer, New York.
18. BUNTINE, W. L. (2002). Variational extensions to EM and multinomial PCA. In Proceedings of the European Conference on Machine Learning (ECML).
19. CANDES, E. J. and TAO, T. (2005). Decoding by linear programming. IEEE Trans. Inform. Theory 51 4203-4215.
20. CEVHER, V., DUARTE, M. F., HEGDE, C. and BARANIUK, R. G. (2008). Sparse signal recovery using Markov random fields. In Advances in Neural Information Processing Systems 20.
21. CHEN, S. S., DONOHO, D. L. and SAUNDERS, M. A. (1998). Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20 33-61.
22. CHEN, X., LIN, Q., KIM, S., CARBONELL, J. G. andXING, E. P. (2011). Smoothing proximal gradient method for general structured sparse learning. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI).
23. COMBETTES, P. L. and PESQUET, J. C. (2010). Proximal splitting methods in signal processing. In Fixed-Point Algorithms for Inverse Problems in Science and Engineering. Springer, New York.
24. D'ASPREMONT, A., BACH, F. and EL GHAOUI, L. (2008). Optimal solutions for sparse principal component analysis. J. Mach. Learn. Res. 9 1269-1294.
25. DONOHO, D. L. and JOHNSTONE, I. M. (1995). Adapting to unknown smoothness via wavelet shrinkage. J. Amer. Statist. Assoc. 90 1200-1224.
26. EFRON, B., HASTIE, T., JOHNSTONE, I. and TIBSHIRANI, R. (2004). Least angle regression. Ann. Statist. 32 407-499.
27. FRIEDMAN, J., HASTIE, T. and TIBSHIRANI, R. (2010). A note on the group Lasso and a sparse group Lasso. Preprint.
28. FRIEDMAN, J., HASTIE, T., HÖFLING, H. and TIBSHIRANI, R. (2007). Pathwise coordinate optimization. Ann. Appl. Stat. 1 302-332.
29. GRAMFORT, A. and KOWALSKI, M. (2009). Improving M/EEG source localization with an inter-condition sparse prior. In IEEE International Symposium on Biomedical Imaging.
30. GRIFFITHS, T. L. and STEYVERS, M. (2004). Finding scientific topics. Proc. Natl. Acad. Sci. USA 101 5228-5235.
31. HASTIE, T., TIBSHIRANI, R. and FRIEDMAN, J. (2001). The Elements of Statistical Learning. Data Mining, Inference, and Prediction. Springer, New York.
32. HUANG, J. and ZHANG, T. (2010). The benefit of group sparsity. Ann. Statist. 38 1978-2004.
33. HUANG, J., ZHANG, T. andMETAXAS, D. (2011). Learning with structured sparsity. J. Mach. Learn. Res. 12 3371-3412.
34. JACOB, L., OBOZINSKI, G. and VERT, J. P. (2009). Group Lasso with overlaps and graph Lasso. In Proceedings of the International Conference on Machine Learning (ICML).
35. JENATTON, R., AUDIBERT, J.-Y. and BACH, F. (2011). Structured variable selection with sparsity-inducing norms. J. Mach. Learn. Res. 12 2777-2824.
36. JENATTON, R., OBOZINSKI, G. and BACH, F. (2010). Structured sparse principal component analysis. In International Conference on Artificial Intelligence and Statistics (AISTATS).
37. JENATTON, R., GRAMFORT, A., MICHEL, V., OBOZINSKI, G., EGER, E., BACH, F. and THIRION, B. (2011a). Multi-scale mining of fMRI data with hierarchical structured sparsity. SIAM J. Imaging Sci. To appear. Technical report. Preprint. Available at arXiv:1105.0363.
38. JENATTON, R., MAIRAL, J., OBOZINSKI, G. and BACH, F. (2011b). Proximal methods for hierarchical sparse coding. J. Mach. Learn. Res. 12 2297-2334.
39. JOLLIFFE, I. T., TRENDAFILOV, N. T. and UDDIN, M. (2003). A modified principal component technique based on the LASSO. J. Comput. Graph. Statist. 12 531-547.
40. KAVUKCUOGLU, K., RANZATO, M. A., FERGUS, R. and LECUN, Y. (2009). Learning invariant features through topographic filter maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
41. KIM, S., SOHN, K. A. and XING, E. P. (2009). A multivariate regression approach to association analysis of a quantitative trait network. Bioinformatics 25 204-212.
42. KIM, S. and XING, E. P. (2010). Tree-guided group Lasso for multi-task regression with structured sparsity. In Proceedings of the International Conference on Machine Learning (ICML).
43. KIMELDORF, G. andWAHBA, G. (1971). Some results on Tchebycheffian spline functions. J. Math. Anal. Appl. 33 82-95.
44. LEE, D. D. and SEUNG, H. S. (1999). Learning the parts of objects by non-negative matrix factorization. Nature 401 788-791.
45. LIN, Y. and ZHANG, H. H. (2006). Component selection and smoothing in multivariate nonparametric regression. Ann. Statist. 34 2272-2297.
46. LIU, H., PALATUCCI, M. and ZHANG, J. (2009). Blockwise coordinate descent procedures for the multi-task lasso, with applications to neural semantic basis discovery. In Proceedings of the International Conference on Machine Learning (ICML).
47. LOUNICI, K., PONTIL, M., TSYBAKOV, A. B. and VAN DE GEER, S. (2009). Taking advantage of sparsity in multi-task learning. In Proceedings of the Conference on Learning Theory.
48. LOUNICI, K., PONTIL, M., VAN DE GEER, S. and TSYBAKOV, A. B. (2011). Oracle inequalities and optimal inference under group sparsity. Ann. Statist. 39 2164-2204.
49. MACKEY, L. (2009). Deflation methods for sparse PCA. In Advances in Neural Information Processing Systems 21.
50. MAIRAL, J. (2010). Sparse coding for machine learning, image processing and computer vision. Ph.D. thesis, École normale supérieure de Cachan-ENS Cachan. Available at http://tel. archives-ouvertes.fr/tel-00595312/fr/.
51. MAIRAL, J., BACH, F., PONCE, J. and SAPIRO, G. (2010). Online learning for matrix factorization and sparse coding. J. Mach. Learn. Res. 11 19-60.
52. MAIRAL, J., JENATTON, R., OBOZINSKI, G. and BACH, F. (2011). Convex and network flow optimization for structured sparsity. J. Mach. Learn. Res. 12 2681-2720.
53. MALLAT, S. G. (1999). A Wavelet Tour of Signal Processing. Academic Press, New York.
54. MARTINEZ, A. M. and KAK, A. C. (2001). PCA versus LDA. In IEEE Transactions on Pattern Analysis and Machine Intelligence 23 228-233.
55. MEINSHAUSEN, N. and BÜHLMANN, P. (2006). Highdimensional graphs and variable selection with the lasso. Ann. Statist. 34 1436-1462.
56. MOGHADDAM, B., WEISS, Y. and AVIDAN, S. (2006). Spectral bounds for sparse PCA: Exact and greedy algorithms. In Advances in Neural Information Processing Systems 18.
57. MOREAU, J.-J. (1962). Fonctions convexes duales et points proximaux dans un espace hilbertien. C. R. Acad. Sci. Paris 255 2897-2899.
58. NEEDELL, D. and TROPP, J. A. (2009). CoSaMP: Iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmon. Anal. 26 301-321.
59. ∞-regularization. IEEE Trans. Inform. Theory 57 3841-3863.
60. NEGAHBAN, S., RAVIKUMAR, P., WAINWRIGHT, M. J. and YU, B. (2009). A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers. In Advances in Neural Information Processing Systems 22.
61. NESTEROV, Y. (2004). Introductory Lectures on Convex Optimization. A Basic Course. Applied Optimization 87. Kluwer Academic, Boston, MA.
62. NESTEROV, Y. (2007). Gradient methods for minimizing composite objective function. Technical report, Center for Operations Research and Econometrics (CORE), Catholic Univ. Louvain.
63. OBOZINSKI, G. and BACH, F. (2012). Convex relaxation for combinatorial penalties. Technical report, HAL.
64. OBOZINSKI, G., JACOB, L. and VERT, J. P. (2011). Group Lasso with overlaps: The latent group Lasso approach. Technical Report No. inria-00628498, HAL.
65. OBOZINSKI, G., TASKAR, B. and JORDAN, M. I. (2010). Joint covariate selection and joint subspace selection for multiple classification problems. Stat. Comput. 20 231-252.
66. OBOZINSKI, G.,WAINWRIGHT, M. J. and JORDAN, M. I. (2011). Support union recovery in high-dimensional multivariate regression. Ann. Statist. 39 1-47.
67. OLSHAUSEN, B. A. and FIELD, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381 607-609.
68. PERCIVAL, D. (2012). Theoretical properties of the overlapping group Lasso. Electron. J. Statist. 6 269-288.
69. ∞ regularization. In Proceedings of the International Conference on Machine Learning (ICML).
70. RAO, N. S., NOWAK, R. D., WRIGHT, S. J. and KINGSBURY, N. G. (2011). Convex approaches to model wavelet sparsity patterns. In International Conference on Image Processing (ICIP).
71. RAPAPORT, F., BARILLOT, E. and VERT, J.-P. (2008). Classification of arrayCGH data using fused SVM. Bioinformatics 24 i375-i382.
72. RAVIKUMAR, P., LAFFERTY, J., LIU, H. and WASSERMAN, L. (2009). Sparse additive models. J. R. Stat. Soc. Ser. B Stat. Methodol. 71 1009-1030.
73. ROTH, V. and FISCHER, B. (2008). The group-Lasso for generalized linear models: Uniqueness of solutions and efficient algorithms. In Proceedings of the International Conference on Machine Learning (ICML).
74. RUDIN, L. I., OSHER, S. and FATEMI, E. (1992). Nonlinear total variation based noise removal algorithms. Phys. D 60 259-268.
75. SCHMIDT, M., LE ROUX, N. and BACH, F. (2011). Convergence rates of inexact proximal-gradient methods for convex optimization. In Advances in Neural Information Processing Systems 24.
76. SCHMIDT, M. andMURPHY, K. (2010). Convex structure learning in log-linear models: Beyond pairwise potentials. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS).
77. SHALEV-SHWARTZ, S., SREBRO, N. and ZHANG, T. (2010). Trading accuracy for sparsity in optimization problems with sparsity constraints. SIAM J. Optim. 20 2807-2832.
78. SHAWE-TAYLOR, J. and CRISTIANINI, N. (2004). Kernel Methods for Pattern Analysis. Cambridge Univ. Press, Cambridge.
79. SHEN, X. and HUANG, H.-C. (2010). Grouping pursuit through a regularization solution surface. J. Amer. Statist. Assoc. 105 727- 739.
80. SINGH, A. P. and GORDON, G. J. (2008). A unified view of matrix factorization models. In Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases.
81. SPRECHMANN, P., RAMIREZ, I., SAPIRO, G. and ELDAR, Y. (2010). Collaborative hierarchical sparse modeling. In 44th Annual Conference on Information Sciences and Systems (CISS) 1-6. IEEE.
82. STOJNIC, M., PARVARESH, F. and HASSIBI, B. (2009). On the reconstruction of block-sparse signals with an optimal number of measurements. IEEE Trans. Signal Process. 57 3075-3085.
83. TIBSHIRANI, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267-288.
84. TIBSHIRANI, R., SAUNDERS, M., ROSSET, S., ZHU, J. and KNIGHT, K. (2005). Sparsity and smoothness via the fused lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 67 91-108.
85. TROPP, J. A. (2004). Greed is good: Algorithmic results for sparse approximation. IEEE Trans. Inform. Theory 50 2231-2242.
86. TROPP, J. A. (2006). Just relax: Convex programming methods for identifying sparse signals in noise. IEEE Trans. Inform. Theory 52 1030-1051.
87. TURLACH, B. A., VENABLES, W. N. andWRIGHT, S. J. (2005). Simultaneous variable selection. Technometrics 47 349-363.
88. 1-regularization in high-dimensional statistical models. In Proceedings of the International Congress of Mathematicians. Volume IV 2351-2369. Hindustan Book Agency, New Delhi.
89. VAROQUAUX, G., JENATTON, R., GRAMFORT, A., OBOZINSKI, G., THIRION, B. and BACH, F. (2010). Sparse structured dictionary learning for brain resting-state activity modeling. In NIPS Workshop on Practical Applications of Sparse Modeling: Open Issues and New Directions.
90. 1-constrained quadratic programming (Lasso). IEEE Trans. Inform. Theory 55 2183-2202.
91. WITTEN, D. M., TIBSHIRANI, R. and HASTIE, T. (2009). A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. Biostatistics 10 515-534.
92. WRIGHT, S. J., NOWAK, R. D. and FIGUEIREDO, M. A. T. (2009). Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57 2479-2493.
93. WU, T. T. and LANGE, K. (2008). Coordinate descent algorithms for lasso penalized regression. Ann. Appl. Stat. 2 224-244.
94. XIANG, Z. J., XI, Y. T., HASSON, U. and RAMADGE, P. J. (2009). Boosting with spatial regularization. In Advances in Neural Information Processing Systems 22.
95. YUAN, M. (2010). High dimensional inverse covariance matrix estimation via linear programming. J. Mach. Learn. Res. 11 2261- 2286.
96. YUAN, M. and LIN, Y. (2006). Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B Stat. Methodol. 68 49-67.
97. YUAN, G.-X., CHANG, K.-W., HSIEH, C.-J. and LIN, C.-J. (2010). A comparison of optimization methods and software for large-scale L1-regularized linear classification. J. Mach. Learn. Res. 11 3183-3234.
98. ZASS, R. and SHASHUA, A. (2007). Nonnegative sparse PCA. In Advances in Neural Information Processing Systems 19.
99. ZHANG, T. (2009). Some sharp performance bounds for least squares regression with L1 regularization. Ann. Statist. 37 2109-2144.
100. ZHAO, P., ROCHA, G. andYU, B. (2009). The composite absolute penalties family for grouped and hierarchical variable selection. Ann. Statist. 37 3468-3497.
101. ZHAO, P. and YU, B. (2006). On model selection consistency of Lasso. J. Mach. Learn. Res. 7 2541-2563.
102. ZHONG, L. W. andKWOK, J. T. (2011). Efficient sparse modeling with automatic feature grouping. In Proceedings of the International Conference on Machine Learning (ICML).
103. ZHOU, Y., JIN, R. and HOI, S. C. H. (2010). Exclusive Lasso for multi-task feature selection. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS).
104. ZOU, H. (2006). The adaptive lasso and its oracle properties. J. Amer. Statist. Assoc. 101 1418-1429.
105. ZOU, H., HASTIE, T. and TIBSHIRANI, R. (2006). Sparse principal component analysis. J. Comput. Graph. Statist. 15 265-286.