A unified framework for high-dimensional analysis of m-estimators with decomposable regularizers

Statistical Science

Volumn 27, Issue 4, 2012, Pages 538-557

  Negahban, Sahand N ^a   Ravikumar, Pradeep ^b   Wainwright, Martin J ^c   Yu, Bin ^c  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b University of Texas at Austin   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Group Lasso; High dimensional statistics; L1 regularization; Lasso; M estimator; Nuclear norm; Sparsity

Indexed keywords

EID: 84871600478     PISSN: 08834237     EISSN: None     Source Type: Journal    
DOI: 10.1214/12-STS400     Document Type: Article

References (82)

1. AGARWAL, A., NEGAHBAN, S. and WAINWRIGHT, M. J. (2011). Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions. Ann. Statist. 40 1171-1197.
2. BACH, F. (2010). Self-concordant analysis for logistic regression. Electron. J. Stat. 4 384-414.
3. BACH, F. R. (2008). Consistency of the group lasso and multiple kernel learning. J. Mach. Learn. Res. 9 1179-1225.
4. BACH, F. R. (2008). Consistency of trace normminimization. J. Mach. Learn. Res. 9 1019-1048.
5. BARANIUK, R. G., CEVHER, V., DUARTE, M. F. and HEGDE, C. (2008). Model-based compressive sensing. Technical report, Rice Univ. Available at arXiv:0808.3572.
6. BICKEL, P. J.,BROWN, J. B.,HUANG, H. and LI, Q. (2009). An overview of recent developments in genomics and associated statistical methods. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 367 4313-4337.
7. BICKEL, P. J. and LEVINA, E. (2008). Covariance regularization by thresholding. Ann. Statist. 36 2577-2604.
8. BICKEL, P. J., RITOV, Y. and TSYBAKOV, A. B. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705-1732.
9. BUNEA, F. (2008). Honest variable selection in linear and logistic regression models via l1 and l1 +l2 penalization. Electron. J. Stat. 2 1153-1194.
10. BUNEA, F., SHE, Y. and WEGKAMP, M. (2010). Adaptive rank penalized estimators in multivariate regression. Technical report, Florida State. Available at arXiv:1004.2995.
11. BUNEA, F., TSYBAKOV, A. and WEGKAMP, M. (2007). Sparsity oracle inequalities for the Lasso. Electron. J. Stat. 1 169-194.
12. BUNEA, F., TSYBAKOV, A. B. and WEGKAMP, M. H. (2007). Aggregation for Gaussian regression. Ann. Statist. 35 1674-1697.
13. CAI, T. and ZHOU, H. (2010). Optimal rates of convergence for sparse covariance matrix estimation. Technical report, Wharton School of Business, Univ. Pennsylvania. Available at http://www-stat.wharton.upenn.edu/~tcai/paper/html/Sparse-Covariance-Matrix.html.
14. CANDES, E. and TAO, T. (2007). The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Statist. 35 2313-2351.
15. CANDÈS, E. J. and RECHT, B. (2009). Exact matrix completion via convex optimization. Found. Comput. Math. 9 717- 772.
16. CANDES, E. J. and TAO, T. (2005). Decoding by linear programming. IEEE Trans. Inform. Theory 51 4203-4215.
17. CANDES, E. J., X. LI, Y. M. and WRIGHT, J. (2010). Stable principal component pursuit. In IEEE International Symposium on Information Theory, Austin, TX.
18. CHANDRASEKARAN, V., SANGHAVI, S., PARRILO, P. A. and WILLSKY, A. S. (2011). Rank-sparsity incoherence for matrix decomposition. SIAM J. Optimiz. 21 572-596.
19. CHEN, S. S., DONOHO, D. L. and SAUNDERS, M. A. (1998). Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20 33-61.
20. DONOHO, D. L. (2006). Compressed sensing. IEEE Trans. Inform. Theory 52 1289-1306.
21. DONOHO, D. L. and TANNER, J. (2005). Neighborliness of randomly projected simplices in high dimensions. Proc. Natl. Acad. Sci. USA 102 9452-9457 (electronic).
22. EL KAROUI, N. (2008). Operator norm consistent estimation of large-dimensional sparse covariance matrices. Ann. Statist. 36 2717-2756.
23. FAZEL, M. (2002). Matrix rank minimization with applications. Ph.D. thesis, Stanford. Available at http://faculty. washington.edu/mfazel/thesis-final.pdf.
24. GIRKO, V. L. (1995). Statistical Analysis of Observations of Increasing Dimension. Theory and Decision Library. Series B: Mathematical and Statistical Methods 28. Kluwer Academic, Dordrecht. Translated from the Russian.
25. GREENSHTEIN, E. and RITOV, Y. (2004). Persistence in high-dimensional linear predictor selection and the virtue of overparametrization. Bernoulli 10 971-988.
26. HSU, D., KAKADE, S. M. and ZHANG, T. (2011). Robust matrix decomposition with sparse corruptions. IEEE Trans. Inform. Theory 57 7221-7234.
27. HUANG, J. and ZHANG, T. (2010). The benefit of group sparsity. Ann. Statist. 38 1978-2004.
28. JACOB, L., OBOZINSKI, G. and VERT, J. P. (2009). Group Lasso with overlap and graph Lasso. In International Conference on Machine Learning (ICML) 433-440, Haifa, Israel.
29. JENATTON, R.,MAIRAL, J., OBOZINSKI, G. and BACH, F. (2011). Proximal methods for hierarchical sparse coding. J. Mach. Learn. Res. 12 2297-2334.
30. KAKADE, S. M., SHAMIR, O., SRIDHARAN, K. and TEWARI, A. (2010). Learning exponential families in highdimensions: Strong convexity and sparsity. In AISTATS, Sardinia, Italy.
31. KESHAVAN, R. H., MONTANARI, A. and OH, S. (2010). Matrix completion from noisy entries. J. Mach. Learn. Res. 11 2057-2078.
32. KIM, Y., KIM, J. and KIM, Y. (2006). Blockwise sparse regression. Statist. Sinica 16 375-390.
33. KOLTCHINSKII, V. and YUAN, M. (2008). Sparse recovery in large ensembles of kernel machines. In Proceedings of COLT, Helsinki, Finland.
34. KOLTCHINSKII, V. and YUAN, M. (2010). Sparsity in multiple kernel learning. Ann. Statist. 38 3660-3695.
35. LAM, C. and FAN, J. (2009). Sparsistency and rates of convergence in large covariance matrix estimation. Ann. Statist. 37 4254-4278.
36. LANDGREBE, D. (2008). Hyperspectral image data analsysis as a high-dimensional signal processing problem. IEEE Signal Processing Magazine 19 17-28.
37. LEE, K. and BRESLER, Y. (2009). Guaranteed minimum rank approximation from linear observations by nuclear norm minimization with an ellipsoidal constraint. Technical report, UIUC. Available at arXiv:0903.4742.
38. LIU, Z. and VANDENBERGHE, L. (2009). Interior-point method for nuclear norm approximation with application to system identification. SIAM J. Matrix Anal. Appl. 31 1235- 1256.
39. LOUNICI, K., PONTIL, M., TSYBAKOV, A. B. and VAN DE GEER, S. (2009). Taking advantage of sparsity in multitask learning. Technical report, ETH Zurich. Available at arXiv:0903.1468.
40. LUSTIG, M., DONOHO, D., SANTOS, J. and PAULY, J. (2008). Compressed sensing MRI. IEEE Signal Processing Magazine 27 72-82.
41. MCCOY, M. and TROPP, J. (2011). Two proposals for robust PCA using semidefinite programming. Electron. J. Stat. 5 1123-1160.
42. MEHTA, M. L. (1991). Random Matrices, 2nd ed. Academic Press, Boston, MA.
43. MEIER, L., VAN DE GEER, S. and BÜHLMANN, P. (2009). High-dimensional additive modeling. Ann. Statist. 37 3779- 3821.
44. MEINSHAUSEN, N. (2008). A note on the Lasso for Gaussian graphical model selection. Statist. Probab. Lett. 78 880-884.
45. MEINSHAUSEN, N. and BÜHLMANN, P. (2006). Highdimensional graphs and variable selection with the lasso. Ann. Statist. 34 1436-1462.
46. MEINSHAUSEN, N. and YU, B. (2009). Lasso-type recovery of sparse representations for high-dimensional data. Ann. Statist. 37 246-270.
47. NARDI, Y. and RINALDO, A. (2008). On the asymptotic properties of the group lasso estimator for linear models. Electron. J. Stat. 2 605-633.
48. NEGAHBAN, S., RAVIKUMAR, P.,WAINWRIGHT, M. J. and YU, B. (2009). A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers. In NIPS Conference, Vancouver, Canada.
49. NEGAHBAN, S., RAVIKUMAR, P.,WAINWRIGHT, M. J. and YU, B. (2012). Supplement to "A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers." DOI:10.1214/12-STS400SUPP.
50. NEGAHBAN, S. and WAINWRIGHT, M. J. (2011). Simultaneous support recovery in high-dimensional regression: Benefits and perils of _1,∞-regularization. IEEE Trans. Inform. Theory 57 3481-3863.
51. NEGAHBAN, S. and WAINWRIGHT, M. J. (2011). Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Ann. Statist. 39 1069-1097.
52. NEGAHBAN, S. and WAINWRIGHT, M. J. (2012). Restricted strong convexity and (weighted) matrix completion: Optimal bounds with noise. J. Mach. Learn. Res. 13 1665-1697.
53. OBOZINSKI, G., WAINWRIGHT, M. J. and JORDAN, M. I. (2011). Support union recovery in high-dimensional multivariate regression. Ann. Statist. 39 1-47.
54. PASTUR, L. A. (1972). The spectrum of random matrices. Teoret. Mat. Fiz. 10 102-112.
55. RASKUTTI, G., WAINWRIGHT, M. J. and YU, B. (2010). Restricted eigenvalue properties for correlated Gaussian designs. J. Mach. Learn. Res. 11 2241-2259.
56. RASKUTTI, G., WAINWRIGHT, M. J. and YU, B. (2011). Minimax rates of estimation for high-dimensional linear regression over _q -balls. IEEE Trans. Inform. Theory 57 6976- 6994.
57. RASKUTTI, G., WAINWRIGHT, M. J. and YU, B. (2012). Minimax-optimal rates for sparse additive models over kernel classes via convex programming. J. Mach. Learn. Res. 13 389-427.
58. RAVIKUMAR, P., LAFFERTY, J., LIU, H. and WASSERMAN, L. (2009). Sparse additive models. J. R. Stat. Soc. Ser. B Stat. Methodol. 71 1009-1030.
59. RAVIKUMAR, P.,WAINWRIGHT, M. J. and LAFFERTY, J. D. (2010). High-dimensional Ising model selection using _1-regularized logistic regression. Ann. Statist. 38 1287-1319.
60. RAVIKUMAR, P.,WAINWRIGHT, M. J., RASKUTTI, G. and YU, B. (2011). High-dimensional covariance estimation by minimizing _1-penalized log-determinant divergence. Electron. J. Stat. 5 935-980.
61. RECHT, B. (2011). A simpler approach to matrix completion. J. Mach. Learn. Res. 12 3413-3430.
62. RECHT, B., FAZEL, M. and PARRILO, P. A. (2010). Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52 471-501.
63. ROHDE, A. and TSYBAKOV, A. B. (2011). Estimation of high-dimensional low-rank matrices. Ann. Statist. 39 887- 930.
64. ROTHMAN, A. J., BICKEL, P. J., LEVINA, E. and ZHU, J. (2008). Sparse permutation invariant covariance estimation. Electron. J. Stat. 2 494-515.
65. RUDELSON, M. and ZHOU, S. (2011). Reconstruction from anisotropic random measurements. Technical report, Univ. Michigan.
66. STOJNIC, M., PARVARESH, F. and HASSIBI, B. (2009). On the reconstruction of block-sparse signals with an optimal number of measurements. IEEE Trans. Signal Process. 57 3075-3085.
67. TIBSHIRANI, R. (1996). Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58 267- 288.
68. TIBSHIRANI, R., SAUNDERS, M., ROSSET, S., ZHU, J. and KNIGHT, K. (2005). Sparsity and smoothness via the fused lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 67 91-108.
69. TROPP, J. A., GILBERT, A. C. and STRAUSS, M. J. (2006). Algorithms for simultaneous sparse approximation. Signal Process. 86 572-602. Special issue on "Sparse approximations in signal and image processing."
70. TURLACH, B. A., VENABLES, W. N. and WRIGHT, S. J. (2005). Simultaneous variable selection. Technometrics 47 349-363.
71. VAN DE GEER, S. A. (2008). High-dimensional generalized linear models and the lasso. Ann. Statist. 36 614-645.
72. VAN DE GEER, S. A. and BÜHLMANN, P. (2009). On the conditions used to prove oracle results for the Lasso. Electron. J. Stat. 3 1360-1392.
73. WAINWRIGHT, M. J. (2009). Sharp thresholds for highdimensional and noisy sparsity recovery using _1-constrained quadratic programming (Lasso). IEEE Trans. Inform. Theory 55 2183-2202.
74. WAINWRIGHT, M. J. (2009). Information-theoretic limits on sparsity recovery in the high-dimensional and noisy setting. IEEE Trans. Inform. Theory 55 5728-5741.
75. WIGNER, E. P. (1955). Characteristic vectors of bordered matrices with infinite dimensions. Ann. of Math. (2) 62 548- 564.
76. XU, H., CARAMANIS, C. and SANGHAVI, S. (2012). Robust PCA via outlier pursuit. IEEE Trans. Inform. Theory 58 3047-3064.
77. YUAN, M., EKICI, A., LU, Z. and MONTEIRO, R. (2007). Dimension reduction and coefficient estimation in multivariate linear regression. J. R. Stat. Soc. Ser. B Stat. Methodol. 69 329-346.
78. YUAN, M. and LIN, Y. (2006). Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B Stat. Methodol. 68 49-67.
79. ZHANG, C.-H. and HUANG, J. (2008). The sparsity and bias of the LASSO selection in high-dimensional linear regression. Ann. Statist. 36 1567-1594.
80. ZHAO, P.,ROCHA, G. and YU, B. (2009). The composite absolute penalties family for grouped and hierarchical variable selection. Ann. Statist. 37 3468-3497.
81. ZHAO, P. and YU, B. (2006). On model selection consistency of Lasso. J. Mach. Learn. Res. 7 2541-2563.
82. ZHOU, S., LAFFERTY, J. and WASSERMAN, L. (2008). Time-varying undirected graphs. In 21st Annual Conference on Learning Theory (COLT), Helsinki, Finland.