Structured regularizers for high-dimensional problems: Statistical and computational issues

Annual Review of Statistics and Its Application

Volumn 1, Issue , 2014, Pages 233-253

  Wainwright, Martin J ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Algorithms; High dimensional statistics; M estimation; Regularization; Statistical machine learning

Indexed keywords

EID: 84906567304     PISSN: 23268298     EISSN: 2326831X     Source Type: Journal    
DOI: 10.1146/annurev-statistics-022513-115643     Document Type: Article

References (101)

1. Agarwal A, Negahban S, Wainwright MJ. 2012a. Fast global convergence of gradient methods for highdimensional statistical recovery. Ann. Stat. 40(5):2452-82
2. Agarwal A, Negahban S,Wainwright MJ. 2012b. Noisy matrix decomposition via convex relaxation: optimal rates in high dimensions. Ann. Stat. 40(2):1171-97
3. Amini AA, Wainwright MJ. 2009. High-dimensional analysis of semidefinite relaxations for sparse principal component analysis. Ann. Stat. 37(5B):2877-921
4. Bach F. 2008. Consistency of trace norm minimization. J. Mach. Learn. Res. 9:1019-48
5. Bach F, Jenatton R,Mairal J, ObozinskiG. 2012. Optimization with sparsity-inducing penalties. Found. Trends Mach. Learn. 4(1):1-106
6. Baraniuk RG, Cevher V, Duarte MF, Hegde C. 2010. Model-based compressive sensing. IEEE Trans. Inf. Theory 56(4):1982-2001
7. BerthetQ, Rigollet P. 2013. Computational lower bounds for sparse PCA.Tech. Rep., Princeton Univ., Princeton, NJ.http://arxiv1304.0828
8. Bertsekas DP. 1995. Nonlinear Programming. Belmont, MA: Athena Sci.
9. Bickel P, Li B. 2006. Regularization in statistics. TEST 15(2):271-344
10. Bickel P, Ritov Y, Tsybakov A. 2009. Simultaneous analysis of Lasso and Dantzig selector. Ann. Stat. 37(4):1705-32
11. Boyd S, Vandenberghe L. 2004. Convex Optimization. Cambridge, UK: Cambridge Univ. Press
12. BrediesK, Lorenz DA. 2008. Linear convergence of iterative soft thresholding. J. Fourier Anal. Appl. 14:813-37
13. B̈uhlmann P, van de Geer S. 2011. Statistics for High-Dimensional Data. New York: Springer
14. Bunea F, Tsybakov A,Wegkamp M. 2007. Sparsity oracle inequalities for the Lasso. Electron. J. Stat. 169-94
15. Cai T, Liu W, Luo X. 2011. A constrained ℓ1-minimization approach to sparse precision matrix estimation. J. Am. Stat. Assoc. 106:594-607
16. Cai TT, Liu W, Zhou HH. 2012. Estimating sparse precision matrices: optimal rates of convergence and adaptive estimation. Tech. Rep.,Wharton Sch., Univ. Pa., Pennsylvania, PA. http://arxiv1212.2882
17. Cand̀es EJ, Li X, Ma Y,Wright J. 2011. Robust principal component analysis? J. ACM 58:11
18. Cand̀es EJ, Plan Y. 2011. Tight oracle bounds for low-rank matrix recovery from a minimal number of random measurements. IEEE Trans. Inf. Theory 57(4):2342-59
19. Cand̀es EJ,Recht B. 2009. Exact matrix completion via convex optimization. Found. Comput. Math. 9(6):717-72
20. Cand̀es EJ, Tao T. 2007. The Dantzig selector: statistical estimation when p is much larger than n. Ann. Stat. 35(6):2313-51
21. Chandrasekaran V, Parrilo PA,Willsky AS. 2012. Latent variable graphical model selection via convex optimization. Ann. Stat. 40(4):1935-67
22. Chandrasekaran V, Sanghavi S, Parrilo PA, Willsky AS. 2011. Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim. 21:572-96
23. Chen S, Donoho DL, Saunders MA. 1998. Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1):33-61
24. Cohen A, Dahmen W, DeVore R. 2009. Compressed sensing and best k-term approximation. J. Am. Math. Soc. 22(1):211-31
25. d'Aspremont A, Banerjee O, El Ghaoui L. 2008. First-order methods for sparse covariance selection. SIAM J. Matrix Anal. Appl. 30(1):55-66
26. Donoho DL. 2006. Compressed sensing. IEEE Trans. Inf. Theory 52(4):1289-306
27. Donoho DL,Tanner JM. 2008. Counting faces of randomly-projected polytopes when the projection radically lowers dimension. J. Am. Math. Soc. 22:1-53
28. Duchi J, Shalev-Shwartz S, Singer Y, Chandra T. 2008. Efficient projections onto the ℓ1 -ball for learning in high dimensions. Presented at Int. Conf. Mach. Learn., 25th, Helsinki, Finland
29. Fan J, Li R. 2001. Variable selection via non-concave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96(456):1348-60
30. Fazel M. 2002. Matrix rank minimization with applications. PhD Thesis, Stanford Univ., Stanford, CA. http://faculty.washington.edu/mfazel/thesis-final. pdf
31. Friedman J, Hastie T, Tibshirani R. 2008. Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9:432-41
32. Fu WJ. 2001. Penalized regression: the bridge versus the Lasso. J. Comput. Graph. Stat. 7(3):397-416
33. Greenshtein E, Ritov Y. 2004. Persistency in high dimensional linear predictor-selection and the virtue of over-parametrization. Bernoulli 10:971-88
34. Gross D. 2011. Recovering low-rank matrices from few coefficients in any basis. IEEE Trans. Inf. Theory 57(3):1548-66
35. Gyorfi L, Kohler M, Krzyzak A, Walk H. 2002. A Distribution-Free Theory of Nonparametric Regression. New York: Springer
36. Hoerl AE, Kennard RW. 1970. Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12:55-67
37. Hsu D, Kakade SM, Zhang T. 2011. Robust matrix decomposition with sparse corruptions. IEEE Trans. Inf. Theory 57(11):7221-34
38. Huang J, Zhang T. 2010. The benefit of group sparsity. Ann. Stat. 38(4):1978-2004
39. Jacob L, Obozinski G, Vert JP. 2009. Group Lasso with overlap and graph Lasso. Presented at the 26th Int. Conf. Mach. Learn., Montreal, Canada
40. Jalali A, Ravikumar P, Sanghavi SS, Ruan C. 2010. A dirty model for multi-task learning. Adv. Neural Inf. Process. Syst. 23:964-72
41. Javanmard A, Montanari A. 2013. Hypothesis testing in high-dimensional regression under the Gaussian random design model: asymptotic theory. Tech. Rep., Stanford Univ., Stanford, CA. http://arxiv1301.4240
42. Kim Y, Kim J, Kim Y. 2006. Blockwise sparse regression. Stat. Sin. 16(2):375-90
43. Koltchinskii V, Lounici K, Tsybakov AB. 2011. Nuclear-norm penalization and optimal rates for noisy lowrank matrix completion. Ann. Stat. 39:2302-29
44. Koltchinskii V, Yuan M. 2008. Sparse recovery in large ensembles of kernel machines. Presented at Annu. Conf. Learn. Theory, 21st, Helsinki, Finland
45. Koltchinskii V, Yuan M. 2010. Sparsity in multiple kernel learning. Ann. Stat. 38:3660-95
46. Lam C, Fan J. 2009. Sparsistency and rates of convergence in large covariance matrix estimation. Ann. Stat. 37:4254-78
47. Lin Y, Zhang HH. 2006. Component selection and smoothing in multivariate nonparametric regression. Ann. Stat. 34:2272-97
48. Liu H, Lafferty J,Wasserman L. 2009. The nonparanormal: semiparametric estimation of high-dimensional undirected graphs. J. Mach. Learn. Res. 10:1-37
49. Loh P, WainwrightMJ. 2012. High-dimensional regression with noisy and missing data: provable guarantees with non-convexity. Ann. Stat. 40(3):1637-64
50. Lounici K, Pontil M, Tsybakov AB, van de Geer S. 2011. Oracle inequalities and optimal inference under group sparsity. Ann. Stat. 39(4):2164-204
51. Mazumder R, Hastie T, Tibshirani R. 2010. Spectral regularization algorithms for learning large incomplete matrices. J. Mach. Learn. Res. 11:2287-322
52. Meier L, van de Geer S, Buhlmann P. 2009. High-dimensional additive modeling. Ann. Stat. 37:3779-821
53. Meinshausen N, B̈uhlmann P. 2006. High-dimensional graphs and variable selection with the Lasso. Ann. Stat. 34:1436-62
54. Micchelli CA,Morales JM, Pontil M. 2013. Regularizers for structured sparsity. Adv. Comput.Math. 38:455-89
55. Negahban S, Ravikumar P,Wainwright MJ, Yu B. 2012. A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers. Stat. Sci. 27(4):538-57
56. Negahban S, WainwrightMJ. 2011a. Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Ann. Stat. 39(2):1069-97
57. Negahban S, WainwrightMJ. 2011b. Simultaneous support recovery in high-dimensional regression: benefits and perils of ℓ1,8-regularization. IEEE Trans. Inf. Theory 57(6):3841-63
58. Negahban S, Wainwright MJ. 2012. Restricted strong convexity and (weighted) matrix completion: optimal bounds with noise. J. Mach. Learn. Res. 13:1665-97
59. Nesterov Y. 2004. Introductory Lectures on Convex Optimization. New York: Kluwer Acad.
60. Nesterov Y. 2007. Gradient methods for minimizing composite objective function. Tech. Rep. 76, Cent. Oper. Res. Econom., Catholic Univ., Louvain, Belg.
61. Nesterov Y. 2012. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 22(2):341-62
62. Obozinski G, Wainwright MJ, Jordan MI. 2011. Union support recovery in high-dimensional multivariate regression. Ann. Stat. 39(1):1-47
63. Oymak S, Jalali A, Fazel M, Eldar YC, Hassibi B. 2012. Simultaneously structured models with applications to sparse and low-rank matrices. Tech. Rep., Calif. Inst. Technol., Pasadena, CA. http://arxiv1212.3753
64. Raskutti G, Wainwright MJ, Yu B. 2010. Restricted eigenvalue conditions for correlated Gaussian designs. J. Mach. Learn. Res. 11:2241-59
65. Raskutti G, Wainwright MJ, Yu B. 2011. Minimax rates of estimation for high-dimensional linear regression over ℓq -balls. IEEE Trans. Inf. Theory 57(10):6976-94
66. Raskutti G, WainwrightMJ, Yu B. 2012. Minimax-optimal rates for sparse additive models over kernel classes via convex programming. J. Mach. Learn. Res. 12:389-427
67. Ravikumar P, Liu H, Lafferty J, Wasserman L. 2009. SpAM: sparse additive models. J. R. Stat. Soc. Ser. B 71(5):1009-30
68. RavikumarP,WainwrightMJ, Raskutti G, Yu B. 2011. High-dimensional covariance estimation by minimizing ℓ1-penalized log-determinant divergence. Electron. J. Stat. 5:935-80
69. Recht B. 2011. A simpler approach to matrix completion. J. Mach. Learn. Res. 12:3413-30
70. Recht B, FazelM, Parrilo P. 2010. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52(3):471-501
71. Rohde A, Tsybakov A. 2011. Estimation of high-dimensional low-rank matrices. Ann. Stat. 39(2):887-930
72. Rothman AJ, Bickel PJ, Levina E, Zhu J. 2008. Sparse permutation invariant covariance estimation. Electron. J. Stat. 2:494-515
73. Rudelson M, Zhou S. 2012. Reconstruction from anisotropic random measurements. IEEE Trans. Inf. Theory 59:3434-47
74. Srebro N, Alon N, Jaakkola TS. 2005. Generalization error bounds for collaborative prediction with low-rank matrices. Presented at Neural Inf. Proc. Syst., 17th, Vancouver
75. Srebro N, Rennie J, Jaakkola TS. 2004. Maximum-margin matrix factorization. Presented at Neural Inf. Proc. Syst., 17th, Vancouver
76. StojnicM, Parvaresh F, Hassibi B. 2009. On the reconstruction of block-sparse signals with an optimal number of measurements. IEEE Trans. Signal Process. 57(8):3075-85
77. Stone CJ. 1982. Optimal global rates of convergence for non-parametric regression. Ann. Stat. 10(4):1040-53
78. Stone CJ. 1985. Additive regression and other non-parametric models. Ann. Stat. 13(2):689-705
79. Tibshirani R. 1996. Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B 58(1):267-88
80. Tikhonov AN. 1943. On the stability of inverse problems. C. R. (Doklady) Acad. Sci. SSSR 39:176-79
81. Tropp JA. 2006. Just relax: convex programming methods for identifying sparse signals in noise. IEEE Trans. Inf. Theory 52(3):1030-51
82. Tropp JA, Gilbert AC, Strauss MJ. 2006. Algorithms for simultaneous sparse approximation. Part I: greedy pursuit. Signal Process. 86:572-88
83. Tseng P. 2001. Convergence of block coordinate descent method for nondifferentiable maximization. J. Opt. Theory Appl. 109(3):474-94
84. Tseng P, Yun S. 2009. A block-coordinate gradient descent method for linearly constrained nonsmooth separable optimization. J. Optim. Theory Appl. 140:513-35
85. Turlach B, Venables WN,Wright SJ. 2005. Simultaneous variable selection. Technometrics 27:349-63
86. van de Geer S. 2000. Empirical Processes in M-Estimation. Cambridge, UK: Cambridge Univ. Press
87. van de Geer S. 2008. High-dimensional generalized linear models and the Lasso. Ann. Stat. 36:614-45
88. van de Geer S. 2012. Weakly decomposable regularization penalties and structured sparsity. Tech. Rep., ETH Zurich, Switz. http://arxiv1204.4813v2
89. van de Geer S, Buhlmann P. 2009. On the conditions used to prove oracle results for the Lasso. Electron. J. Stat. 3:1360-92
90. van de Geer S, Buhlmann P, Ritov Y. 2013. On asymptotically optimal confidence regions and tests for highdimensional models. Tech. Rep., ETH Zurich, Switz. http://arxiv1303.0518
91. WainwrightMJ. 2009. Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ1-constrained quadratic programming (Lasso). IEEE Trans. Inf. Theory 55:2183-202
92. Wu TT, Lange K. 2008. Coordinate descent algorithms for Lasso-penalized regression. Ann. Appl. Stat. 2(1):224-44
93. Xu H, Caramanis C, Sanghavi S. 2012. Robust PCA via outlier pursuit. IEEE Trans. Inf. Theory 58(5):3047-64
94. Yuan M. 2010. High-dimensional inverse covariance matrix estimation via linear programming. J. Mach. Learn. Res. 11:2261-86
95. Yuan M, Lin Y. 2006. Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. B 68:49-67
96. Zhang CH, Huang J. 2008. The sparsity and bias of the Lasso selection in high-dimensional linear regression. Ann. Stat. 36(4):1567-94
97. ZhangCH, Zhang SS. 2011. Confidence intervals for low-dimensional parameters with high-dimensional data. Tech. Rep., Rutgers Univ., New Brunswick, NJ. http://arxiv1110.2563
98. Zhang CH, ZhangT. 2012. A general theory of concave regularization for high-dimensional sparse estimation problems. Stat. Sci. 27(4):576-93
99. Zhao P, RochaG,Yu B. 2009. Grouped and hierarchical model selection through composite absolute penalties. Ann. Stat. 37(6A):3468-97
100. Zhao P, Yu B. 2006. On model selection consistency of Lasso. J. Mach. Learn. Res. 7:2541-67
101. Zhou S, Lafferty J, Wasserman L. 2008. Time-varying undirected graphs. Presented at Annu. Conf. Learn. Theory, 21st, Helsinki