Fast global convergence of gradient methods for high-dimensional statistical recovery

Annals of Statistics

Volumn 40, Issue 5, 2012, Pages 2452-2482

  Agarwal, Alekh ^a   Negahban, Sahand ^b   Wainwright, Martin J ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Convex optimization; High dimensional inference; Regularized Mestimation

Indexed keywords

EID: 84873371070     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/12-AOS1032     Document Type: Article

References (45)

1. AGARWAL, A., NEGAHBAN, S. and WAINWRIGHT, M. J. (2012). Supplement to "Fast global convergence of gradient methods for high-dimensional statistical recovery." DOI:10.1214/12-AOS1032SUPP
2. AGARWAL, A.,NEGAHBAN, S. and WAINWRIGHT, M. J. (2012). Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions. Ann. Statist. 40 1171-1197.
3. AMINI, A. A. and WAINWRIGHT, M. J. (2009). High-dimensional analysis of semidefinite relaxations for sparse principal components. Ann. Statist. 37 2877-2921. MR2541450
4. BECK, A. and TEBOULLE, M. (2009). A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2 183-202. MR2486527
5. BECKER, S., BOBIN, J. and CANDÈS, E. J. (2011). Nesta: A fast and accurate first-order method for sparse recovery. SIAM J. Imaging Sci. 4 1-39. MR2765668
6. BERTSEKAS, D. P. (1995). Nonlinear Programming. Athena Scientific, Belmont, MA
7. BICKEL, P. J., RITOV, Y. and TSYBAKOV, A. B. (2009). Simultaneous analysis of lasso and Dantzig selector. Ann. Statist. 37 1705-1732. MR2533469
8. BOYD, S. and VANDENBERGHE, L. (2004). Convex Optimization. Cambridge Univ. Press, Cambridge. MR2061575
9. BREDIES, K. and LORENZ, D. A. (2008). Linear convergence of iterative soft-thresholding. J. Fourier Anal. Appl. 14 813-837. MR2461608
10. CANDÈS, E. J., LI, X.,MA, Y. and WRIGHT, J. (2011). Robust principal component analysis? J. ACM 58 Art. 11, 37. MR2811000
11. CANDÈS, E. J. and RECHT, B. (2009). Exact matrix completion via convex optimization. Found. Comput. Math. 9 717-772. MR2565240
12. CHANDRASEKARAN, V., SANGHAVI, S., PARRILO, P. A. and WILLSKY, A. S. (2011). Ranksparsity incoherence for matrix decomposition. SIAM J. Optim. 21 572-596. MR2817479
13. CHEN, S. S.,DONOHO, D. L. and SAUNDERS, M. A. (1998). Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20 33-61. MR1639094
14. DUCHI, J., SHALEV-SHWARTZ, S., SINGER, Y. and CHANDRA, T. (2008). Efficient projections onto the -1-ball for learning in high dimensions. In ICML. Omnipress, Helsinki, Finland.
15. FAZEL, M. (2002). Matrix rank minimization with applications. Ph.D. thesis, Stanford. Available at http://faculty.washington.edu/mfazel/thesis- final.pdf.
16. GARG, R. and KHANDEKAR, R. (2009). Gradient descent with sparsification: An iterative algorithm for sparse recovery with restricted isometry property. In ICML. Omnipress, Montreal, Canada.
17. HALE, E. T., YIN, W. and ZHANG, Y. (2008). Fixed-point continuation for l1-minimization: Methodology and convergence. SIAM J. Optim. 19 1107-1130. MR2460734
18. HSU, D., KAKADE, S. M. and ZHANG, T. (2011). Robust matrix decomposition with sparse corruptions. IEEE Trans. Inform. Theory 57 7221-7234. MR2883652
19. HUANG, J. and ZHANG, T. (2010). The benefit of group sparsity. Ann. Statist. 38 1978-2004. MR2676881
20. JI, S. and YE, J. (2009). An accelerated gradient method for trace norm minimization. In ICML. Omnipress, Montreal, Canada.
21. KOLTCHINSKII, V., LOUNICI, K. and TSYBAKOV, A. B. (2011). Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. Ann. Statist. 39 2302-2329. MR2906869
22. LEE, K. and BRESLER, Y. (2009). Guaranteed minimum rank approximation from linear observations by nuclear norm minimization with an ellipsoidal constraint. Technical report, UIUC. Available at arXiv:0903.4742.
23. LOH, P. and WAINWRIGHT, M. J. (2012). High-dimensional regression with noisy and missing data: Provable guarantees with non-convexity. Ann. Statist. 40 1637-1664.
24. LOUNICI, K., PONTIL, M., TSYBAKOV, A. B. and VAN DE GEER, S. (2009). Taking advantage of sparsity in multi-task learning. In COLT. Omnipress, Montreal, Canada.
25. LUO, Z.-Q. and TSENG, P. (1993). Error bounds and convergence analysis of feasible descent methods: A general approach. Ann. Oper. Res. 46/47 157-178. MR1260016
26. MEINSHAUSEN, N. and BÜHLMANN, P. (2006). High-dimensional graphs and variable selection with the lasso. Ann. Statist. 34 1436-1462. MR2278363
27. NEGAHBAN, S., RAVIKUMAR, P.,WAINWRIGHT, M. J. and YU, B. (2009). A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers. In NIPS. MIT Press, Vancouver, Canada.
28. NEGAHBAN, S. and WAINWRIGHT, M. J. (2011). Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Ann. Statist. 39 1069-1097. MR2816348
29. NEGAHBAN, S. and WAINWRIGHT, M. J. (2012). Restricted strong convexity and (weighted) matrix completion: Optimal bounds with noise. J. Mach. Learn. Res. 13 1665-1697.
30. NESTEROV, Y. (2004). Introductory Lectures on Convex Optimization: A Basic Course. Applied Optimization 87. Kluwer Academic, Boston, MA. MR2142598
31. NESTEROV, Y. (2007). Gradient methods for minimizing composite objective function. Technical Report 76, Center for Operations Research and Econometrics (CORE), Catholic Univ. Louvain (UCL).
32. NGAI, H. V. and PENOT, J.-P. (2008). Paraconvex functions and paraconvex sets. Studia Math. 184 1-29. MR2365473
33. RASKUTTI, G.,WAINWRIGHT, M. J. and YU, B. (2010). Restricted eigenvalue properties for correlated Gaussian designs. J. Mach. Learn. Res. 11 2241-2259. MR2719855
34. RASKUTTI, G., WAINWRIGHT, M. J. and YU, B. (2011). Minimax rates of estimation for high-dimensional linear regression over -q -balls. IEEE Trans. Inform. Theory 57 6976- 6994. MR2882274
35. RECHT, B. (2011). A simpler approach to matrix completion. J. Mach. Learn. Res. 12 3413- 3430. MR2877360
36. RECHT, B., FAZEL, M. and PARRILO, P. A. (2010). Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52 471-501. MR2680543
37. ROHDE, A. and TSYBAKOV, A. B. (2011). Estimation of high-dimensional low-rank matrices. Ann. Statist. 39 887-930. MR2816342
38. RUDELSON, M. and ZHOU, S. (2011). Reconstruction from anisotropic random measurements. Technical report, Univ. Michigan.
39. SREBRO, N., ALON, N. and JAAKKOLA, T. S. (2005). Generalization error bounds for collaborative prediction with low-rank matrices. In NIPS. MIT Press, Vancouver, Canada.
40. TIBSHIRANI, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267-288. MR1379242
41. TROPP, J. A. and GILBERT, A. C. (2007). Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans. Inform. Theory 53 4655-4666. MR2446929
42. VAN DE GEER, S. A. and BÜHLMANN, P. (2009). On the conditions used to prove oracle results for the Lasso. Electron. J. Stat. 3 1360-1392. MR2576316
43. XU, H., CARAMANIS, C. and SANGHAVI, S. (2012). Robust PCA via outlier pursuit. IEEE Trans. Inform. Theory 58 3047-3064. MR2952532
44. ZHANG, C.-H. and HUANG, J. (2008). The sparsity and bias of the LASSO selection in highdimensional linear regression. Ann. Statist. 36 1567-1594. MR2435448
45. ZHAO, P., ROCHA, G. and YU, B. (2009). Grouped and hierarchical model selection through composite absolute penalties. Ann. Statist. 37 3468-3497.