Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ1-constrained quadratic programming (Lasso)

IEEE Transactions on Information Theory

Volumn 55, Issue 5, 2009, Pages 2183-2202

  Wainwright, Martin J ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

1 constraints; Compressed sensing; Convex relaxation; High dimensional inference; Model selection; Phase transitions; Signal denoising; Sparse approximation; Subset selection

Indexed keywords

COMPRESSED SENSING; CONVEX RELAXATION; HIGH-DIMENSIONAL INFERENCE; MODEL SELECTION; SIGNAL DENOISING; SPARSE APPROXIMATION; SUBSET SELECTION;

PHASE TRANSITIONS; RELAXATION PROCESSES; SIGNAL PROCESSING; SIGNAL RECONSTRUCTION;

RECOVERY;

EID: 65749083666     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2009.2016018     Document Type: Article

References (37)

1. D. P. Bertsekas, Nonlinear Programming. Belmont, MA: Athena Scientific, 1995.
2. V. V. Buldygin and Y. V. Kozachenko, Metric Characterization of Random Variables and Random Processes. Providence, RI: Amer. Math. Soc., 2000.
3. E. Candes, J. Romberg, and T. Tao, "Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information," IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489-509, Feb. 2004.
4. E. Candes and T. Tao, "Decoding by linear programming," IEEE Trans. Inf. Theory, vol. 51, no. 12, pp. 4203-4215, Dec. 2005.
5. E. Candes and T. Tao, "The dantzig selector: Statistical estimation when p is much larger than n," Ann. Statist., vol. 35, no. 6, pp. 2313-2351, 2007.
6. S. Chen, D. L. Donoho, and M. A. Saunders, "Atomic decomposition by basis pursuit," SIAM J. Sci. Comput., vol. 20, no. 1, pp. 33-61, 1998.
7. K. R. Davidson and S. J. Szarek, "Local operator theory, random matrices, and Banach spaces," in Handbook of Banach Spaces. Amsterdan, The Netherlands: Elsevier, 2001, vol. 1, pp. 317-336.
8. R. A. DeVore and G. G. Lorentz, Constructive Approximation. New York: Springer-Verlag, 1993.
9. D. Donoho, "Compressed sensing," IEEE Trans. Inf.. Theory, vol. 52, no. 4, pp. 1289-1306, Apr. 2006.
10. 1-norm solution is also the sparsest solution," Commun. Pure and Appl. Math., vol. 59, no. 6, pp. 797-829, Jun. 2006.
11. 1-norm near-solution approximates the sparsest near-solution," Commun. Pure and Appl. Math., vol. 59, no. 7, pp. 907-934, Jul. 2006.
12. D. L. Donoho, M. Elad, and V. M. Temlyakov, "Stable recovery of sparse overcomplete representations in the presence of noise," IEEE Trans. Inf. Theory, vol. 52, no. 1, pp. 6-18, Jan. 2006.
13. D. L. Donoho and J. M. Tanner, "Counting faces of randomly-projected polytopes when the projection radically lowers dimension," J. Amer. Math. Soc., Jul. 2008.
14. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, "Least angle regression," Ann. Statist., vol. 32, no. 2, pp. 407-499, 2004.
15. M. Elad and A. M. Bruckstein, "A generalized uncertainty principle and sparse representation in pairs of bases," IEEE Trans. Inf. Theory vol. 48, no. 9, pp. 2558-2567, Sep. 2002.
16. J. Fan and R. Li, "Variable selection via non-concave penalized likelihood and its oracle properties," J. Amer. Stat. Assoc., vol. 96, no. 456, pp. 1348-1360, Dec. 2001.
17. A. Feuer and A. Nemirovski, "On sparse representation in pairs of bases," IEEE Trans. Inf. Theory, vol. 49, no. 6, pp. 1579-1581, Jun. 2003.
18. J. J. Fuchs, "Recovery of exact sparse representations in the presence of noise," IEEE Trans. Inf.. Theory, vol. 51, no. 10, pp. 3601-3608, Oct. 2005.
19. R. M. Gray, "Toeplitz and Circulant Matrices: A Review," Stanford Univ., Information Systems Lab., Tech. Rep., 1990.
20. J. Hiriart-Urruty and C. Lemaréchal, Convex Analysis and Minimization Algorithms. New York: Springer-Verlag, 1993, vol. 1.
21. I. Johnstone, "Chi-square oracle inequalities," in State of the Art in Probability and Statistics, I. M. de Gunst, C. Klaassen, and A. de van Vaart, Eds. Beachwood, OH: Inst. Math. Statist., 2001, vol. 37, IMS Lecture Notes, pp. 399-418.
22. K. Knight and W. J. Fu, "Asymptotics for lasso-type estimators," Ann. Statist., vol. 28, pp. 1356-1378, 2000.
23. B. Laurent and P. Massart, "Adaptive estimation of a quadratic functional by model selection," Ann. Statist., vol. 28, no. 5, pp. 1303-1338, 1998.
24. M. Ledoux, The Concentration of Measure Phenomenon, ser. Mathematical Surveys and Monographs. Providence, RI: Amer. Math. Soc., 2001.
25. M. Ledoux and M. Talagrand, Probability in Banach Spaces: Isoperimetry and Processes. New York: Springer-Verlag, 1991.
26. D. M. Malioutov, M. Cetin, and A. S. Willsky, "Optimal sparse representations in general overcomplete bases," in Proc. Int. Conf. Acoustics, Speech, and Signal Processing, Montreal, QC, Canada, May 2004, vol. 2, pp. II-793.
27. N. Meinshausen and P. Bühlmann, "High-dimensional graphs and variable selection with the Lasso," Ann. Stat., vol. 34, pp. 1436-1462, 2006.
28. A. J. Miller, Subset Selection in Regression. New York: Chapman&Hall, 1990.
29. B. K. Natarajan, "Sparse approximate solutions to linear systems," SIAM J. Comput., vol. 24, no. 2, pp. 227-234, 1995.
30. R. Tibshirani, "Regression shrinkage and selection via the lasso," J. Roy. Statisti. Soc., Ser. B, vol. 58, no. 1, pp. 267-288, 1996.
31. J. Tropp, "Greed is good: Algorithmic results for sparse approximation," IEEE Trans. Inf. Theory, vol. 50, no. 10, pp. 2231-2242, Oct. 2004.
32. J. Tropp, "Just relax: Convex programming methods for identifying sparse signals in noise," IEEE Trans. Inf. Theory, vol. 52, no. 3, pp. 1030-1051, Mar. 2006.
33. 1-Constrained Quadratic Programs," Dep. Statistics, Univ. Calif. Berkeley, Berkeley, Tech. Rep. 709, 2006.
34. M. J. Wainwright, "Information-Theoretic Bounds for Sparsity Recovery in the High-Dimensional and Noisy Setting," Dep. Statistics, Univ. Calif. Berkeley, Berkeley, Tech. Rep. 725, 2007.
35. 1-regularized logistic regression," in Proc. NIPS Conf., Vancouver, BC, Canada, Dec. 2006.
36. W. Wang, M. J. Wainwright, and K. Ramchandran, "Information-Theoretic Limits on Sparse Signal Recovery: Dense Versus Sparse Measurement Matrices," Univ. Calif. Berkeley, Berkeley, Tech. Rep. arXiv:0806.0604, 2008, Presented at the IEEE Int. Symp. Inf. Theory, Toronto, ON, Canada, Jul. 2008.
37. P. Zhao and B.Yu, "On model selection consistency of Lasso," J. Mach. Learning Res., vol. 7, pp. 2541-2567, 2006.