Information-theoretic limits on sparsity recovery in the high-dimensional and noisy setting

IEEE Transactions on Information Theory

Volumn 55, Issue 12, 2009, Pages 5728-5741

  Wainwright, Martin J ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

1 relaxation; Compressed sensing; Fano's method; High dimensional statistical inference; Information theoretic bounds; Lasso; Model selection; Signal denoising; Sparsity pattern; Sparsity recovery; Subset selection; Support recovery

Indexed keywords

COMPRESSED SENSING; FANO'S METHOD; HIGH-DIMENSIONAL; INFORMATION THEORETIC BOUNDS; LASSO; MODEL SELECTION; SIGNAL DENOISING; SPARSITY PATTERNS; SPARSITY RECOVERY; SUBSET SELECTION;

COMPUTATIONAL COMPLEXITY; DECODING; INFORMATION THEORY; MATRIX ALGEBRA; NOISE POLLUTION CONTROL; SET THEORY; SIGNAL PROCESSING; SIGNAL RECONSTRUCTION; STATISTICAL METHODS;

RECOVERY;

EID: 73849097267     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2009.2032816     Document Type: Article

References (36)

1. S. Aeron, M. Zhao, and S. Venkatesh, "Information-theoretic bounds to sensing capacity of sensor networks under fixed snr,;quot; in Proc. IEEE Information Theory Workshop, San Diego, CA, Sep. 2007.
2. M. Akcakaya and V. Tarokh, "Shannon Theoretic Limits on Noisy Compressive Sampling,;quot; Harvard Univ., Cambridge, MA, Tech. Rep., Nov. 2007 [Online]. Available: arXiv:cs.IT:0711.0366
3. P. Bickel, Y. Ritov, and A. Tsybakov, "Simultaneous analysis of Lasso and Dantzig selector,;quot; Ann. Statist., to be published.
4. L. Birgé, "An alternative point of view on Lepski's method," in State of the Art in Probability and Statistics, ser. IMS Lecture Notes, no.37. Beachwood, OH: Inst. Math. Statist., 2001, pp. 113-133.
5. E. Candés and T. Tao,;quot;Decoding by linear programming,;quot; IEEE Trans.Inf. Theory, vol.51, no.12, pp. 4203-4215, Dec. 2005.
6. E. Candés and T. Tao, "The Dantzig selector: Statistical estimation when p is much larger νthan," Ann. Statist., vol.35, no.6, pp. 2313-2351, 2007.
7. S. Chen, D. L. Donoho, and M. A. Saunders, "Atomic decomposition by basis pursuit, " SIAM J. Sci. Comput., vol.20, no.1, pp. 33-61, 1998.
8. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
9. K. R. Davidson and S. J. Szarek, "Local operator theory, random matrices,and Banach spaces, " in Handbook of Banach Spaces. Amsterdan, The Netherlands: Elsevier, 2001, vol.1, pp. 317-336.
10. R. A. DeVore and G. G. Lorentz, Constructive Approximation. New York: Springer-Verlag, 1993.
11. D. Donoho, "Compressed sensing, " IEEE Trans. Inf. Theory, vol.52, no.4, pp. 1289-1306, Apr. 2006.
12. 1 norm solution is also the sparsest solution, " Commun. Pure and Appl. Math., vol.59, no.6, pp. 797-829, Jun. 2006. (Pubitemid 43667226)
13. 1norm near-solution approximates the sparsest near-solution," Commun. Pure and Appl. Math., vol.59, no.7, pp. 907-934, Jul. 2006.
14. D. L. Donoho and J. M. Tanner, "Counting faces of randomly-projected polytopes when the projection radically lowers dimension," J. Amer. Math. Soc., vol.22, pp. 1-53, Jul. 2009.
15. A. K. Fletcher, S. Rangan, and V. K. Goyal, "Necessary and Sufficient Conditions on Sparsity Pattern Recovery," Univ. Calif., Berkeley, Tech. Rep., Apr. 2008 [Online]. Available: arXiv:cs.IT/0804.1839
16. A. K. Fletcher, S. Rangan, V. K. Goyal, and K. Ramchandran, "Denoising by sparse approximation: Error bounds based on rate-distortion theory, " J. Appl. Signal Process., vol.10, pp. 1-19, 2006.
17. J. J. Fuchs,"Recovery of exact sparse representations in the presence of noise," in Proc. Int. Conf. Acoustics, Speech and Signal Processing, Montreal, QC, Canada, 2004, vol.2, pp. 533-536.
18. R. Z. Has'minskii,"A lower bound on the risks of nonparametric estimates of densities in the uniform metric," Theory Prob. Appl., vol.23, pp. 794-798, 1978.
19. R. A. Horn and C. R. Johnson, Matrix Analysis. Cambridge, U.K.: Cambridge Univ. Press, 1985.
20. I. A. Ibragimov and R. Z. Has'minskii, Statistical Estimation: Asymptotic Theory. New York: Springer-Verlag, 1981.
21. B. Laurent and P. Massart,"Adaptive estimation of a quadratic functional by model selection, " Ann. Statist., vol.28, no.5, pp. 1303-1338, 1998.
22. N. Meinshausen and P. Bühlmann, "High-dimensional graphs and variable selection with the Lasso, " Ann. Statist., vol.34, pp. 1436-1462, 2006.
23. N. Meinshausen and B. Yu, "Lasso-type recovery of sparse representations for high-dimensional data, " Ann. Statist., to be published.
24. A. J. Miller, Subset Selection in Regression. New York: Chapmanamp;Hall, 1990.
25. B. K. Natarajan,"Sparse approximate solutions to linear systems,"SIAM J. Comput., vol.24, no.2, pp. 227-234, 1995.
26. G. Reeves and M. Gastpar, "Sampling bounds for sparse support recoveryin the presence of noise, " in Proc. IEEE Int. Symp. Information Theory, Toronto, ON, Canada, Jul. 2008, pp. 2187-2191.
27. S. Sarvotham, D. Baron, and R. G. Baraniuk, "Measurements versus bits: Compressed sensing meets information theory," in Proc. Allerton Conf. Communication, Control, and Computing, Monticello, IL, Sep. 2006.
28. R. Tibshirani, "Regression shrinkage and selection via the lasso," J. Roy. Statist. Soc., ser. B, vol.58, no.1, pp. 267-288, 1996.
29. J. Tropp,"Just relax: Convex programming methods for identifying sparse signals in noise," IEEE Trans. Inf. Theory, vol.52, no.3, pp. 1030-1051, Mar. 2006.
30. M. J. Wainwright, "Information-Theoretic Bounds for Sparsity Recovery in the High-Dimensional and Noisy Setting," Dep. Statist., Univ. Calif., Berkeley, Tech. Rep. 725, Jan. 2007 [Online]. Available: arxiv:math.ST/0702301, presented at the IEEE Int. Symp. Information Theory, Nice, France, June 2007
31. 1 -constrained quadratic programming (Lasso)," IEEE Trans. Inf. Theory, vol.55, no.5, pp. 2183-2202, May 2009.
32. W.Wang, M. J.Wainwright, and K. Ramchandran,"Information-Theoretic Limits on Sparse Signal Recovery: Dense Versus Sparse Measurement Matrices," Univ. Calif., Berkeley, Tech. Rep., June 2008 [Online]. Available: arXiv:0806.0604, presented at the IEEE Int. Symp. Information Theory, Toronto, ON, Canada, Jul. 2008
33. L. Wasserman and K. Roeder, "Multi-stage variable selection: Screen and clean," Ann. Statist., to be published.
34. Y. Yang and A. Barron,"Information-theoretic determination of minimax rates of convergence,;quot; Ann. Statist., vol.27, no.5, pp. 1564-1599, 1999.
35. B.Yu. Assouad, Fano, and Le Cam, In Festschrift for Lucien Le Cam. Berlin, Germany: Springer-Verlag, 1997, pp. 423-435.
36. P. Zhao and B.Yu,"On model selection consistency of Lasso;quot; J. Mach.Learn. Res., vol.7, pp. 2541-2567, 2006.