Optimal phase transitions in compressed sensing

IEEE Transactions on Information Theory

Volumn 58, Issue 10, 2012, Pages 6241-6263

  Wu, Yihong ^a   Verdu, Sergio ^b  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b Princeton University   (United States)

Author keywords

Compressed sensing; joint source channel coding; minimum mean square error (MMSE) dimension; phase transition; random matrix; R nyi information dimension; Shannon theory

Indexed keywords

COMPRESSIVE SENSING; INFORMATION DIMENSIONS; JOINT SOURCE-CHANNEL CODING; MINIMUM MEAN-SQUARE ERROR; RANDOM MATRIX; SHANNON THEORY;

MATRIX ALGEBRA; PHASE TRANSITIONS; SIGNAL RECONSTRUCTION;

OPTIMIZATION;

EID: 84866494005     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2012.2205894     Document Type: Article

References (76)

1. Y. Wu, MMSE Dimension and Noisy Compressed Sensing, Poster in Third Annual School of Information Theory. Los Angeles, CA: Univ. Southern California, 2010.
2. Y. Wu and S. Verdú, "Optimal phase transitions in compressed sensing with noisy measurements," presented at the IEEE Int. Symp. Inf. Theory, 2012.
3. E. Candés, J. Romberg, and T. Tao, "Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information," IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489-509, Feb. 2006. (Pubitemid 43174093)
4. D. L. Donoho, "Compressed sensing," IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289-1306, Apr. 2006.
5. E. Candés and T. Tao, "Near optimal signal recovery from random projections: Universal encoding strategies?," IEEE Trans. Inf. Theory, vol. 52, no. 12, pp. 5406-5425, Dec. 2006. (Pubitemid 46445381)
6. S. S. Chen,D. L. Donoho, andM. A. Saunders, "Atomic decomposition by basis pursuit," SIAM J. Sci. Comput. , vol. 20, no. 1, pp. 33-61, 1999.
7. R. Tibshirani, Regression shrinkage and selection via the LASSO, vol. 58, no. 1, pp. 267-288, 1996.
8. S. G. Mallat and Z. Zhang, "Matching pursuits with time-frequency dictionaries," IEEE Trans. Signal Process. , vol. 41, no. 12, pp. 3397-3415, Dec. 1993.
9. D. L. Donoho, A. Maleki, and A. Montanari, "Message-passing algorithms for compressed sensing," Proc. Nat. Acad. Sci. , vol. 106, no. 45, pp. 18914-18919, Nov. 2009.
10. A. Beck and M. Teboulle, "A fast iterative shrinkage-thresholding algorithm for linear inverse problems," SIAM J. Imag. Sci. , vol. 2, no. 1, pp. 183-202, 2009.
11. D. L. Donoho and J. Tanner, "Counting faces of randomly-projected polytopes when the projection radically lowers dimension," J. Amer. Math. Soc. , vol. 22, no. 1, pp. 1-53, 2009.
12. Y. Wu and S. Verdú, "Rényi information dimension: Fundamental limits of almost lossless analog compression," IEEE Trans. Inf. Theory, vol. 56, no. 8, pp. 3721-3748, Aug. 2010.
13. D. L. Donoho, A. Maleki, and A. Montanari, "The noise-sensitivity phase transition in compressed sensing," IEEE Trans. Inf. Theory, vol. 57, no. 10, pp. 6920-6941, Oct. 2011.
14. D. Guo, D. Baron, and S. Shamai, "A single-letter characterization of optimal noisy compressed sensing," in Proc. 47 Annu. Allerton Conf. Commun. , Control, Comput. , Monticello, IL, Oct. 2009, pp. 52-59.
15. G. Reeves and M. Gastpar, "Sampling bounds for sparse support recovery in the presence of noise," in Proc. 2008 IEEE Int. Symp. Inf. Theory, Toronto, ON, Canada, Jul. 2008, pp. 2187-2191.
16. G. Reeves and M. Gastpar, "The sampling rate-distortion tradeoff for sparsity pattern recovery in compressed sensing," IEEE Trans. Inf. Theory, vol. 58, no. 5, pp. 3065-3092, May 2011.
17. Y. Kabashima, T. Wadayama, and T. Tanaka, "Statistical mechanical analysis of a typical reconstruction limit of compressed sensing," in Proc. IEEE Int. Symp. Inf. Theory, Austin, TX, Jun. 2010, pp. 1533-1537.
18. F. Krzakala, M. Mézard, F. Sausset, Y. Sun, and L. Zdeborová, Statistical physics-based reconstruction in compressed sensing Nov. 2011 [Online]. Available: http://arxiv.org/abs/1109.4424
19. A. M. Tulino, G. Caire, S. Shamai, and S. Verdú, "Support recovery with sparsely sampled free random matrices," in Proc. IEEE Int. Symp. Inf. Theory, Saint Petersburg, Russia, Aug. 2011, pp. 2328-2332.
20. M. J. Wainwright, "Information-theoretic limitations on sparsity recovery in the high-dimensional and noisy setting," IEEE Trans. Inf. Theory, vol. 55, no. 12, pp. 5728-5741, Dec. 2009.
21. A. K. Fletcher, S. Rangan, and V. K. Goyal, "Necessary and sufficient conditions for sparsity pattern recovery," IEEE Trans. Inf. Theory, vol. 55, no. 12, pp. 5758-5772, Dec. 2009.
22. M. Akçakaya and V. Tarokh, "Shannon-theoretic limits on noisy compressive sampling," IEEE Trans. Inf. Theory, vol. 56, no. 1, pp. 492-504, Jan. 2010.
23. M. Lustig, D. Donoho, J. Santos, and J. Pauly, "Compressed sensing MRI," IEEE Signal Process. Mag. , vol. 25, no. 2, pp. 72-82, Mar. 2008.
24. M. A. Herman and T. Strohmer, "High-resolution radar via compressed sensing," IEEE Trans. Signal Process. , vol. 57, no. 6, pp. 2275-2284, Jun. 2009.
25. D. L. Donoho and J. Tanner, "Counting the faces of randomly-projected hypercubes and orthants, with applications," Discrete Comput. Geom. , vol. 43, no. 3, pp. 522-541, 2010.
26. A. Rényi, Probability Theory. Amsterdam, The Netherlands: North Holland, 1970.
27. D. L. Donoho and J. Tanner, "Precise undersampling theorems," Proc. IEEE, vol. 98, no. 6, pp. 913-924, Jun. 2010.
28. A. Rényi, "On the dimension and entropy of probability distributions," Acta Math. Acad. Sci. Hungar. , vol. 10, no. 1-2, pp. 193-215, Mar. 1959.
29. Y. Wu and S. Verdú, "MMSE dimension," IEEE Trans. Inf. Theory, vol. 57, no. 8, pp. 4857-4879, Aug. 2011.
30. Y. Peres and B. Solomyak, "Existence of dimensions and entropy dimension for self-conformal measures," Indiana Univ. Math. J. , vol. 49, no. 4, pp. 1603-1621, 2000.
31. B. R. Hunt and V. Y. Kaloshin, "How projections affect the dimension spectrum of fractal measures," Nonlinearity, vol. 10, pp. 1031-1046, 1997. (Pubitemid 127431071)
32. T. Kawabata and A. Dembo, "The rate-distortion dimension of sets and measures," IEEE Trans. Inf. Theory, vol. 40, no. 5, pp. 1564-1572, Sep. 1994.
33. Y. Wu and S. Verdú, "Functional properties of MMSE and mutual information," IEEE Trans. Inf. Theory, vol. 58, no. 3, pp. 1289-1301, Mar. 2012.
34. A. Guionnet and D. Shlyakhtenko, "On classical analogues of free entropy dimension," J. Funct. Anal. , vol. 251, no. 2, pp. 738-771, Oct. 2007.
35. K. Falconer, Fractal Geometry:Mathematical Foundations and Applications, 2nd ed. New York: Wiley, 2003.
36. Y. B. Pesin, Dimension Theory in Dynamical Systems: Contemporary Views and Applications. Chicago, IL: Univ. Chicago Press, 1997.
37. B. R. Hunt and V. Y. Kaloshin, "Regularity of embeddings of infinitedimensional fractal sets into finite-dimensional spaces," Nonlinearity, vol. 12, no. 5, pp. 1263-1275, 1999.
38. H. Federer, Geometric Measure Theory. New York: Springer-Verlag, 1969.
39. P. Feng and Y. Bresler, "Spectrum-blind minimum-rate sampling and reconstruction of multiband signals," in Proc. IEEE Int. Conf. Acoust. , Speech, Signal Process. , 1996, vol. 3, pp. 1688-1691.
40. J. T. Schwartz, Nonlinear Functional Analysis. New York: Gordon and Breach, 1969.
41. K. Do Ba, P. Indyk, E. Price, and D. P. Woodruff, "Lower bounds for sparse recovery," in Proc. 21 Annu. ACM-SIAM Symp. Discrete Algorithms, 2010, pp. 1190-1197.
42. C. E. Shannon, "Communication in the presence of noise," Proc. IRE, vol. 37, no. 1, pp. 10-21, Jan. 1949.
43. J. L. Massey, "Joint source channel coding," in Communication Systems and Random Process Theory, J. Skwirzynski, Ed. Amsterdam, The Netherlands: Sijthoff and Nordhoff.
44. R. Zamir, "A proof of the Fisher information inequality via a data processing argument," IEEE Trans. Inf. Theory, vol. 44, no. 3, pp. 1246-1250, May 1998. (Pubitemid 128743284)
45. S. P. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, U. K. : Cambridge Univ. Press, 2004.
46. T. Goblick, Jr. , "Theoretical limitations on the transmission of data from analog sources," IEEE Trans. Inf. Theory, vol. 11, no. 4, pp. 558-567, Oct. 1965.
47. J. Ziv, "The behavior of analog communication systems," IEEE Trans. Inf. Theory, vol. 16, no. 5, pp. 587-594, Sep. 1970.
48. M. Gastpar, B. Rimoldi, andM. Vetterli, "To code, or not to code: Lossy source-channel communication revisited," IEEE Trans. Inf. Theory, vol. 49, no. 5, pp. 1147-1158, May 2003.
49. T. Berger, Rate Distortion Theory: A Mathematical Basis for Data Compression. Englewood Cliffs, NJ: Prentice-Hall, 1971.
50. U. Brehm, "Extensions of distance reducing mappings to piecewise congruent mappings on ," J. Geom. , vol. 16, no. 1, pp. 187-193, 1981.
51. A. V. Akopyan and A. S. Tarasov, "A constructive proof of Kirszbraun's theorem," Matematicheskie Zametki, vol. 84, no. 5, pp. 781-784, 2008.
52. T. Tanaka, "A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors," IEEE Trans. Inf. Theory, vol. 48, no. 11, pp. 2888-2910, Nov. 2002. (Pubitemid 35327603)
53. D. Guo and S. Verdú, "Randomly spread CDMA: Asymptotics via statistical physics," IEEE Trans. Inf. Theory, vol. 51, no. 6, pp. 1983-2010, Jun. 2005. (Pubitemid 40843621)
54. D. Guo, S. Shamai, and S. Verdú, "Mutual information and minimum mean-square error in Gaussian channels," IEEE Trans. Inf. Theory, vol. 51, no. 4, pp. 1261-1283, Apr. 2005.
55. D. Guo and C. C. Wang, "Multiuser detection of sparsely spread CDMA," IEEE J. Sel. Areas Commun. , vol. 26, no. 3, pp. 421-431, Apr. 2008. (Pubitemid 351540618)
56. A. Montanari, "Estimating random variables from random sparse observations," Eur. Trans. Telecommun. , vol. 19, no. 4, pp. 385-403, 2008.
57. H. E. Stanley, Introduction to Phase Transitions and Critical Phenomena. London, U. K. : Oxford Univ. Press, 1971.
58. S. Rangan, A. K. Fletcher, and V. K. Goyal, "Asymptotic analysis of map estimation via the replica method and applications to compressed sensing," IEEE Trans. Inf. Theory, vol. 58, no. 3, pp. 1902-1923, Mar. 2012.
59. M. Bayati and A. Montanari, "The LASSO risk for Gaussianmatrices, " IEEE Trans. Inf. Theory, vol. 58, no. 4, pp. 1997-2017, Apr. 2012.
60. A. Maleki, L. Anitori, Z. Yang, andR. G. Baraniuk, "Asymptotic analysis of complex LASSO via complex approximate message passing (CAMP)," IEEE Trans. Inf. Theory, http://arxiv.org/abs/1108.0477, to be published.
61. J. D. Blanchard, C. Cartis, J. Tanner, and A. Thompson, "Phase transitions for greedy sparse approximation algorithms," Appl. Comput. Harmon. Anal. , vol. 30, pp. 188-203, 2011.
62. D. L. Donoho, I. Johnstone, and A. Montanari, "Accurate prediction of phase transitions in compressed sensing via a connection to minimax denoising," IEEE Trans. Inf. Theory 2011, http://arxiv.org/abs/1111.1041, submitted for publication.
63. M. Bayati and A. Montanari, "The dynamics of message passing on dense graphs, with applications to compressed sensing," IEEE Trans. Inf. Theory, vol. 57, no. 2, pp. 764-785, Feb. 2011.
64. A. Montanari, "Graphical models concepts in compressed sensing," in Compressed Sensing: Theory and Applications. Cambridge, U. K. : Cambridge Press, 2012, pp. 394-438.
65. A. Edelman, "Eigenvalues and condition numbers of random matrices," SIAM J. Matrix Anal. Appl. , vol. 9, no. 4, pp. 543-560, Oct. 1988.
66. M. Rudelson and R. Vershynin, "Smallest singular value of a random rectangular matrix," Commun. Pure Appl. Math. , vol. 62, pp. 1707-1739, 2009.
67. P. Mattila, Geometry of Sets and Measures in Euclidean Spaces: Fractals and Rectifiability. Cambridge, U. K. : Cambridge Univ. Press, 1999.
68. H. Steinhaus, "Sur les distances des points des ensembles de mesure positive," Fundamenta Mathematicae, vol. 1, pp. 93-104, 1920.
69. A. György, T. Linder, and K. Zeger, "On the rate-distortion function of random vectors and stationary sources with mixed distributions," IEEE Trans. Inf. Theory, vol. 45, no. 6, pp. 2110-2115, Sep. 1999. (Pubitemid 32077641)
70. D. Guo, Y. Wu, S. Shamai, and S. Verdú, "Estimation in Gaussian noise: Properties of the minimum mean-square error," IEEE Trans. Inf. Theory, vol. 57, no. 4, pp. 2371-2385, Apr. 2011.
71. D. L. Donoho,A. Javanmard, and A. Montanari, "Information- theoretically optimal compressed sensing via spatial coupling and approximate message passing," IEEE Trans. Inf. Theory, http://arxiv.org/abs/1112.0708, submitted for publication.
72. S. Kudekar and H. D. Pfister, "The effect of spatial coupling on compressive sensing," in Proc. 48 Annu. Allerton Conf. Commun. , Control, Comput. , 2010, pp. 347-353.
73. D. L. Donoho, I. Johnstone, A. Maleki, and A. Montanari, "Compressed sensing over -balls: Minimax mean square error," Ann. Statist. 2011.
74. D. L. Donoho, A. Maleki, and A. Montanari, How to design message passing algorithms for compressed sensing 2011 [Online]. Available: http://www.ece.rice. edu/mam15/bpist.pdf
75. K. H. Lee and D. P. Petersen, "Optimal linear coding for vector channels," IEEE Trans. Commun. , vol. 24, no. 12, pp. 1283-1290, Dec. 1976.
76. A. M. Tulino and S. Verdú, "Random matrix theory and wireless communications," J. Commun. Inf. Theory, vol. 1, no. 1, pp. 1-182, Jun. 2004.