Accurate prediction of phase transitions in compressed sensing via a connection to minimax denoising

IEEE Transactions on Information Theory

Volumn 59, Issue 6, 2013, Pages 3396-3433

  Donoho, David L ^a   Johnstone, Iain ^a   Montanari, Andrea ^a  

^a Stanford University   (United States)

Author keywords

Approximate message passing (AMP); group Lasso; James Stein; joint sparsity; Lasso; minimax risk of firm thresholding; minimax risk of soft thresholding; minimax risk over nearly black objects; minimax shrinkage; monotone regression; nonconvex penalization; state evolution; total variation minimization

Indexed keywords

APPROXIMATE MESSAGE PASSING (AMP); GROUP LASSOS; JAMES-STEIN; JOINT SPARSITY; LASSO; MINIMAX; MINIMAX RISK; MONOTONE REGRESSION; NONCONVEX; SOFT THRESHOLDING; STATE EVOLUTIONS; THRESHOLDING; TOTAL VARIATION MINIMIZATION;

INDUSTRY; SHRINKAGE;

SIGNAL RECONSTRUCTION;

EID: 84877914996     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2013.2239356     Document Type: Article

References (72)

1. P. Bickel and J. Collins, "Minimizing Fisher information overmixtures of distributions," Sankhya A, vol. 45, pp. 1-19, 1983.
2. M. J. Best and N. Chakravarti, "Active set algorithms for isotonic regression; A unifying framework," Math. Programm., vol. 47, pp. 425-439, 1990.
3. R. G. Baraniuk, V. Cevher,M. F. Duarte, and C. Hegde, "Model-based compressive sensing," IEEE Trans. Inf. Theory, vol. 56, no. 4, pp. 1982-2001, Apr. 2010.
4. J. D. Blanchard, C. Cartis, and J. Tanner, "Compressed sensing: How sharp is the restricted isometry property?," SIAM Rev., vol. 53, no. 1, pp. 105-125, 2011.
5. R. Berinde, A. C. Gilbert, P. Indyk, H. Karloff, and M. J. Strauss, "Combining geometry and combinatorics: A unified approach to sparse signal recovery," in 47th Annu. Allerton Conf., Monticello, IL, USA, Sep. 2008, pp. 798-805.
6. P. Bickel, "Minimax estimation of the mean of a normal distribution when the parameter space is restricted," Ann. Statist., vol. 9, pp. 1301-1309, 1981.
7. L. D. Brown, I. M. Johnstone, and K. B. MacGibbon, "Variation diminishing transformations: A direct approach to total positivity and its statistical applications," J. Amer. Statist. Assoc., vol. 76, no. 376, pp. 824-832, 1981.
8. M. Bayati, M. Lelarge, and A. Montanari, Universality in Polytope Phase Transitions and Message Passing Algorithms 2012 [Online]. Available: arXiv:1207.7321
9. M. Bayati and A. Montanari, "The dynamics of message passing on dense graphs, with applications to compressed sensing," IEEE Trans. Inf. Theory, vol. 57, no. 2, pp. 764-785, Feb. 2011.
10. M. Bayati and A.Montanari, "The LASSO risk for Gaussianmatrices, " IEEE Trans. Inf. Theory, vol. 58, no. 4, pp. 1997-2017, Apr. 2012.
11. P. J. Bickel, Y. Ritov, and A. B. Tsybakov, "Simultaneous analysis of Lasso and Dantzig selector," Amer. J. Math., vol. 37, pp. 1705-1732, 2009.
12. J. Blanchard and J. Tanner, "Large scale iterative hard thresholding on a graphical processing unit," in Proc. Int. Conf. Numer. Anal. Appl. Math., 2010, vol. 1281, pp. 1730-1734.
13. T. T. Cai, "Adaptive wavelet estimation: A block thresholding and oracle inequality approach," Ann. Statist., vol. 27, no. 3, pp. 898-924, 1999.
14. E. Candès, "Compressive sampling," presented at the Int. Congr. Math., Madrid, Spain, 2006.
15. V.Cevher,C.Hegde, M. F. Duarte, and R. G. Baraniuk, "Sparse signal recovery using Markov random fields," presented at the Neural Inf. Process. Syst., Vancouver, BC, Canada, Dec. 2008.
16. V. Cevher, P. Indyk, L. Carin, and R. G. Baraniuk, "Sparse signal recovery and acquisition with graphical models," IEEE Signal Process. Mag., vol. 27, no. 6, pp. 92-103, Nov. 2010.
17. V. Chandrasekaran, B. Recht, P. A. Parrilo, and A. S. Willsky, The Convex Geometry of Linear Inverse Problems 2011 [Online]. Available: arXiv:1012.0621
18. G. Casella and W. E. Strawderman, "Estimating a bounded normal mean," Ann. Statist., vol. 9, pp. 870-878, 1981.
19. E. J. Candés and T. Tao, "Decoding by linear programming," IEEE Trans. Inf. Theory, vol. 51, no. 12, pp. 4203-4215, Dec. 2005. (Pubitemid 41800353)
20. R. Chartrand and W. Yin, "Iteratively reweighted algorithms for compressive sensing," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2008, pp. 3869-3872.
21. J. C. Hoch, D. L. Donoho, I. M. Johnstone, and A. S. Stern, "Maximum entropy and the nearly black object (with discussion)," J. Roy. Statist. Soc. B, vol. 54, pp. 41-81, 1992.
22. D. L. Donoho and I. M. Johnstone, "Minimax risk over balls," Prob. Th. Rel. Fields, vol. 99, pp. 277-303, 1994.
23. D. L. Donoho and I. M. Johnstone, "Adapting to unknown smoothness via wavelet shrinkage," J. Amer. Statist. Assoc., vol. 90, pp. 1200-1224, 1995. (Pubitemid 126093135)
24. D. Donoho, A. Javanmard, and A. Montanari, Information-Theoretically Optimal Compressed Sensing via Spatial Coupling and Approximate Message Passing 2011 [Online]. Available: arXiv:1112.0708
25. D. L. Donoho, I. Johnstone, and A. Montanari, AMP Public SVN Repository Nov. 2012 [Online]. Available: https://subversion.assembla. com/svn/amp-public/
26. A. Dembo and A. Montanari, "Finite size scaling for the core of large random hypergraphs," Ann. Appl. Prob., vol. 18, pp. 1993-2040, 2008.
27. D. L. Donoho, A. Maleki, and A. Montanari, "Message passing algorithms for compressed sensing," in Proc. Nat. Acad. Sci., 2009, vol. 106, pp. 18914-18919.
28. D. L. Donoho, A. Maleki, and A. Montanari, "Message passing algorithms for compressed sensing: I. Motivation and construction," presented at the IEEE Inf. Theory Workshop, Cairo, Egypt, 2010.
29. D. L. Donoho, A. Maleki, and A. Montanari, "The noise sensitivity phase transition in compressed sensing," IEEE Trans. Inf. Theory, vol. 57, no. 10, pp. 6920-6941, Oct. 2011.
30. D. L. Donoho, "High-dimensional centrally symmetric polytopes with neighborliness proportional to dimension," Discrete Comput. Geom., vol. 35, pp. 617-652, 2006. (Pubitemid 43701356)
31. D. L. Donoho and J. Tanner, "Neighborliness of randomly-projected simplices in high dimensions," in Proc. Nat. Acad. Sci., 2005, vol. 102, no. 27, pp. 9452-9457. (Pubitemid 40981702)
32. D. L. Donoho and J. Tanner, "Counting faces of randomly projected polytopes when the projection radically lowers dimension," J. Amer. Math. Soc., vol. 22, pp. 1-53, 2009.
33. D. L. Donoho and J. Tanner, "Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing," Phil. Trans. R. Soc. A, vol. 367, pp. 4273-4293, 2009.
34. D. L. Donoho and J. Tanner, "Counting the faces of randomly-projected hypercubes and orthants, with applications," Discrete Comput. Geome., vol. 43, no. 3, pp. 522-541, 2010.
35. D. L. Donoho and J. Tanner, "Precise undersampling theorems," Proc. IEEE, vol. 98, no. 6, pp. 913-924, Jun. 2010.
36. M. Fornasier and H. Rauhut, "Iterative thresholding algorithms," Appl. Comp. Harmon. Anal., vol. 25, pp. 187-208, 2008.
37. H.-Y. Gao and A. G. Bruce, "Waveshrink with firm shrinkage," Statist. Sinica, vol. 7, pp. 855-874, 1997.
38. P. Hall, G. Kerkyacharian, and D. Picard, "Block threshold rules for curve estimation using kernel and wavelet methods," Ann. Statist., vol. 26, pp. 922-942, 1998.
39. P. J. Huber and E. Ronchetti, Robust Statistics, second ed. New York, NY, USA: Wiley, 2009.
40. P. J. Huber, "Robust estimation of a location parameter," Ann. Math. Statist., vol. 35, no. 1, pp. 73-101, 1964.
41. A. Javanmard and A. Montanari, State evolution for vector approximate message passing, 2012, to be published.
42. I. M. Johnstone, "Minimax Bayes, asymptotic minimax and sparse wavelet priors," in Statistical Decision Theory and Related Topics, S.S. Gupta and J. O. Berger, Eds. New York, NY, USA: Springer-Verlag, 1994, vol. V., pp. 303-326.
43. I. M. Johnstone, "On minimax estimation of a sparse normal mean vector," Ann. Statist., vol. 22, pp. 271-289, 1994.
44. I. M. Johnstone, Gaussian Estimation: Sequence and Wavelet Models 2012 [Online]. Available: http://www-stat.stanford.edu/imj/GE12-27-11.pdf
45. W. James and C. Stein, "Estimation with quadratic loss," presented at the 4th Berkeley Symp.Math. Statist. Prob, Berkeley, CA, USA, 2010.
46. U. S. Kamilov, V. K. Goyal, and S. Rangan, Message-Passing Estimation from Quantized Samples 2011 [Online]. Available: arXiv:1105. 6368
47. F. Krzakala,M.Mézard, F. Sausset, Y. Sun, and L. Zdeborova, "Statistical physics-based reconstruction in compressed sensing," Phys. Rev. X, vol. 2, pp. 021005-1-021005-18, 2012.
48. Y. Kabashima, T.Wadayama, and T. Tanaka, "A typical reconstruction limit for compressed sensing based on lp-norm minimization," J. Stat. Mech., vol. 2009, p. L09003, 2009.
49. M. A. Khajehnejad, W. Xu, A. S. Avestimehr, and B. Hassibi, "Improved sparse recovery thresholds with two-step reweighted minimization," presented at the IEEE Int. Symp. Inf. Theory, Austin, TX, USA, Jun. 2010.
50. C. L. Mallows, "Minimizing an integral," SIAM Rev., vol. 20, p. 183, 1978.
51. A. Maleki, L. Anitori, A. Yang, and R. Baraniuk, Asymptotic Analysis of Complex LASSO via Complex Approximate Message Passing (CAMP) 2011 [Online]. Available: arXiv:1108.0477
52. A. Maleki and D. L. Donoho, "Optimally tuned iterative reconstruction algorithms for compressed sensing," IEEE J. Sel. Topics Signal Process., vol. 4, no. 2, pp. 330-341, Apr. 2010.
53. M. Mézard and A. Montanari, Information, Physics and Computation. Oxford, U.K.: Oxford Univ. Press, 2009.
54. A. Montanari, "Graphical models concepts in compressed sensing," in Compressed Sensing, Y. C. Eldar and G. Kutyniok, Eds. Cambridge, U.K.: Cambridge Univ. Press, 2012.
55. S. Rangan, "Generalized approximate message passing for estimation with random linear mixing," presented at the IEEE Int. Symp. Inf. Theory, St. Petersburg, Russia, Aug. 2011.
56. S. Rangan, A. K. Fletcher, and V. K. Goyal, "Asymptotic analysis of map estimation via the replica method and applications to compressed sensing," presented at the Neural Inf. Process. Syst., Vancouver, BC, Canada, 2009.
57. L. I. Rudin, S. Osher, and E. Fatemi, "Nonlinear total variation based noise removal algorithms," Phys. D: Nonlinear Phenom., vol. 60, pp. 259-268, 1992.
58. N. Rao, B. Recht, and R. Nowak, Tight Measurement Bounds for Exact Recovery of Structured Sparse Signals 2011 [Online]. Available: arXiv:1106.4355
59. T. J. Richardson and R. Urbanke, Modern Coding Theory. Cambridge, U.K.: Cambridge Univ. Press, 2008.
60. P. Schniter, "Turbo reconstruction of structured sparse signals," presented at the Proc. Conf. Inf. Sci. Syst., Princeton, NJ, USA, 2010.
61. P. Schniter, A Message-Passing Receiver for BICM-OFDM over Unknown Clustered-Sparse Channels 2011 [Online]. Available: arXiv:1101.4724
62. S. Som, L. C. Potter, and P. Schniter, "Compressive imaging using approximate message passing and a Markov-tree prior," presented at the Asilomar Conf. Signals, Syst., Comput., Nov. 2010.
63. L. C. Potter, S. Som, and P. Schniter, "On approximatemessage passing for reconstruction of non-uniformly sparse signals," presented at the Nat. Aereosp. Electron. Conf., Dayton, OH, USA, 2010.
64. M. Stojnic, "Block-length dependent thresholds for 2/1-optimization in block-sparse compressed sensing," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2010, pp. 3918-3921.
65. R. Tibshirani,M. Saunders, S. Rosset, J. Zhu, and K. Knight, "Sparsity and smoothness via the fused Lasso," J. R. Statist. Soc. B, vol. 67, no. 1, pp. 91-108, 2005. (Pubitemid 40167043)
66. C. R. Vogel and M. E. Oman, "Iterative methods for total variation denoising," SIAM J. Sci. Comput., vol. 17, pp. 227-238, 1996. (Pubitemid 126621162)
67. M. J. Wainwright, "Sharp thresholds for high-dimensional and noisy sparsity recovery using-constrained quadratic programming (Lasso)," IEEE Trans. Inf. Theory, vol. 55, no. 5, pp. 2183-2202, May 2009.
68. Y. Wu and S. Verdú, "Rényi information dimension: Fundamental limits of almost lossless analog compression," IEEE Trans. Inf. Theory, vol. 56, no. 8, pp. 3721-3748, Aug. 2010.
69. W. Xu and B. Hassibi, Compressive Sensing over the Grassmann Manifold: A Unified Geometric Framework 2010 [Online]. Available: arXiv:1005.3729
70. M. Yuan and Y. Lin, "Model selection and estimation in regression with grouped variables," J. Royal Statist. Soc. B, vol. 68, pp. 49-67, 2010. (Pubitemid 43415335)
71. J. Berger, "Estimation in continuous exponential families: Bayesian estimation subject to risk restrictions and inadmissibility results," in Statistical Decision Theory and Related Topics III , S. S. Gupta and J. Berger, Eds., 1 ed. New York: Academic, 1982, p. 109.
72. P. J. Bickell, "Parametric robustness: Small biases can be worthwhile," Ann. Statist., vol. 12, no. 3, pp. 864-879, Sep. 1984, published by: Institute of Mathematical Statistics Stable URL: http://www.jstor.org/stable/2240965.