Computational methods for sparse solution of linear inverse problems

Proceedings of the IEEE

Volumn 98, Issue 6, 2010, Pages 948-958

  Tropp, Joel A ^a   Wright, Stephen J ^b  

^a CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^b University of Wisconsin Madison   (United States)

Author keywords

Compressed sensing; Convex optimization; Matching pursuit; Sparse approximation

Indexed keywords

APPROXIMATION ALGORITHMS; COMPRESSED SENSING; CONVEX OPTIMIZATION;

APPLIED MATHEMATICS; COMPUTATIONAL ISSUES; ELEMENTARY SIGNALS; LINEAR COMBINATIONS; LINEAR INVERSE PROBLEMS; MATCHING PURSUIT; SPARSE APPROXIMATIONS; THEORETICAL GUARANTEES;

INVERSE PROBLEMS;

EID: 77952743135     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2010.2044010     Document Type: Article

References (81)

1. S. S. Chen, D. L. Donoho, and M. A. Saunders, "Atomic decomposition by basis pursuit," SIAM Rev., vol.43, no.1, pp. 129-159, 2001.
2. E. Candès, J. Romberg, and T. Tao, "Robust uncertainty principles: Exact signal reconstruction from highly incomplete Fourier information," IEEE Trans. Inf. Theory, vol.52, no.2, pp. 489-509, Feb. 2006. (Pubitemid 43174093)
3. D. L. Donoho, "Compressed sensing," IEEE Trans. Inf. Theory, vol.52, no.4, pp. 1289-1306, Apr. 2006.
4. S. Muthukrishnan, Data Streams: Algorithms and Applications. Boston, MA: Now Publishers, 2005.
5. D. L. Donoho, M. Vetterli, R. A. DeVore, and I. Daubechies, "Data compression and harmonic analysis," IEEE Trans. Inf. Theory, vol.44, no.6, pp. 2433-2452, Oct. 1998. (Pubitemid 128741896)
6. R. G. Baraniuk, V. Cevher, M. Duarte, and C. Hegde, "Model-based compressive sensing," 2008, submitted for publication.
7. S. Mallat and Z. Zhang, "Matching pursuits with time-frequency dictionaries," IEEE Trans. Signal Process., vol.41, no.12, pp. 3397-3415, Dec. 1993.
8. D. Wipf and B. Rao, "Sparse Bayesian learning for basis selection," IEEE Trans. Signal Process., vol.52, no.8, pp. 2153-2164, Aug. 2004.
9. P. Schniter, L. C. Potter, and J. Ziniel, "Fast Bayesian matching pursuit: Model uncertainty and parameter estimation for sparse linear models," IEEE Trans. Signal Process., 2008, submitted for publication.
10. R. Chartrand, "Exact reconstruction of sparse signals via nonconvex minimization," IEEE Signal Process. Lett., vol.14, no.10, pp. 707-710, Oct. 2007. (Pubitemid 47434090)
11. A. J. Miller, Subset Selection in Regression, 2nd ed. London, U.K.: Chapman and Hall, 2002.
12. D. Baron, S. Sarvotham, and R. G. Baraniuk, "Bayesian compressive sensing via belief propagation," IEEE Trans. Signal Process., vol.58, no.1, pp. 269-280, Jan. 2010.
13. D. L. Donoho, A. Maliki, and A. Montanari, "Message-passing algorithms for compressed sensing," Proc. Nat. Acad. Sci., vol.106, no.45, pp. 18 914-18 919, 2009.
14. E. J. Candes and T. Tao, "Near-optimal signal recovery from random projections: Universal encoding strategies?" IEEE Trans. Inf. Theory, vol. 52, no. 12, pp. 5406-5425, Dec. 2006. (Pubitemid 46445381)
15. q-minimization for 0 > q > 1," Appl. Comput. Harmonic Anal., vol.26, no.3, pp. 395-407, 2009.
16. 1 minimization," Proc. Nat. Acad. Sci., vol.100, pp. 2197-2202, Mar. 2003. (Pubitemid 36297476)
17. D. L. Donoho and X. Huo, "Uncertainty principles and ideal atomic decomposition," IEEE Trans. Inf. Theory, vol.47, no.7, pp. 2845-2862, Nov. 2001. (Pubitemid 33053488)
18. R. Berinde, A. C. Gilbert, P. Indyk, H. Karloff, and M. Strauss, "Combining geometry and combinatorics: A unified approach to sparse signal recovery," in Proc. 46th Annu. Allerton Conf. Commun. Control Comput., 2008, pp. 798-805.
19. E. van den Berg, M. P. Friedlander, G. Hennenfent, F. Herrmann, R. Saab, and O. Yilmaz, "Sparco: A testing framework for sparse reconstruction, " ACM Trans. Math. Softw., vol.35, no.4, pp. 1-16, 2009.
20. Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, "Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition," in Proc. 27th Annu. Asilomar Conf. Signals Syst. Comput., Nov. 1993, vol.1, pp. 40-44.
21. G. Davis, S. Mallat, and M. Avellaneda, "Adaptive greedy approximation," J. Constr. Approx., vol.13, pp. 57-98, 1997.
22. A. C. Gilbert, M. Muthukrishnan, and M. J. Strauss, "Approximation of functions over redundant dictionaries using coherence," in Proc. 14th Annu. ACM-SIAM Symp. Discrete Algorithms, Jan. 2003.
23. J. H. Friedman and W. Stuetzle, "Projection pursuit regressions," J. Amer Stat. Assoc., vol.76, no.376, pp. 817-823, Dec. 1981.
24. R. DeVore and V. N. Temlyakov, "Some remarks on greedy algorithms," Adv. Comput. Math., vol.5, pp. 173-187, 1996. (Pubitemid 126729159)
25. 1-penalization for flexible function libraries," Ann. Stat., 2008, submitted for publication.
26. A. R. Barron, A. Cohen, R. A. DeVore, and W. Dahmen, "Approximation and learning by greedy algorithms," Ann. Stat., vol.36, no.1, pp. 64-94, 2008.
27. J. A. Tropp, "Greed is good: Algorithmic results for sparse approximation," IEEE Trans. Inf. Theory, vol.50, no.10, pp. 2231-2242, Oct. 2004.
28. J. A. Tropp, A. C. Gilbert, and M. J. Strauss, "Algorithms for simultaneous sparse approximation. Part I: Greedy pursuit," J. Signal Process., vol.86, Special Issue on Sparse Approximations in Signal and Image Processing, pp. 572-588, Apr. 2006. (Pubitemid 43106572)
29. D. L. Donoho, M. Elad, and V. N. Temlyakov, "Stable recovery of sparse overcomplete representations in the presence of noise," IEEE Trans. Inf. Theory, vol.52, no.1, pp. 6-18, Jan. 2006. (Pubitemid 43263116)
30. T. Zhang, "On the consistency of feature selection using greedy least squares regression," J. Mach. Learning Res., vol.10, pp. 555-568, 2009.
31. J. A. Tropp and A. C. Gilbert, "Signal recovery from random measurements via orthogonal matching pursuit," IEEE Trans. Inf. Theory, vol.53, no.12, pp. 4655-4666, Dec. 2007.
32. A. Fletcher and S. Rangan, "Orthogonal matching pursuit from noisy random measurements: A new analysis," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2009, p. 23.
33. D. L. Donoho, Y. Tsaig, I. Drori, and J.-L. Starck, "Sparse solution of underdetermined linear equations by stagewise orthogonal matching pursuit (StOMP)," Stanford Univ., Palo Alto, CA, Stat. Dept. Tech. Rep. 2006-2102, Mar. 2006.
34. D. Needell and R. Vershynin, "Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit," Found. Comput. Math., vol.9, no.3, pp. 317-334, 2009.
35. D. Needell and R. Vershynin, "Signal recovery from incomplete and inaccurate measurements via regularized orthogonal matching pursuit," IEEE J. Sel. Topics Signal Process., vol.4, no.2, pp. 310-316, Apr. 2009.
36. D. Needell and J. A. Tropp, "CoSaMP: Iterative signal recovery from incomplete and inaccurate samples," Appl. Comput. Harmonic Anal., vol.26, no.3, pp. 301-321, 2009.
37. W. Dai and O. Milenkovic, "Subspace pursuit for compressive sensing: Closing the gap between performance and complexity," IEEE Trans. Inf. Theory, vol.55, no.5, pp. 2230-2249, May 2009.
38. D. Needell and J. A. Tropp, "CoSaMP: Iterative signal recovery from incomplete and inaccurate samples," California Inst. Technol., Pasadena, CA, ACM Rep. 2008-01, 2008.
39. A. Maliki and D. Donoho, "Optimally tuned iterative reconstruction algorithms for compressed sensing," Sep. 2009. [Online]. Available: arXiv:0909.0777
40. A. Cohen, W. Dahmen, and R. DeVore, "Compressed sensing and bestk-term approximation," J. Amer. Math. Soc., vol.22, no.1, pp. 211-231, 2009.
41. T. Blumensath and M. Davies, "Iterative hard thresholding for compressed sensing," Appl. Comput. Harmonic Anal., vol.27, no.3, pp. 265-274, 2009.
42. A. Cohen, R. A. DeVore, and W. Dahmen, "Instance-optimal decoding by thresholding in compressed sensing," 2008.
43. R. Garg and R. Khandekar, "Gradient descent with sparsification: An iterative algorithms for sparse recovery with restricted isometry property," in Proc. 26th Annu. Int. Conf. Mach. Learn., Montreal, QC, Canada, Jun. 2009.
44. R. Meka, P. Jain, and I. S. Dhillon, "Guaranteed rank minimization via singular value projection," [Online]. Available: arXiv:0909.5457
45. J.-L. Starck, M. Elad, and D. L. Donoho, "Redundant multiscale transforms and their application for morphological component analysis," J. Adv. Imaging Electron Phys., vol.132, pp. 287-348, 2004.
46. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, "Least angle regression," Ann. Stat., vol.32, no.2, pp. 407-499, 2004. (Pubitemid 41250302)
47. M. R. Osborne, B. Presnell, and B. Turlach, "A new approach to variable selection in least squares problems," IMA J. Numer. Anal., vol. 20, pp. 389-403, 2000.
48. R. Gribonval and M. Nielsen, "Highly sparse representations from dictionaries are unique and independent of the sparseness measure," Aalborg Univ., Aalborg, Denmark, Tech. Rep., Oct. 2003.
49. R. Tibshirani, "Regression shrinkage and selection via the LASSO," J. R. Stat. Soc. B, vol.58, pp. 267-288, 1996.
50. J.-J. Fuchs, "On sparse representations in arbitrary redundant bases," IEEE Trans. Inf. Theory, vol.50, no.6, pp. 1341-1344, Jun. 2004.
51. J. A. Tropp, "Just relax: Convex programming methods for identifying sparse signals in noise," IEEE Trans. Inf. Theory, vol.52, no.3, pp. 1030-1051, Mar. 2006.
52. J. A. Tropp, "On the conditioning of random subdictionaries," Appl. Comput. Harmonic Anal., vol.25, pp. 1-24, 2008.
53. 1 minimization," Ann. Stat., vol. 37, no. 5A, pp. 2145-2177, 2009.
54. J. A. Tropp, "Norms of random submatrices and sparse approximation," C. R. Acad. Sci. Paris Ser. I Math., vol.346, pp. 1271-1274, 2008.
55. E. J. Cand̀es, J. Romberg, and T. Tao, "Stable signal recovery from incomplete and inaccurate measurements," Commun. Pure Appl. Math., vol.59, pp. 1207-1223, 2006. (Pubitemid 43988295)
56. R. Gribonval and M. Nielsen, "Sparse representations in unions of bases," IEEE Trans. Inf. Theory, vol.49, no.12, pp. 3320-3325, Dec. 2003.
57. M. Rudelson and R. Vershynin, "On sparse reconstruction from Fourier and Gaussian measurements," Commun. Pure Appl. Math., vol.61, no.8, pp. 1025-1045, 2008.
58. 1 minimization: Simple derivations and extensions," Rice Univ., Houston, TX, CAAM Tech. Rep. TR08-11, 2008.
59. D. Donoho and J. Tanner, "Counting faces of randomly projected polytopes when the projection radically lower dimensions," J. Amer. Math. Soc., vol.1, pp. 1-53, 2009.
60. B. Hassibi and W. Xu, "On sharp performance bounds for robust sparse signal recovery," in Proc. IEEE Symp. Inf. Theory, Seoul, Korea, 2009, pp. 493-1397
61. 1-norm minimization problems when the solution may be sparse," IEEE Trans. Inf. Theory, vol.54, no.11, pp. 4789-4812, Nov. 2008.
62. M. A. Saunders, "PDCO: Primal-dual interior-point method for convex objectives," Syst. Optim. Lab., Stanford Univ., Stanford, CA, Nov. 2002.
63. 1-regularized least squares," IEEE J. Sel. Topics Signal Process., vol.1, no.4, pp. 606-617, Dec. 2007. (Pubitemid 351266626)
64. 1-MAGIC: Recovery of sparse signals via convex programming," California Inst. Technol., Pasadena, CA, Tech. Rep., Oct. 2005.
65. P. L. Combettes and V. R. Wajs, "Signal recovery by proximal forward-backward splitting," Multiscale Model. Simul., vol.4, no.4, pp. 1168-1200, 2005.
66. I. Daubechies, M. Defriese, and C. De Mol, "An iterative thresholding algorithm for linear inverse problems with a sparsity constraint," Commun. Pure Appl. Math., vol.LVII, pp. 1413-1457, 2004. (Pubitemid 39427442)
67. 1-minimization: Methodology and convergence," SIAM J. Optim., vol.19, pp. 1107-1130, 2008.
68. S. J. Wright, R. D. Nowak, and M. A. T. Figueiredo, "Sparse reconstruction by separable approximation," IEEE Trans. Signal Process., vol.57, no.8, pp. 2479-2493, Aug. 2009.
69. K. Bredies and D. A. Lorenz, "Linear convergence of iterative soft-thresholding," SIAM J. Sci. Comput., vol.30, no.2, pp. 657-683, 2008.
70. J. M. Bioucas-Dias and M. A. T. Figueiredo, "A new TwIST: Two-step iterative shrinking/thresholding algorithms for image restoration," IEEE Trans. Image Process., vol.16, no.12, pp. 2992-3004, Dec. 2007.
71. M. A. T. Figueiredo, R. D. Nowak, and S. J. Wright, "Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems," IEEE J. Sel. Topics Signal Process., vol.1, no.4, pp. 586-597, Dec. 2007.
72. Z. Wen, W. Yin, D. Goldfarb, and Y. Zhang, "A fast algorithms for sparse reconstruction based on shrinkage, subspace optimization, and continuation," Rice Univ., Houston, TX, CAAM Tech. Rep. 09-101, Jan. 2009.
73. W. Shi, G. Wahba, S. J. Wright, K. Lee, R. Klein, and B. Klein, "LASSO-Patternsearch algorithm with application to opthalmology data," Stat. Interface, vol.1, pp. 137-153, Jan. 2008.
74. E. van den Berg and M. P. Friedlander, "Probing the Pareto frontier for basis pursuit solutions," SIAM J. Sci. Comput., vol.31, no.2, pp. 890-912, 2008.
75. 2," Doklady AN SSSR, vol.269, pp. 543-547, 1983.
76. Y. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course. Norwell, MA: Kluwer, 2004.
77. A. Nemirovski and D. B. Yudin, Problem Complexity and Method Efficiency in Optimization. New York: Wiley, 1983.
78. Y. Nesterov, "Gradient methods for minimizing composite objective function," Catholic Univ. Louvain, Louvain, "elgium, CORE Discussion Paper 2007/2076, Sep. 2007.
79. A. Beck and M. Teboulle, "A fast iterative shrinkage-threshold algorithm for linear inverse problems," SIAM J. Imaging Sci., vol.2, pp. 183-202, 2009.
80. S. Becker, J. Bobin, and E. J. Candès, "NESTA: A fast and accurate first-order method for sparse recovery," Apr. 2009. [Online]. Available: arXiv:0904.3367
81. M. P. Friedlander and M. A. Saunders, "Active-set approaches to basis pursuit denoising," in Talk SIAM Optim. Meeting, Boston, MA, May 2008.