Matching pursuit LASSO part I: Sparse recovery over big dictionary

IEEE Transactions on Signal Processing

Volumn 63, Issue 3, 2015, Pages 727-741

  Tan, Mingkui ^a   Tsang, Ivor W ^b   Wang, Li ^c  

^a UNIVERSITY OF ADELAIDE   (Australia)

^b UNIVERSITY OF TECHNOLOGY SYDNEY   (Australia)

^c BROWN UNIVERSITY   (United States)

Author keywords

big dictionary; compressive sensing; convex programming; LASSO; matching pursuit; Sparse recovery

Indexed keywords

COMPRESSED SENSING; CONVEX OPTIMIZATION; LINEAR PROGRAMMING; SIGNAL RECONSTRUCTION;

COMPRESSIVE SENSING; COMPUTATION COSTS; GLOBAL SOLUTIONS; LASSO; LINEAR PROGRAMS; MATCHING PURSUIT; ROBUST SIGNAL RECOVERIES; SPARSE RECOVERY;

RECOVERY;

EID: 84920973666     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSP.2014.2385036     Document Type: Article

References (82)

1. D. Donoho, "Compressed sensing," IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289-1306, Apr. 2006.
2. J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, "Robust face recognition via sparse representation," IEEE Trans. Pattern Anal. Mach., vol. 31, no. 2, pp. 210-227, Apr. 2009.
3. T. Peleg, Y. Eldar, and M. Elad, "Exploiting statistical dependencies in sparse representations for signal recovery," IEEE Trans. Signal Process., vol. 60, no. 5, pp. 2286-2303, May 2012.
4. , Y. C. Eldar and G. Kutyniok, Eds., Compressed Sensing: Theory and Applications. Cambridge, U.K.: Cambridge Univ. Press, 2012.
5. E. J. Candès and T. Tao, "Decoding by linear programming," IEEE Trans. Inf. Theory, vol. 51, no. 12, pp. 4203-4215, Dec. 2005.
6. M. F. Duarte and Y. C. Eldar, "Structured compressed sensing: From theory to applications," IEEE Trans. Signal Process., vol. 59, no. 9, pp. 4053-4085, Sep. 2011.
7. T. T. Do, L. Gan, N. H. Nguyen, and T. D. Tran, "Fast and efficient compressive sensing using structurally random matrices," IEEE Trans. Signal Process., vol. 60, no. 1, pp. 139-154, Jan. 2012.
8. J. Mairal, M. Elad, and G. Sapiro, "Sparse representation for color image restoration," IEEE Trans. Image Process., vol. 17, no. 1, pp. 53-69, Jan. 2008.
9. J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, "Non-local sparse models for image restoration," presented at the ICCV, 2009.
10. J. Yang, J. Wright, T. Huang, and Y. Ma, "Image super-resolution via sparse representation," IEEE Trans. Image Process., vol. 19, no. 11, pp. 2861-2873, Nov. 2010.
11. D. Xu, Y. Huang, Z. Zeng, and X. Xu, "Human gait recognition using patch distribution feature and locality-constrained group sparse representation," IEEE Trans. Image Process., vol. 21, no. 1, pp. 316-326, Jan. 2012.
12. E. Elhamifar and R. Vidal, "Sparse subspace clustering," presented at the CVPR, 2009.
13. A. Adler, M. Elad, and Y. Hel-Or, "Probabilistic subspace clustering via sparse representations," IEEE Signal Process. Lett., vol. 20, no. 1, pp. 63-66, Jan. 2013.
14. J.Mairal, F. Bach, J. Ponce, and G. Sapiro, "Online dictionary learning for sparse coding," presented at the ICML, 2009.
15. L. Bo, X. Ren, and D. Fox, "Hierarchical matching pursuit for image classification: Architecture and fast algorithms," presented at the NIPS, 2011.
16. A. Coates and A. Ng, "The importance of encoding versus training with sparse coding and vector quantization," presented at the ICML, 2011.
17. K. Yu, Y. Lin, and J. Lafferty, "Learning image representations from the pixel level via hierarchical sparse coding," presented at the CVPR, 2011.
18. G. Davis, S. Mallat, and M. Avellaneda, "Adaptive greedy approximations," Constr. Approx., vol. 13, no. 1, pp. 57-98, 1997.
19. D. Ge, X. Jiang, and Y. Ye, "A note on the complexity of lp minimization," Math. Program., vol. 129, no. 2, pp. 285-299, 2011.
20. D. L. Donoho andM. Elad, "Optimally sparse representation in general (nonorthogonal) dictionaries via minimization," in Proc. Nat. cad. Sci., 2003, vol. 100, no. 5, pp. 2197-2202.
21. B. Efron, T. Hastie, L. Johnstone, and R. Tibshirani, "Least angle regression," Ann. Statist., vol. 32, no. 2, pp. 407-499, 2004.
22. H. Lee, A. Battle, R. Raina, and A. Y. Ng, "Efficient sparse coding algorithms," presented at the NIPS, 2006.
23. T. T. Cai, L. Wang, and G. Xu, "New bounds for restricted isometry constants," IEEE Trans. Inf. Theory, vol. 56, no. 9, pp. 4388-4394, Sep. 2010.
24. E. J. Candès and T. Tao, "The dantzig selector: Statistical estimation when is much larger than ," Ann. Statist., vol. 35, no. 6, pp. 2313-2351, 2007.
25. E. J. Candès, J. Romberg, and T. Tao, "Stable signal recovery from incomplete and inaccuratemeasurements," Commun. Pure Appl.Math, vol. 59, no. 8, pp. 1207-1223, 2006.
26. M. R. Osborne, B. Presnell, and B. A. Turlach, "On the lasso and its dual," J. Comput. Graph. Statist., vol. 9, no. 2, pp. 319-337, 2000.
27. M. A. T. Figueiredo, R. D. Nowak, and S. J. Wright, "Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems," IEEE J. Sel. Top. Signal Proces.: Special Issue Convex Optimiz. Methods Signal Process., 2007.
28. P. J. Bickel, Y. Ritov, and A. B. Tsybakov, "Simultaneous analysis of lasso and dantzig selector," Ann. Statist., vol. 37, no. 4, pp. 1705-1732, 2009.
29. A. Beck and M. Teboulle, "A fast iterative shrinkage-thresholding algorithm for linear inverse problems," SIAM J. Imag. Sci., vol. 2, no. 1, pp. 183-202, 2009.
30. Z. Wen, W. Yin, D. Goldfarb, and Y. Zhang, "A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization, and continuation," SIAM J. Sci. Comput., vol. 32, no. 4, pp. 1832-1857, 2010.
31. A. Yang, A. Ganesh, Y. Ma, and S. Sastry, "Fast l1-minimization algorithms and an application in robust face recognition: A review," presented at the ICIP, 2010.
32. L. Xiao and T. Zhang, "A proximal-gradient homotopy method for the l1-regularized least-squares problem," presented at the ICML, 2012.
33. S.-J. Kim, K. Koh, M. Lustig, S. Boyd, and D. Gorinevsky, "An interior-point method for large-scale-regularized least squares," IEEE J. Sel. Topics Signal Process., vol. 1, no. 4, pp. 606-617, Aug. 2007.
34. Y. Nesterov, "Gradient Methods for Minimizing Composite Objective Function," Center for Operations Research and Econometrics (CORE), Catholic Univ. of Louvain (UCL), 2007 [Online]. Available: http://www.ecore.be/DPs/dp-1191313936.pdf
35. J. Friedman, T. Hastie, and R. Tibshirani, "Regularization paths for generalized linear models via coordinate descent," J. Statist. Software, vol. 33, no. 1, p. 1, 2010.
36. S. Yun and K.-C. Toh, "A coordinate gradient descent method for-regularized convex minimization," Comput. Optimiz. Appl., vol. 48, no. 2, pp. 273-307, 2011.
37. J. K. Bradley, A. Kyrola, D. Bickson, and C. Guestrin, "Parallel coordinate descent for-regularized loss minimization," presented at the ICML, 2011.
38. Z. J.Xiang, H. Xu, and P. J. Ramadge, "Learning sparse representations of high dimensional data on large scale dictionaries," in NIPS, 2012.
39. L. Xiao and T. Zhang, "A proximal-gradient homotopy method for the sparse least-squares problem," SIAM J. Optimiz., vol. 23, no. 2, pp. 1062-1091, 2013.
40. V. Roth and B. Fischer, "The group-lasso for generalized linear models: Uniqueness of solutions and efficient algorithms," presented at the ICML, 2008.
41. J. Kim and H. Park, "Fast active-set-type algorithms for l1-regularized linear regression," presented at the AISTAT, 2010.
42. S. Shalev-Shwartz, N. Srebro, and T. Zhang, "Trading accuracy for sparsity in optimization problems with sparsity constraint," SIAM J. Optimiz., vol. 20, pp. 2807-2832, 2010.
43. M. Kowalski, P.Weiss, A. Gramfort, and S. Anthoine et al., ""Accelerating ISTAWith An Active Set Strategy,"," presented at the OPT 2011: 4th Int. Workshop Optimiz Mach. Learn., 2011.
44. R. Rigamonti, M. A. Brown, and V. Lepetit, "Are sparse representations really relevant for image classification?," presented at the CVPR, 2011.
45. Q. Shi, A. Eriksson, A. V. D. Hengel, and C. Shen, "Is face recognition really a compressive sensing problem?," presented at the CVPR, 2011.
46. E. J. Candès and Y. Plan, "Near-ideal model selection by minimization," Ann. Statist., vol. 37, no. 5A, pp. 2145-2177, 2007.
47. S. G. Mallat and Z. Zhang, "Matching pursuits with time-frequency dictionaries," IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397-3415, Dec. 1993.
48. J. A. Tropp, "Greed is good: Algorithmic results for sparse approximation," IEEE Trans. Inf. Theory, vol. 50, no. 10, pp. 2231-2242, Oct. 2004.
49. Y. C. Pati, R. Rezaiifar, and P. Krishnaprasad, "Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition," in Proc. 27th Asilomar Conf. Signals, Syst., Comput., 1993, pp. 40-44.
50. J. Tropp and A. Gilbert, "Signal recovery from random measurements via orthogonal matching pursuit," IEEE Trans. Inf. Theory, vol. 53, no. 12, pp. 4655-4666, Dec. 2007.
51. T. Blumensath and M. E. Davies, "Gradient pursuits," IEEE Trans. Signal Process., vol. 56, no. 6, pp. 2370-2382, Jun. 2008.
52. T. Zhang, "Sparse recovery with orthogonal matching pursuit under RIP," IEEE Trans. Inf. Theory, vol. 57, no. 9, pp. 6215-6221, Sep. 2011.
53. A. Tewari, P. Ravikumar, and I. S. Dhillon, "Greedy algorithms for structurally constrained high dimensional problems," presented at the NIPS, 2011.
54. X.Yuan and S. Yan, "Forward basis selection for sparse approximation over dictionary," presented at the AISTATS, 2012.
55. D. Needell and J. Tropp, "CoSaMP: Iterative signal recovery from incomplete and inaccurate samples," Appl. Comput. Harmon. Anal., vol. 26, no. 3, pp. 301-321, 2009.
56. W. Dai and O. Milenkovic, "Subspace pursuit for compressive sensing signal reconstruction," IEEE Trans. Inf. Theory, vol. 55, no. 5, pp. 2230-2249, May 2009.
57. T. Blumensath andM. E. Davies, "Iterative hard thresholding for compressed sensing," Appl. Comput. Harmon. Anal., vol. 27, no. 3, pp. 265-274, 2009.
58. T. Blumensath, "Accelerated iterative hard threshoding," Signal Process., vol. 92, no. 3, pp. 752-756, 2011.
59. R. Giryes and M. Elad, "RIP-based near-oracle performance guarantees for subspace-pursuit, CoSaMP, and iterative hard-thresholding," IEEE Trans. Signal Process., vol. 60, no. 3, pp. 1465-1468, Mar. 2012.
60. P. Jain, A. Tewari, and I. S. Dhillon, "Orthogonalmatching pursuit with replacement," presented at the NIPS, 2011.
61. M. Davenport, D. Needell, and M. Wakin, "Signal space CoSaMP for sparse recovery with redundant dictionaries," IEEE Trans. Inf. Theory, vol. 59, no. 10, pp. 6820-6829, Oct. 2013.
62. R. Giryes and M. Elad, "Can we allow linear dependencies in the dictionary in the sparse synthesis framework?," presented at the ICASSP, 2013.
63. R. Giryes and D. Needell, "Greedy Signal Space Methods for Incoherence and Beyond," arXiv:1309.2676, 2013.
64. R. Giryes and M. Elad, "OMP with highly coherent dictionaries," presented at the Int. Conf. Sampling Theory Appl. (SAMPTA), 2013.
65. D. Needell and R. Vershynin, "Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit," J. Found. Comput. Math., vol. 9, no. 3, pp. 317-334, 2009.
66. R. Maleh, "Efficient Sparse Approximation Methods for Medical Imaging," Ph.D. dissertation, Univ. Michigan, Ann Arbor, MI, USA, 2009.
67. E. Liu and V. N. Temlyakov, "The orthogonal super greedy algorithm and applications in compressed sensing," IEEE Trans. Inf. Theory, vol. 58, no. 4, pp. 2040-2047, Apr. 2012.
68. T. Blumensath and M. E. Davies, "Stagewise weak gradient pursuits," IEEE Trans. Signal Process., vol. 57, no. 11, pp. 4333-4346, Nov. 2009.
69. D. L. Donoho, Y. Tsaig, I. Drori, and J. L. Starck, "Sparse solution of underdetermined systems of linear equations by stagewise orthogonal matching pursuit," IEEE Trans. Inf. Theory, vol. 58, no. 2, pp. 1094-1121, Feb. 2012.
70. C. Hegde, P. Indyk, and L. Schmidt, "Approximation-tolerant modelbased compressive sensing," presented at the ACM Symp. Discrete Algorithms (SODA), 2014.
71. R. Giryes, S. Nam,M. Elad, R. Gribonval, andM. E. Davies, "Greedylike algorithms for the cosparse analysismodel," Linear Algebra Appl., vol. 441, pp. 22-60, 2014.
72. G. Raskutti,M. J.Wainwright, and B. Yu, "Restricted eigenvalue properties for correlated gaussian designs," JMLR, vol. 99, pp. 2241-2259, 2010.
73. L. Wang, "L1 Penalized LAD Estimator for High Dimensional Linear Regression," MIT, 2012 [Online]. Available: http://math.mit.edu/~liewang/L1PLAD-Wang.pdf
74. H. Xu, C. Caramanis, and S. Mannor, "Robust regression and lasso," IEEE Trans. Inf. Theory, vol. 56, no. 7, pp. 3561-3574, Jul. 2010.
75. S. P. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge Univ. Press, 2004.
76. E. Y. Pee and J. O. Royset, "On solving large-scale finite minimax problems using exponential smoothing," J. Optimiz. Theory App., vol. 148, no. 2, pp. 390-421, 2011.
77. K. O. Kortanek and H. No, "A central cutting plane algorithm for convex semi-infinite programming problems," SIAM J. Optimiz., vol. 3, no. 4, pp. 901-918, 1993.
78. J. Elzinga and T. G. Moore, "A central cutting plane algorithm for the convex programming problem," Math. Program., vol. 8, no. 1, pp. 134-145, 1975.
79. B. Beckermann and A. B. J. Kuijlaars, "Superlinear convergence of conjugate gradients," SIAM J. Numer. Anal., vol. 39, no. 1, pp. 300-329, 2002.
80. M. Tan, I. Tsang, and L. Wang, "Matching Pursuit LASSO Part II: Applications and Sparse Recovery Over Batch Signals," IEEE Trans. Signal Processing, vol. 63, no. 3, pp. 742-753, Feb. 1, 2015.
81. R. Gribonval and P. Vandergheynst, "On the exponential convergence of matching pursuits in quasi-incoherent dictionaries," IEEE Trans. Inf. Theory, vol. 52, no. 1, pp. 255-261, Jan. 2006.
82. M. Sion, "On general minimax theorems," Pacific J. Math., vol. 8, no. 1, pp. 171-176, 1958.