Online sparsifying transform learning -Part i: Algorithms

IEEE Journal on Selected Topics in Signal Processing

Volumn 9, Issue 4, 2015, Pages 625-636

  Ravishankar, Saiprasad ^a   Wen, Bihan ^a   Bresler, Yoram ^a  

^a University of Illinois at Urbana Champaign   (United States)

Author keywords

big data; denoising; dictionary learning; image representation; machine learning; online learning; Sparse representations; sparsifying transforms

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; BIG DATA; IMAGE DENOISING; IMAGE RECONSTRUCTION; LEARNING ALGORITHMS; LEARNING SYSTEMS; MEDICAL IMAGE PROCESSING; MEDICAL IMAGING; SIGNAL DENOISING; SIGNAL PROCESSING; SOCIAL NETWORKING (ONLINE);

DE-NOISING; DICTIONARY LEARNING; IMAGE REPRESENTATIONS; ONLINE LEARNING; SPARSE REPRESENTATION;

E-LEARNING;

EID: 84929299727     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2015.2417131     Document Type: Article

References (57)

1. M. Elad, P. Milanfar, and R. Rubinstein, Analysis versus synthesis in signal priors, Inverse Problems, vol. 23, no. 3, pp. 947-968, 2007.
2. S. Nam, M. E. Davies, M. Elad, and R. Gribonval, Cosparse analysis modeling-Uniqueness and algorithms, in Proc. ICASSP, 2011, pp. 5804-5807.
3. W. K. Pratt, J. Kane, and H. C. Andrews, Hadamard transform image coding, Proc. IEEE, vol. 57, no. 1, pp. 58-68, Jan. 1969.
4. S. Ravishankar and Y. Bresler, Learning sparsifying transforms, IEEE Trans. Signal Process., vol. 61, no. 5, pp. 1072-1086, May 2013.
5. M. Aharon, M. Elad, and A. Bruckstein, K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation, IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311-4322, Nov. 2006.
6. M. Elad and M. Aharon, Image denoising via sparse and redundant representations over learned dictionaries, IEEE Trans. Image Process., vol. 15, no. 12, pp. 3736-3745, Dec. 2006.
7. M. Yaghoobi, S. Nam, R. Gribonval, and M. E. Davies, Constrained overcomplete analysis operator learning for cosparse signal modelling, IEEE Trans. Signal Process., vol. 61, no. 9, pp. 2341-2355, Sep. 2013.
8. R. Rubinstein, T. Peleg, and M. Elad, Analysis K-SVD: A dictionarylearning algorithm for the analysis sparse model, IEEE Trans. Signal Processing, vol. 61, no. 3, pp. 661-677, Mar. 2013.
9. S. Ravishankar and Y. Bresler, Closed-form solutions within sparsifying transform learning, in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2013, pp. 5378-5382.
10. A. M. Bruckstein, D. L. Donoho, and M. Elad, From sparse solutions of systems of equations to sparse modeling of signals and images, SIAM Rev., vol. 51, no. 1, pp. 34-81, 2009.
11. B. K. Natarajan, Sparse approximate solutions to linear systems, SIAM J. Comput., vol. 24, no. 2, pp. 227-234, Apr. 1995.
12. G. Davis, S. Mallat, and M. Avellaneda, Adaptive greedy approximations, J. Construct. Approx., vol. 13, no. 1, pp. 57-98, 1997.
13. Y. Pati, R. Rezaiifar, and P. Krishnaprasad, Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition, in Proc. Asilomar Conf. Signals, Syst., Comput., 1993, vol. 1, pp. 40-44.
14. S. G. Mallat and Z. Zhang, Matching pursuits with time-frequency dictionaries, IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397-3415, Dec. 1993.
15. W. Dai and O. Milenkovic, Subspace pursuit for compressive sensing signal reconstruction, IEEE Trans. Inf. Theory, vol. 55, no. 5, pp. 2230-2249, May 2009.
16. S. S. Chen, D. L. Donoho, and M. A. Saunders, Atomic decomposition by basis pursuit, SIAM J. Sci. Comput., vol. 20, no. 1, pp. 33-61, 1998.
17. I. F. Gorodnitsky, J. George, and B. D. Rao, Neuromagnetic source imaging with FOCUSS: A recursive weighted minimum norm algorithm, Electrocephal. Clinical Neurophysiol., vol. 95, pp. 231-251, 1995.
18. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, Least angle regression, Ann. Statist., vol. 32, pp. 407-499, 2004.
19. G. Harikumar, Blind Image Deconvolution From Multiple Blurs, and Sparse Approximation, Ph.D. dissertation, Univ. of Illinois at Urbana-Champaign, Urbana, IL, USA, Mar. 1997.
20. R. Chartrand, Exact reconstruction of sparse signals via nonconvex minimization, IEEE Signal Process. Lett., vol. 14, no. 10, pp. 707-710, Oct. 2007.
21. D. Donoho, Compressed sensing, IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289-1306, Apr. 2006.
22. J. A. Tropp, Greed is good: Algorithmic results for sparse approximation, IEEE Trans. Inf. Theory, vol. 50, no. 10, pp. 2231-2242, Oct. 2004.
23. E. Cands and T. Tao, Decoding by linear programming, IEEE Trans. Inf. Theory, vol. 51, no. 12, pp. 4203-4215, Dec. 2005.
24. D. L. Donoho, For most large underdetermined systems of linear equations the minimal l1-norm solution is also the sparsest solution, Commun. Pure Appl. Math., vol. 59, pp. 797-829, 2004.
25. R. Rubinstein, T. Faktor, and M. Elad, K-SVD dictionary-learning for the analysis sparse model, in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2012, pp. 5405-5408.
26. R. Rubinstein and M. Elad, K-SVD dictionary-learning for analysis sparse models, in Proc. SPARS11, Jun. 2011, p. 73.
27. A. Chambolle, An algorithm for total variation minimization and applications, J. Math. Imaging Vis., vol. 20, no. 1-2, pp. 89-97, 2004.
28. M. Yaghoobi, S. Nam, R. Gribonval, and M. E. Davies, Noise aware analysis operator learning for approximately cosparse signals, in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 2012, pp. 5409-5412.
29. S. Mallat, A Wavelet Tour of Signal Processing. San Diego, CA, USA: Academic, 1999.
30. E. J. Cands and D. L. Donoho, Curvelets-A surprisingly effective nonadaptive representation for objects with edges, in Curves and Surfaces. Nashville, TN, USA: Vanderbilt Univ. Press, 1999, pp. 105-120.
31. B. A. Olshausen and D. J. Field, Emergence of simple-cell receptive field properties by learning a sparse code for natural images, Nature, vol. 381, no. 6583, pp. 607-609, 1996.
32. K. Engan, S. O. Aase, and J. H. Husoy, Method of optimal directions for frame design, in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), 1999, pp. 2443-2446.
33. J. Mairal, M. Elad, and G. Sapiro, Sparse representation for color image restoration, IEEE Trans. Image Process., vol. 17, no. 1, pp. 53-69, Jan. 2008.
34. M. Yaghoobi, T. Blumensath, and M. Davies, Dictionary learning for sparse approximations with the majorization method, IEEE Trans. Signal Process., vol. 57, no. 6, pp. 2178-2191, Jun. 2009.
35. S. Ravishankar and Y. Bresler, MR image reconstruction from highly undersampled k-space data by dictionary learning, IEEE Trans. Med. Imag., vol. 30, no. 5, pp. 1028-1041, May 2011.
36. K. Skretting and K. Engan, Recursive least squares dictionary learning algorithm, IEEE Trans. Signal Process., vol. 58, no. 4, pp. 2121-2130, Apr. 2010.
37. G. Peyr and J. Fadili, Learning analysis sparsity priors, in Proc. Int. Conf. Sampling Theory Appl. (SampTA), Singapore, May 2-6, 2011 [Online]. Available: http://hal.archives-ouvertes.fr/hal-00542016/
38. M. Yaghoobi, S. Nam, R. Gribonval, and M. Davies, Analysis operator learning for overcomplete cosparse representations, in Proc. Eur. Signal Process. Conf. (EUSIPCO), 2011, pp. 1470-1474.
39. B. Ophir, M. Elad, N. Bertin, and M. Plumbley, Sequential minimal eigenvalues-An approach to analysis dictionary learning, in Proc. Eur. Signal Process. Conf. (EUSIPCO), 2011, pp. 1465-1469.
40. S. Ravishankar and Y. Bresler, Learning doubly sparse transforms for images, IEEE Trans. Image Process., vol. 22, no. 12, pp. 4598-4612, Dec. 2013.
41. S. Ravishankar and Y. Bresler, Sparsifying transform learning for compressed sensing MRI, in Proc. IEEE Int. Symp. Biomed. Imag., 2013, pp. 17-20.
42. B. Wen, S. Ravishankar, and Y. Bresler, Structured overcomplete sparsifying transform learning with convergence guarantees and applications, Int. J. Comput. Vis. pp. 1-31, 2014.
43. L. Pfister, Tomographic Reconstruction With Adaptive Sparsifying Transforms, Master's thesis, Univ. of Illinois at Urbana-Champaign, Urbana, IL, USA, Aug. 2013.
44. L. Pfister and Y. Bresler, Model-based iterative tomographic reconstruction with adaptive sparsifying transforms, in Proc. SPIE Int. Symp. Electron. Imaging: Comput. Imag. XII, 2014, vol. 9020, pp. 90 200H-1-90 200H-11.
45. S. Ravishankar and Y. Bresler, Online sparsifying transform learning-Part II: Convergence analysis, IEEE J. Sel. Topics Signal Process., vol. 9, no. 2, pp. XXX-XXX, Jun. 2015.
46. S. Ravishankar and Y. Bresler, sparsifying transform learning with efficient optimal updates and convergence guarantees, IEEE Trans. Signal Process., 2015, to be published.
47. J. Mairal, F. Bach, J. Ponce, and G. Sapiro, Online learning for matrix factorization and sparse coding, J. Mach. Learn. Res., vol. 11, pp. 19-60, 2010.
48. C. Lu, J. Shi, and J. Jia, Online robust dictionary learning, in Proc. IEEE Conf. Comput. Vis. Pattern Recogn. (CVPR), Jun. 2013, pp. 415-422.
49. J. Xing, J. Gao, B. Li, W. Hu, and S. Yan, Robust object tracking with online multi-lifespan dictionary learning, in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2013, pp. 665-672.
50. G. Zhang, Z. Jiang, and L. S. Davis, Online semi-supervised discriminative dictionary learning for sparse representation, in Proc. Comput. Vis. ACCV12, 2013, pp. 259-273.
51. L. Wang, K. Lu, P. Liu, R. Ranjan, and L. Chen, Ik-svd: Dictionary learning for spatial big data via incremental atom update, Comput. Sci. Eng., vol. 16, no. 4, pp. 41-52, 2014.
52. S. Mukherjee and C. S. Seelamantula, A split-and-merge dictionary learning algorithm for sparse representation, http://arxiv.org/abs/ 1403.4781, 2014.
53. P. Stange, On the efficient update of the singular value decomposition, Proc. Appl. Math. Mech., vol. 8, no. 1, pp. 10 827-10 828, 2008.
54. P. Stange, On the efficient update of the singular value decomposition subject to rank-one modifications, 2011 [Online]. Available: http:// www.digibib.tu-bs.de/?docid=00040371
55. D. L. Donoho and I. M. Johnstone, Ideal spatial adaptation by wavelet shrinkage, Biometrika, vol. 81, no. 3, pp. 425-455, 1994.
56. M. Elad, Michael Elad personal page, [Online]. Available: http:/ /www.cs.technion.ac.il/~elad/Various/KSVD-Matlab-ToolBox.zip, accessed 2014
57. The USC-SIPI image database, [Online]. Available: http://sipi.usc. edu/database/database.php?volume=misc, accessed Jul. 2014