Efficient algorithms for convolutional sparse representations

IEEE Transactions on Image Processing

Volumn 25, Issue 1, 2016, Pages 301-315

  Wohlberg, Brendt ^a  

^a LOS ALAMOS NATIONAL LABORATORY   (United States)

Author keywords

ADMM; convolutional sparse representation; dictionary learning; sparse coding; Sparse representation

Indexed keywords

COMPUTER VISION; CONVOLUTION; IMAGE PROCESSING;

ADMM; COMPUTATIONAL EXPENSE; DICTIONARY LEARNING; LINEAR COMBINATIONS; OPTIMIZATION PROBLEMS; SIGNAL AND IMAGE PROCESSING; SPARSE CODING; SPARSE REPRESENTATION;

OPTIMIZATION;

EID: 85009373801     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2015.2495260     Document Type: Article

References (67)

1. A. M. Bruckstein, D. L. Donoho, and M. Elad, "From sparse solutions of systems of equations to sparse modeling of signals and images," SIAM Rev., vol. 51, no. 1, pp. 34-81, 2009.
2. J. Mairal, F. Bach, and J. Ponce, "Sparse modeling for image and vision processing," Found. Trends Comput. Graph. Vis., vol. 8, nos. 2-3, pp. 85-283, 2014.
3. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, "Distributed optimization and statistical learning via the alternating direction method of multipliers," Found. Trends Mach. Learn., vol. 3, no. 1, pp. 1-122, 2010.
4. S. G. Mallat and Z. Zhang, "Matching pursuits with timefrequency dictionaries," IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397-3415, Dec. 1993.
5. Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, "Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition," in Proc. Conf. Rec. 27th Asilomar Conf. Signals, Syst. Comput., Nov. 1993, pp. 40-44.
6. R. Rubinstein, A. M. Bruckstein, and M. Elad, "Dictionaries for sparse representation modeling," Proc. IEEE, vol. 98, no. 6, pp. 1045-1057, Jun. 2010.
7. S. S. Chen, D. L. Donoho, and M. A. Saunders, "Atomic decomposition by basis pursuit," SIAM J. Sci. Comput., vol. 20, no. 1, pp. 33-61, 1998.
8. B. Wohlberg, "Efficient convolutional sparse coding," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2014, pp. 7173-7177.
9. P.-K. Jao, Y.-H. Yang, and B. Wohlberg, "Informed monaural source separation of music based on convolutional sparse coding," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Apr. 2015, pp. 236-240.
10. A. Cogliati, Z. Duan, and B. Wohlberg, "Piano music transcription with fast convolutional sparse coding," in Proc. IEEE Int. Workshop Mach. Learn. Signal Process. (MLSP), Sep. 2015.
11. K. Kavukcuoglu, P. Sermanet, Y.-L. Boureau, K. Gregor, M. Mathieu, and Y. LeCun, "Learning convolutional feature hierarchies for visual recognition," in Proc. Adv. Neural Inf. Process. Syst., vol. 23. 2010, pp. 1090-1098.
12. M. D. Zeiler, G. W. Taylor, and R. Fergus, "Adaptive deconvolutional networks for mid and high level feature learning," in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Nov. 2011, pp. 2018-2025.
13. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
14. M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus, "Deconvolutional networks," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2010, pp. 2528-2535.
15. J. Yang, K. Yu, and T. S. Huang, "Supervised translation-invariant sparse coding," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2010, pp. 3517-3524.
16. P. Sermanet, K. Kavukcuoglu, S. Chintala, and Y. LeCun, "Pedestrian detection with unsupervised multi-stage feature learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2013, pp. 3626-3633.
17. B. Chen, G. Polatkan, G. Sapiro, D. Blei, D. Dunson, and L. Carin, "Deep learning with hierarchical convolutional factor analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1887-1901, Aug. 2013.
18. A. D. Szlam, K. Kavukcuoglu, and Y. LeCun. (2010). "Convolutional matching pursuit and dictionary training." Online. Available: http://arxiv.org/abs/1010.0422
19. M. Pachitariu, A. M. Packer, N. Pettit, H. Dalgleish, M. Hausser, and M. Sahani, "Extracting regions of interest from biological images with convolutional sparse block coding," in Proc. Adv. Neural Inf. Process. Syst., vol. 26. 2013, pp. 1745-1753.
20. R. Chalasani, J. C. Principe, and N. Ramakrishnan, "A fast proximal method for convolutional sparse coding," in Proc. Int. Joint Conf. Neural Net. (IJCNN), Aug. 2013, pp. 1-5.
21. H. Bristow, A. Eriksson, and S. Lucey, "Fast convolutional sparse coding," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2013, pp. 391-398.
22. M. Aharon, M. Elad, and A. Bruckstein, "K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation," IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311-4322, Nov. 2006.
23. M. S. Lewicki and T. J. Sejnowski, "Coding time-varying signals using sparse, shift-invariant representations," in Proc. Adv. Neural Inf. Process. Syst., vol. 11. 1999, pp. 730-736.
24. W. Hashimoto and K. Kurata, "Properties of basis functions generated by shift invariant sparse representations of natural images," Biol. Cybern., vol. 83, no. 2, pp. 111-118, 2000.
25. H. Wersing, J. Eggert, and E. Körner, "Sparse coding with invariance constraints," in Artificial Neural Networks and Neural Information Processing (Lecture Notes in Computer Science), vol. 2714. Berlin, Germany: Springer-Verlag, 2003, pp. 385-392.
26. B. A. Olshausen, "Learning sparse, overcomplete representations of time-varying natural images," in Proc. IEEE Int. Conf. Image Process. (ICIP), vol. 1. Sep. 2003, pp. I-41-I-44.
27. T. Blumensath and M. E. Davies, "On shift-invariant sparse coding," in Independent Component Analysis and Blind Signal Separation (Lecture Notes in Computer Science), vol. 3195. Berlin, Germany: Springer-Verlag, 2004, pp. 1205-1212.
28. T. Blumensath and M. E. Davies, "Shift-invariant sparse coding for single channel blind source separation," in Proc. Signal Process. Adapt. Sparse Struct. Represent. (SPARS), Nov. 2005, pp. 75-78.
29. M. D. Plumbley, S. A. Abdallah, T. Blumensath, and M. E. Davies, "Sparse representations of polyphonic music," Signal Process., vol. 86, no. 3, pp. 417-431, 2006.
30. T. Blumensath and M. E. Davies, "Sparse and shift-invariant representations of music," IEEE Trans. Audio, Speech, Language Process., vol. 14, no. 1, pp. 50-57, Jan. 2006.
31. P. Jost, P. Vandergheynst, S. Lesage, and R. Gribonval, "MoTIF: An efficient algorithm for learning translation invariant dictionaries," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), vol. 5. May 2006, pp. 857-860.
32. K. Skretting, J. H. Husøy, and S. O. Aase, "General design algorithm for sparse frame expansions," Signal Process., vol. 86, no. 1, pp. 117-126, Jan. 2006.
33. M. Aharon, "Overcomplete dictionaries for sparse representation of signals," Ph.D. dissertation, Dept. Comput. Sci., Israel Inst. Technol., Haifa, Israel, Nov. 2006.
34. K. Engan, K. Skretting, and J. H. Husøy, "Family of iterative LS-based dictionary learning algorithms, ILS-DLA, for sparse signal representation," Digit. Signal Process., vol. 17, no. 1, pp. 32-49, 2007.
35. R. Grosse, R. Raina, H. Kwong, and A. Y. Ng, "Shift-invariant sparse coding for audio classification," in Proc. 23rd Conf. Uncertainty Artif. Intell. (UAI), Jul. 2007, pp. 149-158.
36. B. Mailhé, S. Lesage, R. Gribonval, F. Bimbot, and P. Vandergheynst, "Shift-invariant dictionary learning for sparse representations: Extending K-SVD," in Proc. 16th Eur. Signal Process. Conf. (EUSIPCO), 2008, pp. 1-5.
37. M. Mørup, M. N. Schmidt, and L. K. Hansen, "Shift invariant sparse coding of image and music data," DTU Informatics, Tech. Univ. Denmark, Kongens Lyngby, Denmark, Tech. Rep. IMM2008-04659, 2008.
38. J. J. Thiagarajan, K. N. Ramamurthy, and A. Spanias, "Shift-invariant sparse representation of images using learned dictionaries," in Proc. IEEE Workshop Mach. Learn. Signal Process. (MLSP), Oct. 2008, pp. 145-150.
39. M. Nakashizuka, H. Nishiura, and Y. Iiguni, "Sparse image representations with shift-invariant tree-structured dictionaries," in Proc. IEEE Int. Conf. Image Process. (ICIP), Nov. 2009, pp. 2145-2148.
40. Y.-K. Tomokusa, M. Nakashizuka, and Y. Iiguni, "Sparse image representations with shift and rotation invariance constraints," in Proc. Int. Symp. Intell. Signal Process. Commun. Syst. (ISPACS), Jan. 2009, pp. 256-259.
41. M. Mørup and M. N. Schmidt, "Transformation invariant sparse coding," in Proc. IEEE Int. Workshop Mach. Learn. Signal Process. (MLSP), Sep. 2011, pp. 1-6.
42. Q. Barthélemy, A. Larue, A. Mayoue, D. Mercier, and J. I. Mars, "Shift and 2D rotation invariant sparse coding for multivariate signals," IEEE Trans. Signal Process., vol. 60, no. 4, pp. 1597-1611, Apr. 2012.
43. G. Pope, C. Aubel, and C. Studer, "Learning phase-invariant dictionaries," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2013, pp. 5979-5983.
44. C. Rusu, B. Dumitrescu, and S. A. Tsaftaris, "Explicit shift-invariant dictionary learning," IEEE Signal Process. Lett., vol. 21, no. 1, pp. 6-9, Jan. 2014.
45. R. P. N. Rao and D. H. Ballard, "Development of localized oriented receptive fields by learning a translation-invariant code for natural images," Netw., Comput. Neural Syst., vol. 9, no. 2, pp. 219-234, 1998.
46. C. Ekanadham, D. Tranchina, and E. P. Simoncelli, "Recovery of sparse translation-invariant signals with continuous basis pursuit," IEEE Trans. Signal Process., vol. 59, no. 10, pp. 4735-4744, Oct. 2011.
47. T. Blumensath and M. E. Davies, "Unsupervised learning of sparse and shift-invariant decompositions of polyphonic music," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), vol. 5. May 2004, pp. 497-500.
48. B. Kong and C. C. Fowlkes, "Fast convolutional sparse coding (FCSC)," Univ. California, Irvine, CA, USA, Tech. Rep., May 2014.
49. J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, "Non-local sparse models for image restoration," in Proc. IEEE 12th Int. Conf. Comput. Vis. (ICCV), Sep./Oct. 2009, pp. 2272-2279.
50. B. S. He, H. Yang, and S. L. Wang, "Alternating direction method with self-adaptive penalty parameters for monotone variational inequalities," J. Optim. Theory Appl., vol. 106, no. 2, pp. 337-356, 2000.
51. R. Grosse. (2007). Shift-Invariant Sparse Coding. Online. Available: http://www.cs.toronto.edu/∼rgrosse/uai07-sisc-code.tar.gz
52. M. V. Afonso, J. M. Bioucas-Dias, and M. A. T. Figueiredo, "Fast image recovery using variable splitting and constrained optimization," IEEE Trans. Image Process., vol. 19, no. 9, pp. 2345-2356, Sep. 2010.
53. K. Engan, S. O. Aase, and J. H. Husøy, "Method of optimal directions for frame design," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), vol. 5. Mar. 1999, pp. 2443-2446.
54. M. V. Afonso, J. M. Bioucas-Dias, and M. A. T. Figueiredo, "An augmented Lagrangian approach to the constrained optimization formulation of imaging inverse problems," IEEE Trans. Image Process., vol. 20, no. 3, pp. 681-695, Mar. 2011.
55. Q. Liu, J. Luo, S. Wang, M. Xiao, and M. Ye, "An augmented Lagrangian multi-scale dictionary learning algorithm," EURASIP J. Adv. Signal Process., vol. 2011, p. 58, Sep. 2011.
56. M. J. Huiskes and M. S. Lew, "The MIR Flickr retrieval evaluation," in Proc. ACM Int. Conf. Multimedia Inf. Retr. (MIR), 2008, pp. 39-43.
57. J. Mairal, G. Sapiro, and M. Elad, "Learning multiscale sparse representations for image and video restoration," Multiscale Model. Simul., vol. 7, no. 1, pp. 214-241, 2008.
58. R. Gribonval, "Fast matching pursuit with a multiscale dictionary of Gaussian chirps," IEEE Trans. Signal Process., vol. 49, no. 5, pp. 994-1001, May 2001.
59. B. A. Olshausen, P. Sallee, and M. S. Lewicki, "Learning sparse image codes using a wavelet pyramid architecture," in Proc. Adv. Neural Inf. Process. Syst., vol. 13. 2000, pp. 887-893.
60. P. Sallee and B. A. Olshausen, "Learning sparse multiscale image representations," in Proc. Adv. Neural Inf. Process. Syst., vol. 15. 2002, pp. 1351-1358.
61. B. Ophir, M. Lustig, and M. Elad, "Multi-scale dictionary learning using wavelets," IEEE J. Sel. Topics Signal Process., vol. 5, no. 5, pp. 1014-1024, Sep. 2011.
62. K. Skretting and K. Engan, "Image compression using learned dictionaries by RLS-DLA and compared with K-SVD," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2011, pp. 1517-1520.
63. J. M. Hughes, D. N. Rockmore, and Y. Wang, "Bayesian learning of sparse multiscale image representations," IEEE Trans. Image Process., vol. 22, no. 12, pp. 4972-4983, Dec. 2013.
64. J. Mairal, G. Sapiro, and M. Elad, "Multiscale sparse image representationwith learned dictionaries," in Proc. IEEE Int. Conf. Image Process. (ICIP), vol. 3. Sep. 2007, pp. 105-108.
65. B. Wohlberg. (2015). Sparse Optimization Research Code (SPORCO). Online. Available: http://math.lanl.gov/∼brendt/Software/SPORCO/
66. W. W. Hager, "Updating the inverse of a matrix," SIAM Rev., vol. 31, no. 2, pp. 221-239, Jun. 1989.
67. N. Egidi and P. Maponi, "A Sherman-Morrison approach to the solution of linear systems," J. Comput. Appl. Math., vol. 189, nos. 1-2, pp. 703-718, May 2006.