Simultaneous codeword optimization (SimCO) for dictionary update and learning

IEEE Transactions on Signal Processing

Volumn 60, Issue 12, 2012, Pages 6340-6353

  Dai, Wei ^a   Xu, Tao ^b   Wang, Wenwu ^b  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF SURREY   (United Kingdom)

Author keywords

Dictionary learning; Grassmann manifold; optimization; singularity; sparse representation

Indexed keywords

CODE-WORDS; CODEWORD; DICTIONARY LEARNING; GRASSMANN MANIFOLD; LINEAR COMBINATIONS; LOCAL MINIMUMS; OVER-COMPLETE; SECOND ORDER OPTIMIZATION; SINGULAR POINTS; SINGULARITY; SPARSE CODING; SPARSE REPRESENTATION; SPARSITY PATTERNS; TRAINING SIGNAL;

ELECTRONICS ENGINEERING; SIGNAL PROCESSING;

OPTIMIZATION;

EID: 84870486964     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSP.2012.2215026     Document Type: Article

References (68)

1. P. Foldiak, "Forming sparse representations by local anti-Hebbian learning," Biolog. Cybern., vol. 64, pp. 165-170, 1990.
2. M. S. Lewicki and T. J. Sejnowski, "Learning overcomplete representations," Neural Comput., vol. 12, no. 2, pp. 337-365, 2000.
3. J. Tropp, "Topics in Sparse Approximations," Ph.D., Computational and Applied Math. Dept., Univ. Texas, Austin, 2004.
4. B.A. Olshausen, C. F. Cadieu, and D. K. Warland, "Learning real and complex overcomplete representations fromthe statistics of natural images," Proc. SPIE, vol. 7446, pp. 74460S-74460S-11, 2009.
5. B. A. Olshausen and D. J. Field, "Emergence of simple-cell receptive field properties by learning a sparse code for natural images," Nature, vol. 381, pp. 607-609, 1996. (Pubitemid 26177476)
6. I. Tošic and P. Frossard, "Dictionary learning for stereo image representation," IEEE Trans. Image Process., vol. 20, no. 4, pp. 921-934, 2011.
7. M. Zibulevsky and B. A. Pearlmutter, "Blind source separation by sparse decomposition of a signal dictionary," Neural Comput., vol. 13, no. 4, pp. 863-882, 2001. (Pubitemid 33594308)
8. R. Gribonval, "Sparse decomposition of stereo signals with matching pursuit and application to blind separation of more than two sources from a stereo mixture," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2002, vol. 3, pp. 3057-3060.
9. T. Xu and W. Wang, "Methods for learning adaptive dictionary in underdetermined speech separation," in Proc. IEEE Int. Conf. Mach. Learn. Signal Process., Beijing, China, 2011, pp. 1-6.
10. M. Aharon, M. Elad, and A. Bruckstein, "K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation," IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311-4322, 2006. (Pubitemid 44637761)
11. M. G. Jafari and M. D. Plumbley, "Fast dictionary learning for sparse representations of speech signals," IEEE J. Sel. Topics in Signal Process., vol. 5, no. 5, pp. 1025-1031, 2011.
12. P. Schmid-Saugeon and A. Zakhor, "Dictionary design for matching pursuit and application to motion-compensated video coding," IEEE Trans. Circuits Syst. Video Technol., vol. 14, no. 6, pp. 880-886, 2004.
13. E. Kokiopoulou and P. Frossard, "Semantic coding by supervised dimensionality reduction," IEEE Trans. Multimed., vol. 10, no. 5, pp. 806-818, 2008.
14. M. D. Plumbley, T. Blumensath, L. Daudet, R. Gribonval, and M. E. Davies, "Sparse representations in audio and music: From coding to source separation," Proc. IEEE, vol. 98, no. 6, pp. 995-1005, 2010.
15. K. Huang and S. Aviyente, "Sparse representation for signal classification," in Proc. Conf. Neur. Inf. Process. Syst., 2006, pp. 609-616.
16. J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, "Discriminative learned dictionaries for local image analysis," in Proc. IEEE Int. Conf. Comput. Vision Pattern Recogn., 2008, pp. 1-8.
17. K. Schnass and P.Vandergheynst, "A union of incoherent spacesmodel for classification," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2010, pp. 5490-5493.
18. J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, "Robust face recognition via sparse representation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 2, pp. 210-227, 2009.
19. V. Cevher and A. Krause, "Greedy dictionary selection for sparse representation," IEEE J. Sel. Topics Signal Process., vol. 5, no. 5, pp. 979-988, 2011.
20. A. Adler, V. Emiya, M. G. Jafari, M. Elad, R. Gribonval, and M. D. Plumbley, "Audio inpainting," in Proc. IEEE Trans. on Audio, Speech Lang. Process., 2011 [Online]. Available: http://www.cs.technion. ac.il/elad/publications/journals
21. R. G. Baraniuk, E. J. Candès, M. Elad, and Y. Ma, "Applications of sparse representation and compressive sensing," Proc. IEEE, vol. 98, no. 6, pp. 906-909, 2010.
22. S. Chen, D. Donoho, and M. Saunders, "Atomic decomposition by basis pursuit," SIAM J. Sci. Comput., vol. 20, no. 1, pp. 33-61, 1999. (Pubitemid 128689501)
23. S. G. Mallat and Z. Zhang, "Matching pursuits with time-frequency dictionaries," IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397-3415, 1993.
24. Y. C. Pati, R. Rezaiifar, and P. S.Krishnaprasad, "Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition," in Proc. IEEE Asilomar Conf. Signals, Syst. Comput., 1993, pp. 40-44.
25. J. Tropp and A. C. Gilbert, "Signal recovery from random measurements via orthogonal matching pursuit," IEEE Trans. Inf. Theory, vol. 53, no. 12, pp. 4655-4666, 2007.
26. W. Dai and O. Milenkovic, "Subspace pursuit for compressive sensing signal reconstruction," IEEE Trans. Inf. Theory, vol. 55, pp. 2230-2249, 2009.
27. D. Needell and J. A. Tropp, "Cosamp: Iterative signal recovery from incomplete and inaccurate samples," Appl. Comp. Harmon. Anal., vol. 26, no. 3, pp. 301-321, May 2009.
28. R. Tibshirani, "Regression shrinkage and selection via the lasso," J. R. Stat. Soc. Ser. B (Method.), vol. 58, no. 1, pp. 267-288, 1996.
29. I. Gorodnitsky and B. Rao, "Sparse signal reconstruction from limited data using FOCUSS: A re-weighted minimum norm algorithm," IEEE Trans. Signal Process., vol. 45, no. 3, pp. 600-616, 1997. (Pubitemid 127765947)
30. T. Blumensath and M. E. Davies, "Gradient pursuits," IEEE Trans. Signal Process., vol. 56, no. 6, pp. 2370-2382, 2008. (Pubitemid 351795890)
31. I. W. Selesnick, R. G. Baraniuk, and N. C. Kingsbury, "The dual-tree complex wavelet transform," IEEE Signal Process. Mag., vol. 22, no. 6, pp. 123-151, 2005. (Pubitemid 46404908)
32. E. J. Candès and D. L. Donoho, "Curvelets-A surprisingly effective nonadaptive representation for objects with edges," in Curves and Surface Fitting: Saint-Malo Proceedings, A. Cohen, C. Rabut, and L. Schumaker, Eds. Nashville, TN: Vanderbilt Univ. Press, 2000, pp. 105-120.
33. M. N. Do and M. Vetterli, "The contourlet transform: An efficient directional multiresolution image representation," IEEE Trans. Image Process., vol. 14, no. 12, pp. 2091-2106, 2005. (Pubitemid 41781516)
34. E. LePennec and S. Mallat, "Sparse geometric image representations with bandelets," IEEE Trans. Image Process., vol. 14, no. 4, pp. 423-438, 2005. (Pubitemid 40552380)
35. B. A. Olshausen and D. J. Field, "Sparse coding with an overcomplete basis set: A strategy employed by V1," Vis. Res, vol. 37, no. 23, pp. 3311-3325, 1997. (Pubitemid 27493805)
36. K. Engan and S. A. H. Husøy, "Method of optimal directions for frame design," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Process., 1999, vol. 5, pp. 2443-2446.
37. K. Kreutz-Delgado, J. Murray, B. Rao, K. Engan, T.-W. Lee, and T. J. Sejnowski, "Dictionary learning algorithms for sparse representation," Neural Comput., vol. 15, no. 2, pp. 349-396, 2003. (Pubitemid 37049825)
38. D. P. Wipf and B. D. Rao, "Sparse Bayesian learning for basis selection," IEEE Trans. Signal Process., vol. 52, no. 8, pp. 2153-2164, 2004.
39. M. D. Plumbley, "Dictionary learning for L1-exact sparse coding," Lect. Notes Comput. Sci, vol. 4666, pp. 406-413, 2007.
40. M. G. Jafari andM. D. Plumbley, "Speech denoising based on a greedy adaptive dictionary algorithm," in Proc. Conf. Eur. Signal Process., 2009, pp. 1423-1426.
41. R. Gribonval and K. Schnass, "Dictionary identification-Sparse matrix factorization via l1 minimization," IEEE Trans. Inf. Theory, vol. 56, no. 7, pp. 3523-3539, 2010.
42. Q. Geng, H. Wang, and J. Wright, "On the local correctness of L minimization for dictionary learning," 2011 [Online]. Available: http://arxiv.org/abs/1101.5672
43. R. Rubinstein, A.M. Bruckstein, and M. Elad, "Dictionaries for sparse representation modelling," Proc. IEEE, vol. 98, no. 6, pp. 1045-1057, 2010.
44. K. Engan, K. Skretting, and J. H. Husøy, "Family of iterative LS-based dictionary learning algorithms, ILS-DLA, for sparse signal representation," Dig. Signal Process., vol. 17, no. 1, pp. 32-49, 2007. (Pubitemid 44828547)
45. K. Skretting and K. Engan, "Recursive least squares dictionary learning algorithm," IEEE Trans. Signal Process., vol. 58, no. 4, pp. 2121-2130, 2010.
46. M. Yaghoobi, T. Blumensath, and M. E. Davies, "Dictionary learning for sparse approximations with the majorization method," IEEE Trans. Signal Process., vol. 57, no. 6, pp. 2178-2191, 2009.
47. M. Yaghoobi, L. Daudet, and M. E. Davies, "Parametric dictionary design for sparse coding," IEEE Trans. Signal Process., vol. 57, no. 12, pp. 4800-4810, 2009.
48. P. Sallee and B. A. Olshausen, "Learning sparse multiscale image representations," in Proc. Conf. Neural Inf. Process. Syst., 2002, pp. 1327-1334.
49. C. Cadieu and B. A. Olshausen, "Learning transformational invariants from time-varying natural images," in Proc. Conf. Neur. Inf. Process. Syst., 2009, vol. 21, pp. 209-216.
50. T. Blumensath and M. Davies, "Sparse and shift-invariant representations of music," IEEE Trans. Speech Audio Process., vol. 14, no. 1, pp. 50-57, 2006. (Pubitemid 43863452)
51. B. Mailhé, S. Lesage, R. Gribonval, F. Bimbot, and P. Vandergheynst, "Shift invariant dictionary learning for sparse representations: Extending K-SVD," in Proc. Eur. Signal Process. Conf., 2008, vol. 4.
52. M. Aharon and M. Elad, "Sparse and redundant modeling of image content using an image-signature-dictionary," SIAM J. Imag. Sci., vol. 1, no. 3, pp. 228-247, 2008.
53. P. Smaragdis, B. Raj, and M. Shashanka, "Sparse and shift-invariant feature extraction from non-negative data," in Proc. IEEE Int. Conf. on Audio Speech Signal Process., 2008, pp. 2069-2072.
54. R. Gribonval and M. Nielsen, "Sparse representations in unions of bases," IEEE Trans. Inf. Theory, vol. 49, no. 12, pp. 3320-3325, 2003.
55. P. Jost, P. Vandergheynst, S. Lesage, and R. Gribonval, "Bmotif: An efficient algorithm for learning translation invariant dictionaries," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2006, vol. 5, pp. 857-860.
56. K. Labusch, E. Barth, and T. Martinetz, "Robust and fast learning of sparse codes with stochastic gradient descent," IEEE J. Sel. Top. Signal Process., vol. 5, no. 5, pp. 1048-1060, 2011.
57. J.Mairal, F. Bach, J. Ponce, andG. Sapiro, "Online learning for matrix factorization and sparse coding," J. Mach. Learn. Res., vol. 11, pp. 16-60, 2010.
58. G. Monaci, P. Vandergheynst, and F. T. Sommer, "Learning bimodal structure in audio-visual data," IEEE Trans. Neural Netw., vol. 20, no. 12, pp. 1898-1910, 2009.
59. A. L. Casanovas, G. Monaci, P. Vandergheynst, and R. Gribonval, "Blind audiovisual source separation based on sparse redundant representations," IEEE Trans. Multimed., vol. 12, no. 5, pp. 358-371, 2010.
60. I. Tošic and P. Frossard, "Dictionary learning: What is the right representation for my signal," IEEE Signal Process. Mag., vol. 28, no. 2, pp. 27-38, Mar. 2011.
61. E. Candes and T. Tao, "Decoding by linear programming," IEEE Trans. Inf. Theory, vol. 51, no. 12, pp. 4203-4215, Dec. 2005. (Pubitemid 41800353)
62. A. Edelman, T. A. Arias, and S. T. Smith, "The geometry of algorithms with orthogonality constraints," SIAM J. Matrix Anal. Appl., vol. 20, no. 2, pp. 303-353, 1999. (Pubitemid 129333771)
63. W. Dai, E. Kerman, and O. Milenkovic, "A geometric approach to lowrank matrix completion," IEEE Trans. Inf. Theory, vol. 58, no. 1, pp. 237-247, Jan. 2012.
64. J. Nocedal and S. J. Wright, Numerical Optimization. NewYork: Springer, 2006.
65. W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes: The art of Scientific Computing, Golden Section Search in one Dimension, 3rd ed. Cambridge, U.K.: Cambridge Univ. Press, 2007, ch. 10.2.
66. P.-A. Absil, R. Mahony, and R. Sepulchre, Optimization Algorithms on Matrix Manifolds. Princeton, NJ: Princeton Univ. Press, 2008.
67. M. Elad and M. Aharon, "Image denoising via sparse and redundant representations over learned dictionaries," IEEE Trans. Image Process., vol. 15, no. 12, pp. 3736-3745, Dec. 2006. (Pubitemid 44811686)
68. W. Dai, Y. Liu, and B. Rider, "Quantization bounds on Grassmann manifolds and applications to MIMO systems," IEEE Trans. Inf. Theory, vol. 54, no. 3, pp. 1108-1123, Mar. 2008. (Pubitemid 351418908)