K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation

IEEE Transactions on Signal Processing

Volumn 54, Issue 11, 2006, Pages 4311-4322

  Aharon, Michal ^a   Elad, Michael ^a   Bruckstein, Alfred ^a  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

Atom decomposition; Basis pursuit; Codebook; Dictionary; FOCUSS; Gain shape VQ; K means; K SVD; Matching pursuit; Sparse representation; Training; Vector quantization

Indexed keywords

ALGORITHMS; CONSTRAINT THEORY; FEATURE EXTRACTION; GLOSSARIES; INVERSE PROBLEMS; ITERATIVE METHODS; VECTOR QUANTIZATION;

BASIS PURSUIT; K-MEANS CLUSTERING; MATCHING PURSUIT; SPARSE SIGNAL REPRESENTATION;

SIGNAL ENCODING;

EID: 33750383209     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSP.2006.881199     Document Type: Article

References (47)

1. M. W. Marcellin, M. J. Gormish, A. Bilgin, and M. P. Boliek, "An overview of JPEG-2000," in Proc. Data Compression Conf., 2000, pp. 523-541.
2. D. L. Donoho and I. M. Johnstone, "Ideal denoising in an orthonormal basis chosen from a library of bases," Comp. Rend. Acad. Sci., ser. A, vol. 319, pp. 1317-1322, 1994.
3. R. Coifman and D. L. Donoho, "Translation invariant denoising," in Wavelets and Statistics. New York: Springer-Verlag, 1995, vol. 103, Lecture Notes in Statistics, pp. 120-150.
4. E. P. Simoncelli, W. T. Freeman, E. H. Adelsom, and D. J. Heeger, "Shiftable multi-scale transforms," IEEE Trans. Inf. Theory, vol. 38, pp. 587-607, 1992.
5. J. L. Starck, E. J. Candes, and D. L. Donoho, "The curvelet transform for image denoising," IEEE Trans. Image Process., vol. 11, pp. 670-684, 2002.
6. R. Gastaud and J. L. Starck, "Dynamic range compression : A new method based on wavelet transform," in Astron. Data Anal. Software Systems Conf., Strasbourg, 2003.
7. J. L. Starck, M. Elad, and D. L. Donoho, "Image decomposition: Separation of texture from piece-wise smooth content," in SPIE Conf. Signal Image Process.: Wavelet Applicat. Signal Image Process. X, SPIE 48th Annu. Meeting, San Diego, CA, Aug. 3-8, 2003.
8. J. L. Starck, M. Elad, and D. L. Donoho, "Image decomposition via the combination of sparse representations and a variational approach," IEEE Trans. Image Process., Feb. 2004.
9. M. Elad, J. L. Starck, P. Querre, and D. L. Donoho, "Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA)," J. Appl. Comput. Harmon. Anal., submitted for publication.
10. M. Aharon, M. Elad, and A. M. Bruckstein, K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representation Technion - Israel Inst. of Technology, 2005, Tech. Ref..
11. G. Davis, S. Mallat, and M. Avellaneda, "Adaptive greedy approximations," J. Construct. Approx., vol. 13, pp. 57-98, 1997.
12. S. Mallat and Z. Zhang, "Matching pursuits with time-frequency dictionaries," IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397-3415, 1993.
13. S. Chen, S. A. Billings, and W. Luo, "Orthogonal least squares methods and their application to non-linear system identification," Int. J. Contr., vol. 50, no. 5, pp. 1873-96, 1989.
14. G. Davis, S. Mallat, and Z. Zhang, "Adaptive time-frequency decompositions," Opt. Eng., vol. 33, no. 7, pp. 2183-91, 1994.
15. Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, "Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition," in Conf. Rec. 27th Asilomar Conf. Signals, Syst. Comput., 1993, vol. 1.
16. J. A. Tropp, "Greed is good: Algorithmic results for sparse approximation," IEEE Trans. Inf. Theory, vol. 50, pp. 2231-2242, Oct. 2004.
17. S. S. Chen, D. L. Donoho, and M. A. Saunders, "Atomic decomposition by basis pursuit," SIAM Rev., vol. 43, no. 1, pp. 129-159, 2001.
18. I. F. Gorodnitsky and B. D. Rao, "Sparse signal reconstruction from limited data using FOCUSS: A re-weighted norm minimization algorithm," IEEE Trans. Signal Process., vol. 45, pp. 600-616, 1997.
19. B. D. Rao and K. Kreutz-Delgado, "An affine scaling methodology for best basis selection," IEEE Trans. Signal Process., vol. 47, pp. 187-200, 1999.
20. B. D. Rao and K. Kreutz-Delgado, "Deriving algorithms for computing sparse solutions to linear inverse problems," in Conf. Rec. 31st Asilomar Conf. Signals, Syst. Comput., 1998, vol. 1, pp. 955-959.
21. B. D. Rao, K. Engan, S. F. Cotter, J. Palmer, and K. Kreutz-Delgado, "Subset selection in noise based on diversity measure minimization," IEEE Trans. Signal Process., vol. 51, pp. 760-770, 2003.
22. M. S. Lewicki and B. A. Olshausen, "A probabilistic framework for the adaptation and comparison of image codes," J. Opt. Soc. Amer. A: Opt., Image Sci. Vision, vol. 16, no. 7, pp. 1587-1601, 1999.
23. B. A. Olshausen and D. J. Field, "Natural image statistics and efficient coding," Network Comp. Neural Syst., vol. 7, no. 2, pp. 333-339, 1996.
24. B. A. Olshausen and B. J. Field, "Sparse coding with an overcomplete basis set: A strategy employed by v1?," Vision Res., vol. 37, pp. 3311-3325, 1997.
25. M. S. Lewicki and T. J. Sejnowski, "Learning overcomplete representations," Neural Comp., vol. 12, pp. 337-365, 2000.
26. D. L. Donoho and X. Huo, "Uncertainty principles and ideal atomic decomposition," IEEE Trans. Inf. Theory, vol. 47, pp. 2845-62, 1999.
27. M. Elad and A. M. Bruckstein, "A generalized uncertainty principle and sparse representation in pairs of bases," IEEE Trans. Inf. Theory, vol. 48, pp. 2558-2567, Sep. 2002.
28. 1 minimization," Proc. Nat. Acad. Sci., vol. 100, no. 5, pp. 2197-2202, 2003.
29. R. Gribonval and M. Nielsen, "Sparse decompositions in unions of bases," IEEE Trans. Inf. Theory, vol. 49, pp. 3320-3325, 2003.
30. J. J. Fuchs, "On sparse representations in arbitrary redundant bases," IEEE Trans. Inf. Theory, vol. 50, pp. 1341-1344, 2004.
31. D. L. Donoho, M. Elad, and V. Temlyakov, "Stable recovery of sparse overcomplete representations in the presence of noise," submitted for publication.
32. J. A. Tropp, "Just relax: Convex programming methods for subset selection and sparse approximation," UT-Austin, 2004, ICES Rep. 04-04.
33. 1 -norm solution is also the sparsest solution," Dept. of Statistics, Stanford Univ., 2004, Tech. Rep. 2004-10.
34. 1 -norm near-solution approximates the sparsest near-solution," Dept. of Statistics, Stanford Univ., 2004, Tech. Rep. 2004-11.
35. E. Candes and J. Romberg, "Quantitative robust uncertainty principles and optimally sparse decompositions," to be published.
36. K. Engan, S. O. Aase, and J. H. Husoy, "Multi-frame compression: Theory and design," EURASIP Signal Process., vol. 80, no. 10, pp. 2121-2140, 2000.
37. K. Kreutz-Delgado, J. F. Murray, B. D. Rao, K. Engan, T. Lee, and T. J. Sejnowski, "Dictionary learning algorithms for sparse representation," Neural Comp., vol. 15, no. 2, pp. 349-396, 2003.
38. J. A. Tropp, "Topics in sparse approximation," Ph.D. dissertation, Univ. of Texas at Austin, Austin, 2004.
39. A. Gersho and R. M. Gray, Vector Quantization and Signal Compression. Norwell, MA: Kluwer Academic, 1991.
40. K. Engan, S. O. Aase, and J. H. Hakon-Husoy, "Method of optimal directions for frame design," in IEEE Int. Conf. Acoust., Speech, Signal Process., 1999, vol. 5, pp. 2443-2446.
41. K. Engan, B. D. Rao, and K. Kreutz-Delgado, "Frame design using focuss with method of optimal directions (mod)," in Norwegian Signal Process. Symp., 1999, vol. 65-69.
42. J. F. Murray and K. Kreutz-Delgado, "An improved focuss-based learning algorithm for solving sparse linear inverse problems," in IEEE Int. Conf. Signals, Syst. Comput., 2001, vol. 4119-53.
43. A. J. Bell and T. J. Sejnowski, "An information maximisation approach to blind separation and blind deconvolution," Neural Comp., vol. 7, no. 6, pp. 1129-1159, 1996.
44. K. Kreutz-Delgado and B. D. Rao, "FOCUSS-based dictionary learning algorithms," in Wavelet Applications in Signal and Image Process. VIII, 2000, vol. 4119-53.
45. S. Lesage, R. Gribonval, F. Bimbot, and L. Benaroya, "Learning unions of orthonormal bases with thresholded singular value decomposition," in IEEE Int. Conf. Acoust., Speech, Signal Process., 2005.
46. S. Sardy, A. G. Bruce, and P. Tseng, "Block coordinate relaxation methods for nonparametric signal denoising withwavelet dictionaries," J. Comp. Graph. Statist., vol. 9, pp. 361-379, 2000.
47. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. Roy. Statist. Soc., ser. B, vol. 39, no. 1, pp. 1-38, 1977.