An MDL framework for sparse coding and dictionary learning

IEEE Transactions on Signal Processing

Volumn 60, Issue 6, 2012, Pages 2913-2927

  Ramirez, Ignacio ^a   Sapiro, Guillermo ^a  

^a University of Minnesota   (United States)

Author keywords

Classification; Denoising; Dictionary learning; Low rank matrix completion; Minimum description length principle (MDL); Sparse coding

Indexed keywords

DE-NOISING; DICTIONARY LEARNING; LOW-RANK MATRICES; MINIMUM DESCRIPTION LENGTH PRINCIPLE; SPARSE CODING;

CLASSIFICATION (OF INFORMATION); INFORMATION THEORY; LEARNING ALGORITHMS; SIGNAL PROCESSING; STATISTICAL METHODS;

MATRIX ALGEBRA;

EID: 84861121463     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSP.2012.2187203     Document Type: Article

References (50)

1. A. Bruckstein, D. Donoho, and M. Elad, "From sparse solutions of systems of equations to sparse modeling of signals and images," SIAM Rev., vol. 51, no. 1, pp. 34-81, Feb. 2009.
2. R. Rubinstein, A. Bruckstein, and M. Elad, "Dictionaries for sparse representation modeling," Proc. IEEE, vol. 98, no. 6, pp. 1045-1057, Jun. 2010.
3. J. Wright, Y.Ma, J.Mairal, G. Sapiro, T. Huang, and S. Yan, "Sparse representation for computer vision and pattern recognition," Proc. IEEE, vol. 98, no. 6, pp. 1031-1044, Jun. 2010.
4. M. Zhou, H. Chen, J. Paisley, L. Ren, L. Li, Z. Xing, D. Dunson, G. Sapiro, and L. Carin, "Nonparametric Bayesian dictionary learning for analysis of noisy and incomplete images," IEEE Trans. Image Process., 2011, submitted for publication.
5. H. Akaike, "A new look at the statistical model identification, " IEEE Trans. Autom. Contr., vol. 19, pp. 716-723, 1974.
6. G. Schwartz, "Estimating the dimension of a model," Ann. Statist., vol. 6, no. 2, pp. 461-464, 1978.
7. J. Rissanen, "Modeling by shortest data description," Automatica, vol. 14, pp. 465-471, 1978.
8. J. Rissanen, "Universal coding, information, prediction, and estimation," IEEE Trans. Inf. Theory, vol. 30, no. 4, pp. 629-636, 1984.
9. A. Barron, J. Rissanen, and B. Yu, "The minimum description length principle in coding and modeling," IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2743-2760, 1998.
10. T. Cover and J. Thomas, Elements of Information Theory, 2nd ed. New York: Wiley, 2006.
11. S. Kritchman and B. Nadler, "Non-parametric detection of the number of signals: Hypothesis testing and random matrix theory," IEEE Trans. Signal Process., vol. 57, no. 10, pp. 3930-3941, Oct. 2009.
12. B. Nadler, "Nonparametric detection of signals by information theoretic criteria: Performance analysis and an improved estimator," IEEE Trans. Signal Process., vol. 58, no. 5, pp. 2746-2756, May 2010.
13. N. Saito, "Simultaneous noise suppression and signal compression using a library of orthonormal bases and the MDL criterion," in Wavelets in Geophysics, E. Foufoula-Georgiou and P. Kumar, Eds. New York: Academic, 1994, pp. 299-324.
14. H. Krim and J.-C. Pesquet, On the Statistics of Best Bases CriteriaMIT LIDS, Tech. Rep., 1995.
15. P. Moulin and J. Liu, "Analysis of multiresolution image denoising schemes using generalized-Gaussian and complexity priors," IEEE Trans. Inf. Theory, Apr. 1999.
16. J. Rissanen, "MDL denoising," IEEE Trans. Inf. Theory, vol. 46, no. 7, pp. 2537-2543, 2000.
17. T. Roos, P. Myllymäki, and J. Rissanen, "MDL denoising revisited," IEEE Trans. Signal Process., vol. 57, no. 9, pp. 3347-3360, 2009.
18. I. Ramírez and G. Sapiro, "Sparse coding and dictionary learning based on the MDL principle," in Proc. ICASSP 2011, Prague, Czech. Republic, May 2011.
19. I. Ramírez and G. Sapiro, "Low-rank data modeling via the minimum description length principle," in Proc. ICASSP 2012, 2011, arXiv:1109. 6297v1, to be published.
20. I. Ramírez and G. Sapiro, "Universal regularizers for robust sparse coding and modeling," Aug. 2010, Preprint available in arXiv:1003. 2941v2 [cs.IT].
21. P. J. Huber, "Robust estimation of a location parameter," Ann. Statist., vol. 53, pp. 73-101, 1964.
22. S. Mallat and Z. Zhang, "Matching pursuit in a time-frequency dictionary," IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397-3415, 1993.
23. R. Tibshirani, "Regression shrinkage and selection via the LASSO," J. Royal Statist. Soc.: Ser. B, vol. 58, no. 1, pp. 267-288, 1996.
24. E. J. Candès, "Compressive sampling," in Proc. Int. Congress ofMath., Aug. 2006, vol. 3.
25. D. L. Donoho and I. M. Johnston, "Ideal spatial adaptation by wavelet shrinkage," Biometrika, vol. 81, no. 3, pp. 425-455, Sep. 1994.
26. J. Friedman, H. Hastie, and R. Tibshirani, "Regularized paths for generalized linear models via coordinate descent," J. Stat. Soft., vol. 33, no. 1, 2008.
27. A. Beck and M. Teboulle, "A fast iterative shrinkage-thresholding algorithm for linear inverse problems," SIAM J. Imag. Sci., vol. 2, no. 1, pp. 183-202, 2009.
28. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, "Least angle regression," Ann. Statist., vol. 32, no. 2, pp. 407-499, 2004.
29. M. Aharon, M. Elad, and A. Bruckstein, "The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representations," IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311-4322, Nov. 2006.
30. J.Mairal, F. Bach, J. Ponce, andG. Sapiro, "Online learning for matrix factorization and sparse coding," J. Mach. Learn. Res., vol. 11, pp. 19-60, 2010.
31. J. Mairal, M. Elad, and G. Sapiro, "Sparse representation for color image restoration," IEEE Trans. Image Process., vol. 17, no. 1, pp. 53-69, Jan. 2008.
32. J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, "Discriminative learned dictionaries for local image analysis," in Proc. IEEE CVPR, Anchorage, AK, Jun. 2008.
33. I. Ramírez, P. Sprechmann, and G. Sapiro, "Classification and clustering via dictionary learning with structured incoherence and shared features," in Proc. CVPR, Jun. 2010.
34. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: DataMining, Inference and Prediction, 2nd ed. NewYork: Springer, Feb. 2009.
35. P. Grünwald, The Minimum Description Length Principle. Cambridge, U.K.: MIT Press, Jun. 2007.
36. Y. Shtarkov, "Universal sequential coding of single messages," Probl. Inf. Transmiss., vol. 23, no. 3, pp. 3-17, Jul. 1987.
37. H. Ishwaran and J. S. Rao, "Spike and slab variable selection: Frequentist and Bayesian strategies," Ann. Statist., vol. 33, no. 2, pp. 730-773, 2005.
38. T. M. Cover, "Enumerative source encoding," IEEE Trans. Inf. Theory, vol. 19, no. 1, pp. 73-77, 1973.
39. G. Motta, E. Ordentlich, I. Ramirez, G. Seroussi, and M. Weinberger, "The iDUDE framework for grayscale image denoising," IEEE Trans. Image Process., vol. 20, no. 1, pp. 1-21, Jan. 2011.
40. E. Lam and J. Goodman, "A mathematical analysis of the DCT coefficient distributions for images," IEEE Trans. Image Process., vol. 9, no. 10, pp. 1661-1666, 2000.
41. R. E. Krichevsky and V. K. Trofimov, "The performance of universal encoding," IEEE Trans. Inf. Theory, no. 2, pp. 199-207, 1981.
42. M. Zhou, H. Yang, G. Sapiro, D. Dunson, and L. Carin, "Dependent hierarchical beta process for image interpolation and denoising," in Proc. AISTATS, 2011.
43. S. Chen,D.Donoho, andM. Saunders, "Atomic decomposition by basis pursuit," SIAM J. Scientif. Comput., vol. 20, no. 1, pp. 33-61, 1998.
44. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, "Image denoising by sparse 3d transform-domain collaborative filtering," IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080-2095, Aug. 2007.
45. J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, "Non-local sparse models for image restoration," in Proc. ICCV, Oct. 2009, pp. 2272-2279.
46. E. Candès, X. Li, Y. Ma, and J. Wright, "Robust principal component analysis?," J. ACM, vol. 58, no. 3, May 2011.
47. J. Wright, A. Ganesh, S. Rao, Y. Peng, and Y. Ma, "Robust principal component analysis: Exact recovery of corrupted low-rank matrices by convex optimization," in Proc. Adv. NIPS, Dec. 2009.
48. Z. Lin, M. Chen, and Y. Ma, "The augmented Lagrange multiplier method for exact recovery of corrupted low-rank matrices," [Online]. Available: http://arxiv.org/abs/1009.5055
49. J. Rissanen, "A universal prior for integers and estimation byminimum description length," Ann. Statist., vol. 11, no. 2, pp. 416-431, 1983.
50. L. Li, W. Huang, I. Gu, and Q. Tian, "Statistical modeling of complex backgrounds for foreground object detection," IEEE Trans. Image Process., vol. 13, no. 11, pp. 1459-1472, 2004.