Sample complexity of dictionary learning and other matrix factorizations

IEEE Transactions on Information Theory

Volumn 61, Issue 6, 2015, Pages 3469-3486

  Gribonval, Rémi ^a   Jenatton, Rodolphe ^b,c,d   Bach, Francis ^b   Kleinsteuber, Martin ^e   Seibert, Matthias ^e  

^a CAMPUS DE BEAULIEU   (France)

^b ECOLE NORMALE SUPÉRIEURE   (France)

^c CENTRE DE MATHÉMATIQUES APPLIQUÉES   (France)

^d AMAZON   (United States)

^e TECHNICAL UNIVERSITY OF MUNICH   (Germany)

Author keywords

Dictionary learning; K means clustering; non negative matrix factorization; principal component analysis; sample complexity; sparse coding; structured learning

Indexed keywords

ARTIFICIAL INTELLIGENCE; COST FUNCTIONS; DATA HANDLING; LEARNING SYSTEMS; MATRIX ALGEBRA; PRINCIPAL COMPONENT ANALYSIS; SIGNAL PROCESSING; TENSORS;

DICTIONARY LEARNING; K-MEANS CLUSTERING; NONNEGATIVE MATRIX FACTORIZATION; SAMPLE COMPLEXITY; SPARSE CODING; STRUCTURED LEARNING;

FACTORIZATION;

EID: 84930227237     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2015.2424238     Document Type: Conference Paper

References (55)

1. S. Mallat, A Wavelet Tour of Signal Processing: The Sparse Way, 3rd ed. New York, NY, USA: Academic, 2008.
2. M. Elad, M. A. T. Figueiredo, and Y. Ma, "On the role of sparse and redundant representations in image processing," Proc. IEEE, vol. 98, no. 6, pp. 972-982, Jun. 2010.
3. M. Elad, Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing. New York, NY, USA: Springer-Verlag, 2010.
4. L. R. Tucker, "Some mathematical notes on three-mode factor analysis," Psychometrika, vol. 31, no. 3, pp. 279-311, 1966.
5. B. A. Olshausen and D. J. Field, "Sparse coding with an overcomplete basis set: A strategy employed by V1?" Vis. Res., vol. 37, no. 23, pp. 3311-3325, 1997.
6. K. Engan, S. O. Aase, and J. Håkon Husøy, "Method of optimal directions for frame design," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Mar. 1999, pp. 2443-2446.
7. K. Kreutz-Delgado, J. F. Murray, B. D. Rao, K. Engan, T.-W. Lee, and T. J. Sejnowski, "Dictionary learning algorithms for sparse representation," Neural Comput., vol. 15, no. 2, pp. 349-396, 2003.
8. M. Aharon, M. Elad, and A. Bruckstein, "K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation," IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311-4322, Nov. 2006.
9. J. Mairal, F. Bach, J. Ponce, and G. Sapiro, "Online learning for matrix factorization and sparse coding," J. Mach. Learn. Res., vol. 11, no. 1, pp. 19-60, 2010.
10. I. Tošić and P. Frossard, "Dictionary learning," IEEE Signal Process. Mag., vol. 28, no. 2, pp. 27-38, Mar. 2011.
11. R. Rubinstein, A. M. Bruckstein, and M. Elad, "Dictionaries for sparse representation modeling," Proc. IEEE, vol. 98, no. 6, pp. 1045-1057, Jun. 2010.
12. S. Hawe, M. Seibert, and M. Kleinsteuber, "Separable dictionary learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2013, pp. 438-445.
13. R. Rubinstein, M. Zibulevsky, and M. Elad, "Double sparsity: Learning sparse dictionaries for sparse signal approximation," IEEE Trans. Signal Process., vol. 58, no. 3, pp. 1553-1564, Mar. 2010.
14. D. D. Lee and H. S. Seung, "Algorithms for non-negative matrix factorization," in Advances in Neural Information Processing Systems, vol. 13. Cambridge, MA, USA: MIT Press, 2001, pp. 556-562.
15. A. Gersho and R. M. Gray, Vector Quantization and Signal Compression. New York, NY, USA: Springer-Verlag, 1992.
16. A. d'Aspremont, L. El Ghaoui, M. I. Jordan, and G. R. G. Lanckriet. (Jun. 2004). "A direct formulation for sparse PCA using semidefinite programming." [Online]. Available: http://arxiv.org/abs/cs/0406021
17. D. M. Witten, R. Tibshirani, and T. Hastie, "A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis," Biostatistics, vol. 10, no. 3, pp. 515-534, 2009.
18. Y. Zhang and L. El Ghaoui. (Oct. 2012). "Large-scale sparse principal component analysis with application to text data." [Online]. Available: http://arxiv.org/abs/1210.7054
19. A. Maurer and M. Pontil, "K-dimensional coding schemes in Hilbert spaces," IEEE Trans. Inf. Theory, vol. 56, no. 11, pp. 5839-5846, Nov. 2010.
20. D. Vainsencher, S. Mannor, and A. M. Bruckstein, "The sample complexity of dictionary learning," J. Mach. Learn. Res., vol. 12, pp. 3259-3281, Feb. 2011.
21. R. Jenatton, J.-Y. Audibert, and F. Bach, "Structured variable selection with sparsity-inducing norms," J. Mach. Learn. Res., vol. 12, pp. 2777-2824, Feb. 2011.
22. R. Chartrand, "Exact reconstruction of sparse signals via nonconvex minimization," IEEE Signal Process. Lett., vol. 14, no. 10, pp. 707-710, Oct. 2007.
23. B. Mailhé, S. Lesage, R. Gribonval, F. Bimbot, and P. Vandergheynst, "Shift-invariant dictionary learning for sparse representations: Extending K-SVD," in Proc. 16th Eur. Signal Process. Conf. (EUSIPCO), 2008.
24. R. Jenatton, J. Mairal, G. Obozinski, and F. Bach, "Proximal methods for hierarchical sparse coding," J. Mach. Learn. Res., vol. 12, pp. 2297-2334, Feb. 2011.
25. L. De Lathauwer, B. De Moor, and J. Vandewalle, "A multilinear singular value decomposition," SIAM J. Matrix Anal. Appl., vol. 21, no. 4, pp. 1253-1278, 2000.
26. R. Jenatton, R. Gribonval, and F. Bach, "Local stability and robustness of sparse dictionary learning in the presence of noise," CMAP, Res. Rep. hal-00737152, Oct. 2012.
27. V. V. Buldygin and Y. V. Kozachenko, Metric Characterization of Random Variables and Random Processes, vol. 188. Providence, RI, USA: AMS, 2000.
28. D. A. Spielman, H. Wang, and J. Wright, "Exact recovery of sparselyused dictionaries," J. Mach. Learn. Res., vol. 23, pp. 37.1-37.18, Jun. 2012.
29. S. Arora, R. Ge, and A. Moitra. (Aug. 2013). "New algorithms for learning incoherent and overcomplete dictionaries." [Online]. Available: http://arxiv.org/abs/1308.6273
30. A. Agarwal, A. Anandkumar, P. Jain, and P. Netrapalli. (Oct. 2013). "Learning sparsely used overcomplete dictionaries via alternating minimization." [Online]. Available: http://arxiv.org/abs/1310.7991
31. F. Cucker and S. Smale, "On the mathematical foundations of learning," Bull. Amer. Math. Soc., vol. 39, no. 1, pp. 1-49, 2002.
32. E. J. Candès, J. Romberg, and T. Tao, "Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information," IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489-509, Feb. 2006.
33. S. N. Negahban, P. Ravikumar, M. J. Wainwright, and B. Yu, "A unified framework for high-dimensional analysis of m-estimators with decomposable regularizers," Statist. Sci., vol. 27, no. 4, pp. 538-557, Nov. 2012.
34. S. P. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge Univ. Press, 2004.
35. C. Hage and M. Kleinsteuber, "Robust PCA and subspace tracking from incomplete observations using -0-surrogates," Comput. Statist., vol. 29, nos. 3-4, pp. 467-487, 2014.
36. R. Vershynin. (2010). "Introduction to the non-asymptotic analysis of random matrices." [Online]. Available: http://arxiv.org/abs/1011.3027
37. D. Hsu, S. M. Kakade, and T. Zhang, "A tail inequality for quadratic forms of subgaussian random vectors," Electron. Commun. Probab., vol. 17, no. 52, pp. 1-6, 2012.
38. G. Bennett, "Probability inequalities for the sum of independent random variables," J. Amer. Statist. Assoc., vol. 57, no. 297, pp. 33-45, 1962.
39. J. Shawe-Taylor, C. K. I. Williams, N. Cristianini, and J. Kandola, "On the eigenspectrum of the Gram matrix and the generalization error of kernel-PCA," IEEE Trans. Inf. Theory, vol. 51, no. 7, pp. 2510-2522, Jul. 2005.
40. G. Blanchard, O. Bousquet, and L. Zwald, "Statistical properties of kernel principal component analysis," Mach. Learn., vol. 66, nos. 2-3, pp. 259-294, 2007.
41. D. Pollard, "A central limit theorem for k-means clustering," Ann. Probab., vol. 10, no. 4, pp. 919-926, Nov. 1982.
42. T. Linder, "On the training distortion of vector quantizers," IEEE Trans. Inf. Theory, vol. 46, no. 4, pp. 1617-1623, Jul. 2000.
43. P. L. Bartlett, T. Linder, and G. Lugosi, "The minimax distortion redundancy in empirical quantizer design," IEEE Trans. Inf. Theory, vol. 44, no. 5, pp. 1802-1813, Sep. 1998.
44. A. Antos, "Improved minimax bounds on the test and training distortion of empirically designed vector quantizers," IEEE Trans. Inf. Theory, vol. 51, no. 11, pp. 4022-4032, Nov. 2005.
45. T. Linder, G. Lugosi, and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Trans. Inf. Theory, vol. 40, no. 6, pp. 1728-1740, Nov. 1994.
46. G. Biau, L. Devroye, and G. Lugosi, "On the performance of clustering in Hilbert spaces," IEEE Trans. Inf. Theory, vol. 54, no. 2, pp. 781-790, Feb. 2008.
47. K. Sridharan, S. Shalev-Shwartz, and N. Srebro, "Fast rates for regularized objectives," in Advances in Neural Information Processing Systems, vol. 21. Red Hook, NY, USA: Curran & Associates Inc., 2008, pp. 1545-1552.
48. C. Levrard, "Fast rates for empirical vector quantization," Electron. J. Statist., vol. 7, pp. 1716-1746, 2013.
49. C. Levrard, "Nonasymptotic bounds for vector quantization in Hilbert spaces," Ann. Statist., vol. 43, no. 2, pp. 592-619, Apr. 2015.
50. R. Gribonval and K. Schnass, "Dictionary identification-Sparse matrixfactorization via l1-minimization," IEEE Trans. Inf. Theory, vol. 56, no. 7, pp. 3523-3539, Jul. 2010.
51. Q. Geng, H. Wang, and J. Wright. (Jan. 2011). "On the local correctness of -1-minimization for dictionary learning." [Online]. Available: http://arxiv.org/abs/1101.5672
52. K. Schnass. (2013). "On the identifiability of overcomplete dictionaries via the minimisation principle underlying K-SVD." [Online]. Available: http://arxiv.org/abs/1301.3375
53. W. Gautschi, "The incomplete gamma functions since Tricomi," in Tricomi's Ideas and Contemporary Applied Mathematics, Atti dei Convegni Lincei, vol. 147. Via della Lungara, Italy: Accademia Nazionale dei Lincei, 1998.
54. S. J. Szarek, "Metric entropy of homogeneous spaces and Finsler geometry of classical Lie groups," in Quantum Probability, vol. 43. Warsaw, Poland: Polish Academy Sciences, 1998, pp. 395-410.
55. A. Edelman, T. A. Arias, and S. T. Smith, "The geometry of algorithms with orthogonality constraints," SIAM J. Matrix Anal. Appl., vol. 20, no. 2, pp. 303-353, 1998.