Phase Transitions and Sample Complexity in Bayes-Optimal Matrix Factorization

IEEE Transactions on Information Theory

Volumn 62, Issue 7, 2016, Pages 4228-4265

  Kabashima, Yoshiyuki ^a   Krzakala, Florent ^b   Mezard, Marc ^c   Sakata, Ayaka ^d,e   Zdeborova, Lenka ^f  

^a TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

^b SORBONNE UNIVERSITÉ   (France)

^c ECOLE NORMALE SUPÉRIEURE   (France)

^d INSTITUTE OF STATISTICAL MATHEMATICS   (Japan)

^e GRADUATE UNIVERSITY FOR ADVANCED STUDIES   (Japan)

^f CNRS   (France)

Author keywords

computational barriers; dictionary learning; message passing algorithms; phase transitions; probabilistic matrix factorization; sparse coding; statistical and computational tradeoff; Statistical inference; statistical physics

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; BINARY CODES; BLIND SOURCE SEPARATION; COMPUTATIONAL EFFICIENCY; ERRORS; FACE RECOGNITION; FACTORIZATION; INFERENCE ENGINES; LEARNING ALGORITHMS; LEARNING SYSTEMS; MATRIX ALGEBRA; MEAN SQUARE ERROR; MESSAGE PASSING; PHASE TRANSITIONS; STATISTICAL MECHANICS; STATISTICAL METHODS;

COMPUTATIONAL BARRIERS; DICTIONARY LEARNING; MESSAGE PASSING ALGORITHM; PROBABILISTIC MATRIX FACTORIZATIONS; SPARSE CODING; STATISTICAL AND COMPUTATIONAL TRADEOFF; STATISTICAL INFERENCE; STATISTICAL PHYSICS;

PRINCIPAL COMPONENT ANALYSIS;

EID: 84976449740     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2016.2556702     Document Type: Conference Paper

References (74)

1. Y. Bengio, A. Courville, P. Vincent, "Representation learning: A review and new perspectives," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1798-1828, Aug. 2013.
2. Y. LeCun, Y. Bengio, G. Hinton, "Deep learning," Nature, vol. 521, pp. 436-444, May 2015.
3. B. A. Olshausen and D. J. Field, "Emergence of simple-cell receptive field properties by learning a sparse code for natural images," Nature, vol. 381, no. 6583, pp. 607-609, 1996.
4. B. A. Olshausen and D. J. Field, "Sparse coding with an overcomplete basis set: A strategy employed by v1?" Vis. Res., vol. 37, no. 23, pp. 3311-3325, 1997.
5. K. Kreutz-Delgado, J. F. Murray, B. D. Rao, K. Engan, T.-W. Lee, T. J. Sejnowski, "Dictionary learning algorithms for sparse representation," Neural Comput., vol. 15, no. 2, pp. 349-396, 2003.
6. H. Zou, T. Hastie, R. Tibshirani, "Sparse principal component analysis," J. Comput. Graph. Statist., vol. 15, no. 2, pp. 265-286, Jun. 2006.
7. A. Belouchrani, K. Abed-Meraim, J.-F. Cardoso, E. Moulines, "A blind source separation technique using second-order statistics," IEEE Trans. Signal Process., vol. 45, no. 2, pp. 434-444, Feb. 1997.
8. E. J. Candès and B. Recht, "Exact matrix completion via convex optimization," Found. Comput. Math., vol. 9, no. 6, pp. 717-772, 2009.
9. E. J. Candès and T. Tao, "The power of convex relaxation: Near-optimal matrix completion," IEEE Trans. Inf. Theory, vol. 56, no. 5, pp. 2053-2080, May 2010.
10. E. J. Candès, X. Li, Y. Ma, J. Wright, "Robust principal component analysis?" J. ACM, vol. 58, no. 3, p. 11, May 2011.
11. D. L. Donoho, A. Maleki, A. Montanari, "Message-passing algorithms for compressed sensing," Proc. Nat. Acad. Sci. USA, vol. 106, no. 45, pp. 18914-18919, 2009.
12. M. Mézard, G. Parisi, M. Virasoro, Spin Glass Theory and Beyond: An Introduction to The Replica Method and Its Applications, vol. 9. Singapore: World Scientific, 1987.
13. M. Mézard and A. Montanari, Information, Physics, Computation. Oxford, U.K.: Oxford Univ. Press, 2009.
14. A. Sakata and Y. Kabashima, "Sample complexity of Bayesian optimal dictionary learning," in Proc. IEEE Int. Symp. Inf. Theory (ISIT), Jul. 2013, pp. 669-673.
15. F. Krzakala, M. Meézard, L. Zdeborová, "Phase diagram and approximate message passing for blind calibration and dictionary learning," in Proc. IEEE Int. Symp. Inf. Theory (ISIT), Jul. 2013, pp. 659-663.
16. P. Schniter, J. T. Parker, V. Cevher, "Bilinear generalized approximate message passing (BiG-AMP) for matrix recovery problem," in Proc. Workshop Inf. Theory Appl. (ITA), San Diego, CA, USA, 2012.
17. J. T. Parker, P. Schniter, V. Cevher, "Bilinear generalized approximate message passing-Part I: Derivation," IEEE Trans. Inf. Theory, vol. 62, no. 22, pp. 5839-5853, Nov. 2014.
18. J. T. Parker, P. Schniter, V. Cevher, "Bilinear generalized approximate message passing-Part II: Applications," IEEE Trans. Inf. Theory, vol. 62, no. 22, pp. 5854-5867, Nov. 2014.
19. M. Elad, "Sparse and redundant representation modeling-What next?" IEEE Signal Process. Lett., vol. 19, no. 12, pp. 922-928, Dec. 2012.
20. M. Mézard, G. Parisi, R. Zecchina, "Analytic and algorithmic solution of random satisfiability problems," Science, vol. 297, no. 5582, pp. 812-815, 2002.
21. J. Ding, A. Sly, N. Sun. (2014). "Proof of the satisfiability conjecture for large k." [Online]. Available: https://arxiv.org/abs/1411.0650
22. T. Tanaka, "A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors," IEEE Trans. Inf. Theory, vol. 48, no. 11, pp. 2888-2910, Nov. 2002.
23. D. Guo and S. Verdú, "Randomly spread CDMA: Asymptotics via statistical physics," IEEE Trans. Inf. Theory, vol. 51, no. 6, pp. 1983-2010, Jun. 2005.
24. A. Montanari and D. Tse, "Analysis of belief propagation for non-linear problems: The example of CDMA (or: How to prove Tanaka's formula)," in Proc. IEEE Inf. Theory Workshop (ITW), Punta del Este, Uruguay, Mar. 2006, pp. 160-164.
25. M. Bayati and A. Montanari, "The dynamics of message passing on dense graphs, with applications to compressed sensing," IEEE Trans. Inf. Theory, vol. 57, no. 2, pp. 764-785, Feb. 2011.
26. M. A. Davenport, Y. Plan, E. van den Berg, M. Wootters, "1-bit matrix completion," Inf. Inference, vol. 3, no. 3, pp. 189-223, 2014.
27. M. Davenport and J. Romberg, "An overview of low-rank matrix recovery from incomplete observations," to be published.
28. T. Lesieur, F. Krzakala, L. Zdeborová. (2015). "MMSE of probabilistic low-rank matrix estimation: Universality with respect to the output channel." [Online]. Available: http://arxiv.org/abs/1507.03857
29. K. G. Jöreskog, "A general approach to confirmatory maximum likelihood factor analysis," Psychometrika, vol. 34, no. 2, pp. 183-202, 1969.
30. M. S. Lewicki and T. J. Sejnowski, "Learning overcomplete representations," Neural Comput., vol. 12, no. 2, pp. 337-365, 2000.
31. K. Engan, S. O. Aase, J. H. Husoy, "Method of optimal directions for frame design," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Mar. 1999, pp. 2443-2446.
32. M. Aharon, M. Elad, A. M. Bruckstein, "K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation," IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311-4322, Nov. 2006.
33. M. Aharon, M. Elad, A. M. Bruckstein, "On the uniqueness of overcomplete dictionaries, a practical way to retrieve them," Linear Algebra Appl., vol. 416, no. 1, pp. 48-67, 2006.
34. D. Vainsencher, S. Mannor, A. M. Bruckstein, "The sample complexity of dictionary learning," J. Mach. Learn. Res., vol. 12, pp. 3259-3281, Feb. 2011.
35. R. Jenatton, R. Gribonval, F. Bach. (2012). "Local stability and robustness of sparse dictionary learning in the presence of noise." [Online]. Available: http://arxiv.org/abs/1210.0685
36. D. A. Spielman, H. Wang, J. Wright, "Exact recovery of sparsely-used dictionaries," in Proc. 23rd Int. Joint Conf. Artif. Intell., 2013, pp. 3087-3090.
37. S. Arora, R. Ge, A. Moitra. (2013). "New algorithms for learning incoherent and overcomplete dictionaries." [Online]. Available: http://arxiv.org/abs/1308.6273
38. A. Agarwal, A. Anandkumar, P. Netrapalli. (2013). "A clustering approach to learn sparsely-used overcomplete dictionaries." [Online]. Available: http://arxiv.org/abs/1309.1952
39. R. Gribonval, R. Jenatton, F. Bach, M. Kleinsteuber, M. Seibert. (2013). "Sample complexity of dictionary learning and other matrix factorizations." [Online]. Available: http://arxiv.org/abs/1312.3790
40. T.-W. Lee, M. S. Lewicki, M. Girolami, T. J. Sejnowski, "Blind source separation of more sources than mixtures using overcomplete representations," IEEE Signal Process. Lett., vol. 6, no. 4, pp. 87-90, Apr. 1999.
41. M. Zibulevsky and B. Pearlmutter, "Blind source separation by sparse decomposition in a signal dictionary," Neural Comput., vol. 13, no. 4, pp. 863-882, Apr. 2001.
42. P. Bofill and M. Zibulevsky, "Underdetermined blind source separation using sparse representations," Signal Process., vol. 81, no. 11, pp. 2353-2362, 2001.
43. P. Georgiev, F. Theis, A. Cichocki, "Sparse component analysis and blind source separation of underdetermined mixtures," IEEE Trans. Neural Netw., vol. 16, no. 4, pp. 992-996, Jul. 2005.
44. A. d'Aspremont, L. El Ghaoui, M. I. Jordan, G. R. Lanckriet, "A direct formulation for sparse PCA using semidefinite programming," SIAM Rev., vol. 49, no. 3, pp. 434-448, 2007.
45. R. Gribonval et al., "A survey of sparse component analysis for blind source separation: Principles, perspectives, new challenges," in Proc. 14th Eur. Symp. Artif. Neural Netw. (ESANN), 2006, pp. 323-330.
46. E. J. Candès and Y. Plan, "Matrix completion with noise," Proc. IEEE, vol. 98, no. 6, pp. 925-936, Jun. 2010.
47. R. H. Keshavan, A. Montanari, S. Oh, "Matrix completion from a few entries," IEEE Trans. Inf. Theory, vol. 56, no. 6, pp. 2980-2998, Jun. 2010.
48. V. Chandrasekaran, S. Sanghavi, P. A. Parrilo, A. S. Willsky, "Sparse and low-rank matrix decompositions," in Proc. 47th Annu. Allerton Conf. Commun., Control, Comput. (Allerton), 2009, pp. 962-967.
49. X. Yuan and J. Yang, "Sparse and low-rank matrix decomposition via alternating direction methods," to be published.
50. J. Wright, A. Ganesh, S. Rao, Y. Peng, Y. Ma, "Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization," in Proc. Adv. Neural Inf. Process. Syst., 2009, pp. 2080-2088.
51. S. Rangan and A. K. Fletcher, "Iterative estimation of constrained rank-one matrices in noise," in Proc. IEEE Int. Symp. Inf. Theory (ISIT), Jul. 2012, pp. 1246-1250.
52. Y. Deshpande and A. Montanari. (2014). "Information-theoretically optimal sparse PCA." [Online]. Available: http://arxiv.org/abs/1402.2238
53. M. Opper and D. Haussler, "Calculation of the learning curve of bayes optimal classification algorithm for learning a perceptron with noise," in Proc. 4th Annu. Workshop Comput. Learn. Theory, 1991, pp. 75-87.
54. Y. Iba, "The Nishimori line and Bayesian statistics," J. Phys. A, Math. General, vol. 32, no. 21, p. 3875, 1999.
55. H. Nishimori, Statistical Physics of Spin Glasses and Information Processing: An Introduction. Oxford, U.K.: Oxford Univ. Press, 2001.
56. D. L. Donoho, A. Javanmard, A. Montanari, "Informationtheoretically optimal compressed sensing via spatial coupling and approximate message passing," in Proc. IEEE Int. Symp. Inf. Theory (ISIT), Nov. 2012, pp. 7434-7464.
57. M. Mézard, G. Parisi, M. A. Virasoro, "SK model: The replica solution without replicas," Europhys. Lett., vol. 1, no. 2, pp. 77-82, 1986.
58. T. Richardson and R. Urbanke, Modern Coding Theory. Cambridge, U.K.: Cambridge Univ. Press, 2008.
59. D. J. Aldous, "The ? (2) limit in the random assignment problem," Random Struct. Algorithms, vol. 18, no. 4, pp. 381-418, 2001.
60. A.,rea, "Estimating random variables from random sparse observations," Eur. Trans. Telecommun., vol. 19, no. 4, pp. 385-403, 2008.
61. Y. Kabashima, T. Wadayama, T. Tanaka, "A typical reconstruction limit of compressed sensing based on L p-norm minimization," J. Statist. Mech., Theory Experim., vol. 2009, p. L09003, Sep. 2009.
62. A. P. Dempster, N. M. Laird, D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," J. Roy. Statist. Soc. B (Methodological), vol. 38, no. 1, pp. 1-38, 1977.
63. F. Krzakala, M. Mézard, F. Sausset, Y. Sun, L. Zdeborová, "Probabilistic reconstruction in compressed sensing: Algorithms, phase diagrams, threshold achieving matrices," J. Statist. Mech., Theory Experim., vol. 2012, p. P08009, Aug. 2012.
64. J. Vila and P. Schniter, "Expectation-maximization Bernoulli-Gaussian approximate message passing," in Proc. 45th Asilomar Conf. Signals, Syst., Comput., Pacific Grove, CA, USA, 2011, pp. 799-803.
65. D. L. Donoho, A. Maleki, A. Montanari, "Message passing algorithms for compressed sensing: I. Motivation and construction," in Proc. IEEE Inf. Theory Workshop (ITW), Jan. 2010, pp. 1-5.
66. S. Rangan, "Generalized approximate message passing for estimation with random linear mixing," in Proc. IEEE Int. Symp. Inf. Theory (ISIT), Jul./Aug. 2011, pp. 2168-2172.
67. F. R. Kschischang, B. J. Frey, H.-A. Loeliger, "Factor graphs and the sum-product algorithm," IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 498-519, Feb. 2001.
68. D. J. Thouless, P. W.,erson, R. G. Palmer, "Solution of 'solvable model of a spin-glass,"' Philos. Mag., vol. 35, no. 3, pp. 593-601, 1977.
69. S. Rangan, "Estimation with random linear mixing, belief propagation and compressed sensing," in Proc. 44th Annu. Conf. Inf. Sci. Syst. (CISS), 2010, pp. 1-6.
70. J. S. Yedidia, W. T. Freeman, Y. Weiss, "Understanding belief propagation and its generalizations," in Exploring Artificial Intelligence in the New Millennium. San Francisco, CA, USA: Morgan Kaufmann, 2003, pp. 236-239.
71. F. Krzakala, A. Manoel, E. W. Tramel, L. Zdeborová, "Variational free energies for compressed sensing," in Proc. IEEE Int. Symp. Inf. Theory (ISIT), Jun./Jul. 2014, pp. 1499-1503.
72. J. Vila, P. Schniter, S. Rangan, F. Krzakala, L. Zdeborová, "Adaptive damping and mean removal for the generalized approximate message passing algorithm," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Apr. 2015, pp. 2021-2025.
73. S. Rangan, P. Schniter, E. Riegler, A. Fletcher, V. Cevher, "Fixed points of generalized approximate message passing with arbitrary matrices," in Proc. IEEE Int. Symp. Inf. Theory (ISIT), Jul. 2013, pp. 664-668.
74. F. Caltagirone, F. Krzakala, L. Zdeborová, "On convergence of approximate message passing," in Proc. IEEE Int. Symp. Inf. Theory (ISIT), Jun./Jul. 2014, pp. 1812-1816.