Global analytic solution of fully-observed variational bayesian matrix factorization

Journal of Machine Learning Research

Volumn 14, Issue 1, 2013, Pages 1-37

  Nakajima, Shinichi ^a   Sugiyama, Masashi ^b   Babacan, S Derin ^c   Tomioka, Ryota ^d  

^a NIKON CORPORATION   (Japan)

^b TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

^c UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^d UNIVERSITY OF TOKYO   (Japan)

Author keywords

Empirical Bayes; Matrix factorization; Model induced regularization; Probabilistic PCA; Variational Bayes

Indexed keywords

EMPIRICAL BAYES; MATRIX FACTORIZATIONS; MODEL-INDUCED REGULARIZATION; PROBABILISTIC PCA; VARIATIONAL BAYES;

BAYESIAN NETWORKS; PRINCIPAL COMPONENT ANALYSIS;

APPROXIMATION ALGORITHMS;

EID: 84873435460     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (48)

1. T. W. Anderson. An Introduction to Multivariate Statistical Analysis. Wiley, New York, second edition, 1984.
2. A. Asuncion and D.J. Newman. UCI machine learning repository, 2007. URL http://www.ics.uci.edu/mlearn/MLRepository.html.
3. H. Attias. Inferring parameters and structure of latent variable models by variational Bayes. In Proceedings of the Fifteenth Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI-99), pages 21-30, San Francisco, CA, 1999. Morgan Kaufmann.
4. S. D. Babacan,M. Luessi, R.Molina, and A. K. Katsaggelos. Sparse Bayesian methods for low-rank matrix estimation. IEEE Trans. on Signal Processing, 60(8):3964-3977, 2012.
5. P. F. Baldi and K. Hornik. Learning in linear neural networks: A survey. IEEE Transactions on Neural Networks, 6(4):837-858, 1995.
6. J. Besag. On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society B, 48: 259-302, 1986.
7. C. M. Bishop. Variational principal components. In Proc. of ICANN, volume 1, pages 514-509, 1999.
8. C. M. Bishop. Pattern Recognition and Machine Learning. Springer, New York, NY, USA, 2006.
9. J. F. Cai, E. J. Candes, and Z. Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization, 20(4):1956-1982, 2010.
10. E. J. Candes, X. Li, Y. Ma, and J. Wright. Robust principal component analysis CoRR, abs/0912.3599, 2009.
11. J. D. Carroll and J. J. Chang. Analysis of individual differences in multidimensional scaling via an N-way generalization of 'Eckart-Young' decomposition. Psychometrika, 35:283-319, 1970.
12. S. Funk. Try this at home. http://sifter.org/simon/journal/20061211.html, 2006. D. R. Hardoon, S. R. Szedmak, and J. R. Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16(12):2639-2664, 2004.
13. R. A. Harshman. Foundations of the parafac procedure: Models and conditions for an "explanatory" multimodal factor analysis. UCLA Working Papers in Phonetics, 16:1-84, 1970.
14. M. Hazewinkel, editor. Encyclopaedia of Mathematics. Springer, 2002.
15. H. Hotelling. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24:417-441, 1933.
16. H. Hotelling. Relations between two sets of variates. Biometrika, 28(3-4):321-377, 1936.
17. Hyvarinen, J. Karhunen, and E. Oja. Independent Component Analysis. Wiley, New York, 2001.
18. A. Ilin and T. Raiko. Practical approaches to principal component analysis in the presence of missing values. JMLR, 11:1957-2000, 2010.
19. W. James and C. Stein. Estimation with quadratic loss. In Proceedings of the 4th Berkeley Sympo- sium on Mathematical Statistics and Probability, volume 1, pages 361-379, Berkeley, CA., USA, 1961. University of California Press.
20. H. Jeffreys. An invariant form for the prior probability in estimation problems. In Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, volume 186, pages 453-461, 1946.
21. S. Ji and J. Ye. An accelerated gradient method for trace norm minimization. In Proceedings of International Conference on Machine Learning, pages 457-464, 2009.
22. T. G. Kolda and B. W. Bader. Tensor decompositions and applications. SIAM Review, 51(3):455- 500, 2009.
23. J. A. Konstan, B. N. Miller, D. Maltz, J. L. Herlocker, L. R. Gordon, and J. Riedl. Grouplens: Applying collaborative filtering to Usenet news. Communications of the ACM, 40(3):77-87, 1997. (Pubitemid 127441134)
24. Y. J. Lim and T.W. Teh. Variational Bayesian approach to movie rating prediction. In Proceedings of KDD Cup and Workshop, 2007.
25. D. J. C. Mackay. Local minima, symmetry-breaking, and model pruning in variational free energy minimization. Available from http://www.inference.phy. cam.ac.uk/mackay/minima.pdf. 2001.
26. A. W. Marshall, I. Olkin, and B. C. Arnold. Inequalities: Theory of Majorization and Its Applica- tions, Second Edition. Springer, 2009.
27. R. Mazumder, T. Hastie, and R. Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. Journal of Machine Learning Research, 11:2287-2322, 2010.
28. S. Nakajima. Variational Bayesian algorithm for relational tensor factorization. Under Preparation, 2012. S. Nakajima and M. Sugiyama. Theoretical analysis of Bayesian matrix factorization. Journal of Machine Learning Research, 12:2579-2644, 2011.
29. S. Nakajima, M. Sugiyama, and R. Tomioka. Global analytic solution for variational Bayesian matrix factorization. In Advances in Neural Information Processing Systems 23, pages 1759- 1767, 2010.
30. S. Nakajima, M. Sugiyama, and S. D. Babacan. Global solution of fully-observed variational Bayesian matrix factorization is column-wise independent. In Advances in Neural Information Processing Systems 24, 2011.
31. S. Nakajima, M. Sugiyama, and S. D. Babacan. Sparse additive matrix factorization for robust PCA and its generalization. In Proceedings of Fourth Asian Conference on Machine Learning, 2012a.
32. S. Nakajima, R. Tomioka, M. Sugiyama, and S. D. Babacan. Perfect dimensionality recovery by variational Bayesian PCA. In Advances in Neural Information Processing Systems 25, 2012b.
33. A. Paterek. Improving regularized singular value decomposition for collaborative filtering. In Proceedings of KDD Cup and Workshop, 2007.
34. G. R. Reinsel and R. P. Velu. Multivariate Reduced-Rank Regression: Theory and Applications. Springer, New York, 1998.
35. J. D. M. Rennie and N. Srebro. Fast maximum margin matrix factorization for collaborative prediction. In Proceedings of the 22nd International Conference on Machine learning, pages 713-719, 2005.
36. R. Rosipal and N. Kramer. Overview and recent advances in partial least squares. In C. Saunders,
37. M. Grobelnik, S. Gunn, and J. Shawe-Taylor, editors, Subspace, Latent Structure and Feature
38. Selection Techniques, volume 3940 of Lecture Notes in Computer Science, pages 34-51, Berlin, 2006. Springer.
39. A. Ruhe. Perturbation bounds for means of eigenvalues and invariant subspaces. BIT Numerical Mathematics, 10:343-354, 1970.
40. R. Salakhutdinov and A.Mnih. Probabilistic matrix factorization. In J. C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 1257- 1264, Cambridge, MA, 2008. MIT Press.
41. M. Seeger and G. Bouchard. Fast variational Bayesian inference for non-conjugate matrix factorization models. In Proc. of AISTATS, La Palma, Spain, 2012.
42. N. Srebro, J. Rennie, and T. Jaakkola. Maximum margin matrix factorization. In Advances in Neural Information Processing Systems 17, 2005.
43. G. W. Stewart. On the early history of the singular value decomposition. SIAM Review, 35(4): 551-556, 1993.
44. M. E. Tipping and C. M. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society, 61:611-622, 1999.
45. R. Tomioka, T. Suzuki, M. Sugiyama, and H. Kashima. An efficient and general augmented Lagrangian algorithm for learning low-rank matrices. In Proceedings of International Conference on Machine Learning, 2010.
46. L. R. Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31:279-311, 1996.
47. S. Watanabe. Algebraic Geometry and Statistical Learning. Cambridge University Press, Cambridge, UK, 2009.
48. K. J. Worsley, J-B. Poline, K. J. Friston, and A. C. Evanss. Characterizing the response of PET and fMRI data using multivariate linear models. NeuroImage, 6(4):305-319, 1997. (Pubitemid 28018719)