Theoretical analysis of Bayesian matrix factorization

Journal of Machine Learning Research

Volumn 12, Issue , 2011, Pages 2583-2648

  Nakajima, Shinichi ^a   Sugiyama, Masashi ^b  

^a NIKON CORPORATION   (Japan)

^b Tokyo Institute of Technology ^*  (Japan)

Author keywords

Empirical Bayes; Matrix factorization; Model induced regularization; Non identifiable model; Positive part James Stein shrinkage; Variational Bayes

Indexed keywords

EMPIRICAL BAYES; MATRIX FACTORIZATIONS; MODEL-INDUCED REGULARIZATION; NON-IDENTIFIABLE MODEL; POSITIVE-PART JAMES-STEIN SHRINKAGE; VARIATIONAL BAYES;

FACTORIZATION; SHRINKAGE; TRACE ANALYSIS; VARIATIONAL TECHNIQUES;

MATRIX ALGEBRA;

EID: 80555154425     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (57)

1. T. W. Anderson. An Introduction to Multivariate Statistical Analysis. Wiley, New York, second edition, 1984.
2. H. Attias. Inferring parameters and structure of latent variable models by variational Bayes. In Proceedings of the Fifteenth Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI-99), pages 21-30, San Francisco, CA, 1999. Morgan Kaufmann.
3. P. Baldi and S. Brunak. Bioinformatics: The Machine Learning Approach. MIT Press, Cambridge, MA, USA, 1998.
4. P. F. Baldi and K. Hornik. Learning in linear neural networks: A survey. IEEE Transactions on Neural Networks, 6(4):837-858, 1995.
5. A. J. Baranchik. Multiple regression and estimation of the mean of a multivariate normal distribution. Technical Report 51, Department of Statistics, Stanford University, Stanford, CA, USA, 1964.
6. J. Besag. On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society B, 48:259-302, 1986.
7. C. M. Bishop. Pattern Recognition and Machine Learning. Springer, New York, NY, USA, 2006.
8. J. F. Cai, E. J. Candes, and Z. Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization, 20(4):1956-1982, 2010.
9. O. Chapelle and Z. Harchaoui. A machine learning approach to conjoint analysis. In Advances in Neural Information Processing Systems, volume 17, pages 257-264, 2005.
10. W. Chu and Z. Ghahramani. Probabilistic models for incomplete multi-dimensional arrays. In Proceedings of International Conference on Artificial Intelligence and Statistics, 2009.
11. A. Cichocki, R. Zdunek, A. H. Phan, and S. Amari. Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation. John Wiley & Sons, West Sussex, UK, 2009.
12. M. J. Daniels and R. E. Kass. Shrinkage estimators for covariance matrices. Biometrics, 57(4):1173-1184, 2001. (Pubitemid 33108110)
13. B. Efron and C. Morris. Stein's estimation rule and its competitors-an empirical Bayes approach. Journal of the American Statistical Association, 68:117-130, 1973.
14. S. Funk. Try this at home. http://sifter.org/~simon/journal/20061211. html, 2006.
15. A. Gelman. Parameterization and Bayesian modeling. Journal of the American Statistical Association, 99:537-545, 2004. (Pubitemid 38750488)
16. Y. Y. Guo and N. Pal. A sequence of improvements over the James-Stein estimator. Journal of Multivariate Analysis, 42(2):302-317, 1992.
17. K. Hayashi, J. Hirayama, and S. Ishii. Dynamic exponential family matrix factorization. In T. Theeramunkong, B. Kijsirikul, N. Cercone, and T.-B. Ho, editors, Advances in Knowledge Discovery and Data Mining, volume 5476 of Lecture Notes in Computer Science, pages 452-462, Berlin, 2009. Springer.
18. H. Hotelling. Relations between two sets of variates. Biometrika, 28(3-4):321-377, 1936.
19. A. Hyvärinen, J. Karhunen, and E. Oja. Independent Component Analysis. Wiley, New York, 2001.
20. W. James and C. Stein. Estimation with quadratic loss. In Proceedings of the 4th Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pages 361-379, Berkeley, CA., USA, 1961. University of California Press.
21. H. Jeffreys. An invariant form for the prior probability in estimation problems. In Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, volume 186, pages 453-461, 1946.
22. J. A. Konstan, B. N. Miller, D. Maltz, J. L. Herlocker, L. R. Gordon, and J. Riedl. Grouplens: Applying collaborative filtering to Usenet news. Communications of the ACM, 40(3):77-87, 1997. (Pubitemid 127441134)
23. S. Kullback and R. A. Leibler. On information and sufficiency. Annals of Mathematical Statistics, 22:79-86, 1951.
24. O. Ledoit and M. Wolf. A well-conditioned estimator for large dimensional covariance matrices. Journal of Multivariate Analysis, pages 365-411, 2004. (Pubitemid 38033261)
25. D. D. Lee and S. Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755):788-791, 1999.
26. E. L. Lehmann. Theory of Point Estimation. Wiley, New York, 1983.
27. Y. J. Lim and T. W. Teh. Variational Bayesian approach to movie rating prediction. In Proceedings of KDD Cup and Workshop, 2007.
28. D. J. C. MacKay. Bayesian interpolation. Neural Computation, 4(2):415-447, 1992.
29. D. J. C. MacKay. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, Cambridge, UK, 2003.
30. A. W. Marshall, I. Olkin, and B. C. Arnold. Inequalities: Theory of Majorization and Its Applications, Second Edition. Springer, 2009.
31. S. Nakajima and M. Sugiyama. Implicit regularization in variational Bayesian matrix factorization. In A. T. Joachims and J. Fürnkranz, editors, Proceedings of 27th International Conference on Machine Learning (ICML2010), Haifa, Israel, Jun. 21-25 2010.
32. S. Nakajima and S. Watanabe. Variational Bayes solution of linear neural networks and its generalization performance. Neural Computation, 19(4):1112-1153, 2007.
33. R. M. Neal. Bayesian Learning for Neural Networks. Springer, 1996.
34. A. Paterek. Improving regularized singular value decomposition for collaborative filtering. In Proceedings of KDD Cup and Workshop, 2007.
35. T. Raiko, A. Ilin, and J. Karhunen. Principal component analysis for large scale problems with lots of missing values. In J. Kok, J. Koronacki, R. Lopez de Mantras, S. Matwin, D. Mladenic, and A. Skowron, editors, Proceedings of the 18th European Conference on Machine Learning, volume 4701 of Lecture Notes in Computer Science, pages 691-698, Berlin, 2007. Springer-Verlag.
36. G. R. Reinsel and R. P. Velu. Multivariate Reduced-Rank Regression: Theory and Applications. Springer, New York, 1998.
37. J. D. M. Rennie and N. Srebro. Fast maximum margin matrix factorization for collaborative prediction. In Proceedings of the 22nd International Conference on Machine learning, pages 713-719, 2005.
38. J. Rissanen. Stochastic complexity and modeling. Annals of Statistics, 14(3):1080-1100, 1986.
39. R. Rosipal and N. Krämer. Overview and recent advances in partial least squares. In C. Saunders, M. Grobelnik, S. Gunn, and J. Shawe-Taylor, editors, Subspace, Latent Structure and Feature Selection Techniques, volume 3940 of Lecture Notes in Computer Science, pages 34-51, Berlin, 2006. Springer. (Pubitemid 44029878)
40. R. Salakhutdinov and A. Mnih. Probabilistic matrix factorization. In J. C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 1257-1264, Cambridge, MA, 2008. MIT Press.
41. B. Schölkopf and A. J. Smola. Learning with Kernels. MIT Press, Cambridge, MA, 2002.
42. P. Y.-S. Shao and W. E. Strawderman. Improving on the James-Stein positive-part estimator. The Annals of Statistics, 22:1517-1538, 1994.
43. N. Srebro and T. Jaakkola. Weighted low rank approximation. In T. Fawcett and N. Mishra, editors, Proceedings of the Twentieth International Conference on Machine Learning. AAAI Press, 2003.
44. N. Srebro, J. Rennie, and T. Jaakkola. Maximum margin matrix factorization. In Advances in NIPS, volume 17, 2005.
45. C. Stein. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proc. of the 3rd Berkeley Symp. on Math. Stat. and Prob., pages 197-206, 1956.
46. C. Stein. Estimation of a covariance matrix. In Rietz Lecture, 39th Annual Meeting IMS, 1975.
47. W. E. Strawderman. Proper Bayes minimax estimators of the multivariate normal mean. Annals of Mathematical Statistics, 42:385-388, 1971.
48. D. Tao, M. Song, X. Li, J. Shen, J. Sun, X. Wu, C. Faloutsos, and S. J. Maybank. Tensor approach for 3-D face modeling. IEEE Transactions on Circuits and Systems for Video Technology, 18(10):1397-1410, 2008.
49. K. Watanabe and S. Watanabe. Stochastic complexities of Gaussian mixtures in variational Bayesian approximation. Journal of Machine Learning Research, 7:625-644, 2006. (Pubitemid 43668112)
50. S. Watanabe. Algebraic analysis for nonidentifiable learning machines. Neural Computation, 13(4):899-933, 2001. (Pubitemid 33594310)
51. S. Watanabe. Algebraic Geometry and Statistical Learning. Cambridge University Press, Cambridge, UK, 2009.
52. D. Wipf and S. Nagarajan. A new view of automatric relevance determination. In J. C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 1625-1632, Cambridge, MA, 2008. MIT Press.
53. H. Wold. Estimation of principal components and related models by iterative least squares. In P. R. Krishnaiah, editor, Multivariate Analysis, pages 391-420. Academic Press, New York, NY, USA, 1966.
54. K. J. Worsley, J-B. Poline, K. J. Friston, and A. C. Evanss. Characterizing the response of PET and fMRI data using multivariate linear models. NeuroImage, 6(4):305-319, 1997. (Pubitemid 28018719)
55. K. Yamazaki and S. Watanabe. Singularities in mixture models and upper bounds of stochastic complexity. Neural Networks, 16(7):1029-1038, 2003. (Pubitemid 38147281)
56. K. Yu, V. Tresp, and A. Schwaighofer. Learning Gaussian processes from multiple tasks. In Proceedings of the Twenty-Second International Conference on Machine learning, pages 1012-1019, 2005.
57. S. Yu, J. Bi, and J. Ye. Probabilistic interpretations and extensions for a family of 2D PCA-style algorithms. In KDD Workshop on Data Mining using Matrices and Tensors, 2008.