Semiparametric mean field variational bayes: General principles and numerical issues

Journal of Machine Learning Research

Volumn 17, Issue , 2016, Pages

  Rohde, David ^a   Wand, Matt P ^a  

^a UNIVERSITY OF TECHNOLOGY SYDNEY   (Australia)

Author keywords

Bayesian Computing; Factor Graph; Fixed form Variational Bayes; Fixedpoint Iteration; Non conjugate Variational Message Passing; Nonlinear Conjugate Gradient Method

Indexed keywords

ITERATIVE METHODS; MESSAGE PASSING; NONLINEAR PROGRAMMING;

BAYESIAN; FACTOR GRAPHS; FIXED-POINT ITERATIONS; NONLINEAR CONJUGATE GRADIENT METHOD; VARIATIONAL BAYES;

CONJUGATE GRADIENT METHOD;

EID: 84995444661     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (46)

1. A.S. Ackleh, E.J. Allen, R.B. Hearfott, and P. Seshaiyer. Classical and Modern Numerical Analysis: Theory, Methods and Practice. Chapman&Hall, CRC Press, Boca Raton, Florida, 2010.
2. S. Amari. Natural gradient works efficiently in learning. Neural Computation, 10:251-276, 1998.
3. C. Archambeau, D. Cornford, M. Opper, and J. Shawe-Taylor. Gaussian process approximations of stochastic differential equations. Journal of Machine Learning Research: Workshop and Conference Proceedings, 1:1-16, 2007.
4. A. Azzalini and A. Dalla Valle. The Multivariate Skew-Normal distribution. Biometrika, 83:715-726, 1996.
5. D. Barber and C.M. Bishop. Ensemble learning for multi-layer networks. In M.I. Jordan, K.J. Kearns, and S.A. Solla, editors, Advances in Neural Information Processing Systems 10, pages 395-401, 1997.
6. M.J. Beal and Z. Ghahramani. Variational Bayesian learning of directed graphical models with hidden variables. Bayesian Analysis, 4:793-832, 2006.
7. C.M. Bishop. Pattern Recognition and Machine Learning. Springer, New York, 2006.
8. L. Bottou. Stochastic learning. In O. Bousquet, U. von Luxburg, and G. Rätsch, editors, Advanced Lectures on Machine Learning, pages 146-168, 2004.
9. S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, Cambridge, UK, 2004.
10. E. Challis and D. Barber. Gaussian Kullback-Leibler approximate inference. Journal of Machine Learning Research, 14:2239-2286, 2013.
11. Y.H. Dai and Y. Yuan. A nonlinear conjugate gradient method with a strong global convergence property. SIAM Journal on Optimization, 10:177-182, 1999.
12. S.J. Gershman, M.D. Hoffman, and D.M. Blei. Nonparametric variational inference. In Proceedings of the Twenty Ninth International Conference on Machine Learning, pages 633-670, 2012.
13. G.H. Givens and J.A. Hoeting. Computational Statistics. Wiley, Hoboken, New Jersey, 2005.
14. M.R. Hestenes and E. Stiefel. Methods of conjugate gradients for solving linear systems. Journal of Research of the National Bureau of Standards, 49:409-436, 1952.
15. G.E. Hinton and D. van Camp. Keeping neural networks simple by minimizing description length of the weights. In L. Pitt, editor, Proceedings of the Sixth Annual Conference on Computational Learning Theory, pages 5-13, 1993.
16. M.D. Hoffman, D.M. Blei, C. Wang, and J. Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14:1303-1347, 2013.
17. A. Honkela and H. Valpola. Unsupervised variational Bayesian learning of nonlinear models. Advances in Neural Information Processing Systems, 17:593-600, 2005.
18. A. Honkela, H. Valpola, A. Ilin, and J. Karhunen. Blind separation of nonlinear mixtures by variational Bayesian learning. Digital Signal Processing, 17:914-934, 2007.
19. A. Honkela, M. Tornio, T. Raiko, and J. Karhunen. Natural conjugate gradient in variational inference. In M. Ishikawa, K. Doya, H. Miyamoto, and T. Yamakawa, editors, Proceedings of the Fourteenth International Conference on Neural Information Processing, pages 305-314, 2008.
20. A. Honkela, T. Raiko, M. Kuusela, M. Tornio, and J. Karhunen. Approximate Riemannian conjugate gradient learning for fixed-form variational Bayes. Journal of Machine Learning Research, 11:3235-3268, 2010.
21. D.A. Knowles and T.P. Minka. Non-conjugate message passing for multinomial and binary regression. In J. Shawe-Taylor, R.S. Zamel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 24, 2011.
22. H. Lappalainen and A. Honkela. Bayesian nonlinear independent component analysis by multilayer perceptrons. In M. Girolami, editor, Advances in Independent Component Analysis, pages 93-121, 2000.
23. D.G. Luenberger and Y. Ye. Linear and Nonlinear Programming, Third Edition. Springer, New York, 2008.
24. J. Luts and M.P. Wand. Variational inference for count response semiparametric regression. Bayesian Analysis, 10:991-1023, 2015.
25. J.R. Magnus and H. Neudecker. Matrix Differential Calculus with Applications in Statistics and Econometrics, Revised Edition. Wiley, Chichester, UK, 1999.
26. M. Menictas and M.P. Wand. Variational inference for heteroscedastic semiparametric regression. Australian and New Zealand Journal of Statistics, 57:119-138, 2015.
27. M.K. Murray and J.W. Rice. Differential Geometry and Statistics. Chapman&Hall, London, 1993.
28. J.A. Nelder and R. Mead. A simplex method for function minimization. Computer Journal, 7:308-313, 1965.
29. S.E. Neville, J.T. Ormerod, and M.P. Wand. Mean field variational Bayes for continuous sparse signal shrinkage: pitfalls and remedies. Electronic Journal of Statistics, 8:1113-1151, 2014.
30. H. Nickisch and C.E. Rasmussen. Approximations for binary Gaussian process classification. Journal of Machine Learning Research, 9:2035-2078, 2008.
31. M. Opper and C. Archambeau. The variational Gaussian approximation revisited. Neural Computation, 21:786-792, 2009.
32. J.T. Ormerod and M.P. Wand. Explaining variational approximations. The American Statistician, 64:140-153, 2010.
33. J.M. Ortega. Numerical Analysis: A Second Course. SIAM, Philadelphia, 1990.
34. T. Pham, J.T. Ormerod, and M.P. Wand. Mean field variational Bayesian inference for nonparametric regression with measurement error. Computational Statistics and Data Analysis, 68:375-387, 2013.
35. W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flanney. Numerical Recipes: The Art of Scientific Computing, Third Edition. Cambridge University Press, New York, 2007.
36. T. Raiko, H. Valpola, M. Harva, and J. Karhunen. Building blocks for variational Bayesian learning of latent variable models. Journal of Machine Learning Research, 8:155-201, 2007.
37. R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2016. URL http: //www.R-project.org/ .
38. T. Salimans and D.A. Knowles. Fixed-form variational posterior approximation through stochastic linear regression. Bayesian Analysis, 8:837-882, 2013.
39. M. Sato. Online model selection based on the variational Bayes. Neural Computation, 13: 1649-1681, 2001.
40. L.S.L. Tan and D.J. Nott. Variational inference for generalized linear mixed models using partially noncentered parametrizations. Statistical Science, 28:168-188, 2013.
41. M.J. Wainwright and M.I. Jordan. Graphical models, exponential families and variational inference. Foundations and Trends in Machine Learning, 1:1-305, 2008.
42. M.P. Wand. Vector differential calculus in statistics. The American Statistician, 56:55-62, 2002.
43. M.P. Wand. Fully simplified Multivariate Normal updates in non-conjugate variational message passing. Journal of Machine Learning Research, 15:1351-1369, 2014.
44. M.P. Wand, J.T. Ormerod, S.A. Padoan, and R. Frühwirth. Mean field variational Bayes for elaborate distributions. Bayesian Analysis, 6:847-900, 2011.
45. B. Wang and D.M. Titterington. Inadequacy of interval estimates corresponding to varia-tional Bayes approximations. In Z. Ghahramani and R. Cowell, editors, Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, pages 373-380, 2005.
46. O. Zobay. Variational Bayesian inference with Gaussian-mixture approximations. Electronic Journal of Statistics, 8:355-389, 2014.