Automatic Differentiation Variational Inference

Journal of Machine Learning Research

Volumn 18, Issue , 2017, Pages 1-45

  Kucukelbir, Alp ^a   Blei, David M ^a   Gelman, Andrew ^b   Ranganath, Rajesh ^c   Tran, Dustin ^c  

^a Department of Earth and Environmental Sciences   (United States)

^b PRINCETON UNIVERSITY   (United States)

^c NONE   (United States)

Author keywords

Approximate inference; Bayesian inference; Probabilistic programming

Indexed keywords

BAYESIAN NETWORKS; ITERATIVE METHODS;

APPROXIMATE INFERENCE; AUTOMATIC DIFFERENTIATIONS; BAYESIAN INFERENCE; PROBABILISTIC MODELING; PROBABILISTIC MODELS; PROBABILISTIC PROGRAMMING; SIMPLE MODELING; VARIATIONAL INFERENCE;

INFERENCE ENGINES;

EID: 85016397096     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (78)

1. Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zhang. Tensorflow: a system for large-scale machine learning. arXiv:1605.08695, 2016.
2. Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2):251-276, 1998.
3. Charles E Antoniak. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. The Annals of Statistics, 2(6):1152-1174, 1974.
4. David Barber. Bayesian Reasoning and Machine Learning. Cambridge University Press, 2012.
5. Atilim Gunes Baydin, Barak A Pearlmutter, and Alexey Andreyevich Radul. Automatic differentiation in machine learning: a survey. arXiv:1502.05767, 2015.
6. Jose M Bernardo and Adrian FM Smith. Bayesian Theory. John Wiley & Sons, 2009.
7. Christopher M Bishop. Pattern Recognition and Machine Learning. Springer New York, 2006.
8. Christopher M Bishop, David Spiegelhalter, and John Winn. VIBES: a variational inference engine for Bayesian networks. In Neural Information Processing Systems, 2002.
9. David M Blei. Build, compute, critique, repeat: data analysis with latent variable models. Annual Review of Statistics and Its Application, 1:203-232, 2014.
10. David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: a review for statisticians. arXiv:1601.00670, 2016.
11. Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In Proceedings of The 32nd International Conference on Machine Learning, pages 1613-1622, 2015.
12. Leon Bottou. Stochastic gradient descent tricks. In Neural Networks: Tricks of the Trade, pages 421-436. Springer, 2012.
13. JohnCanny. GaP: a factormodel fordiscrete data. InSIGIRConference on ResearchandDevelopment in Information Retrieval, 2004.
14. Bob Carpenter, Matthew D Hoffman, Marcus Brubaker, Daniel Lee, Peter Li, and Michael Betancourt. The Stan math library: reverse-mode automatic differentiation in C++. arXiv:1509.07164, 2015.
15. Ali Taylan Cemgil. Bayesian inference for nonnegative matrix factorisation models. Computational Intelligence and Neuroscience, 2009.
16. Edward Challis and David Barber. Affine independent variational inference. In Neural Information Processing Systems, 2012.
17. Edward Challis and David Barber. Gaussian Kullback-Leibler approximate inference. The Journal of Machine Learning Research, 14(1):2239-2286, 2013.
18. Erhan Cinlar. Probability and Stochastics. Springer, 2011.
19. SamanthaR Cook, Andrew Gelman, and Donald B Rubin. Validation of software for Bayesian models using posterior quantiles. Journal of Computational and Graphical Statistics, pages 675-692, 2006.
20. Persi Diaconis and Donald Ylvisaker. Conjugate priors for exponential families. The Annals of Statistics, 7(2):269-281, 1979.
21. Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components estimation. arXiv:1410.8516, 2014.
22. Jan Drugowitsch. Variational Bayesian inference for linear and logistic regression. arXiv:1310.5438, 2013.
23. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121-2159, 2011.
24. Kai Fan, Ziteng Wang, Jeff Beck, James Kwok, and Katherine A Heller. Fast second order stochastic backpropagation for variational inference. In Neural Information Processing Systems, 2015.
25. Andrew Gelman and Jennifer Hill. Data Analysis using Regression and Multilevel/Hierarchical Models. Cambridge University Press, 2006.
26. Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian Data Analysis. CRC Press, 2013.
27. Noah D Goodman and Andreas Stuhlmiiller. The Design and Implementation of Probabilistic Programming Languages, 2014.
28. Noah D Goodman, Vikash K Mansinghka, Daniel Roy, Keith Bonawitz, and Joshua B Tenenbaum. Church: a language for generative models. In Uncertainty in Artificial Intelligence, 2008.
29. Wolfgang Hardle and Leopold Simar. Applied Multivariate Statistical Analysis. Springer, 2012.
30. Matthew D Hoffman and Andrew Gelman. The No-U-Turn sampler. The Journal of Machine Learning Research, 15(1):1593-1623, 2014.
31. Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303-1347, 2013.
32. Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-233, 1999.
33. Mohammad E Khan, Pierre Baque, Francois Fleuret, and Pascal Fua. Kullback-Leibler proximal variational inference. In Neural Information Processing Systems, 2015.
34. Sangjoon Kim, Neil Shephard, and Siddhartha Chib. Stochastic volatility: likelihood inference and comparison with ARCH models. The Review of Economic Studies, 65(3):361-393, 1998.
35. Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
36. Diederik P Kingma and Jimmy Ba Adam. A method for stochastic optimization. In International Conference on Learning Representation, 2015.
37. David A Knowles. Stochastic gradient variational Bayes for Gamma approximating distributions. arXiv:1509.01631, 2015.
38. Alp Kucukelbir, Rajesh Ranganath, Andrew Gelman, and David Blei. Automatic variational inference in Stan. In Neural Information Processing Systems, 2015.
39. D. Lee and H. Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401 (6755):788-791, 1999.
40. Dougal Maclaurin, David Duvenaud, and Ryan P Adams. Autograd: reverse-mode differentiation of native Python. In ICML Workshop on Automatic Machine Learning, 2015.
41. Maren Mahsereci and Philipp Hennig. Probabilistic line searches for stochastic optimization. In Neural Information Processing Systems, 2015.
42. Vikash Mansinghka, Daniel Selsam, and Yura Perov. Venture: a higher-order probabilistic programming platform with programmable inference. arXiv:1404.0099, 2014.
43. Renate Meyer, David A Fournier, and Andreas Berg. Stochastic volatility: Bayesian computation using automatic differentiation and the extended Kalman filter. The Econometrics Journal, 6(2): 408-420, 2003.
44. T. Minka, J. M. Winn, J. P. Guiver, S. Webster, Y. Zaykov, B. Yangel, A. Spengler, and J. Bronskill. Infer.NET 2.6, 2014. Microsoft Research Cambridge. http://research.microsoft.com/infernet.
45. Kevin P Murphy. Machine Learning: a Probabilistic Perspective. MIT Press, 2012.
46. Radford M Neal. MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 2: 113-162, 2011.
47. David J Olive. Statistical Theory and Inference. Springer, 2014.
48. Manfred Opper and Cedric Archambeau. The variational Gaussian approximation revisited. Neural Computation, 21(3):786-792, 2009.
49. AviPfeffer. Figaro: an object-oriented probabilistic programming language. Charles River Analytics Technical Report, 137, 2009.
50. Jose C Pinheiro and Douglas M Bates. Unconstrained parametrizations for variance-covariance matrices. Statistics and Computing, 6(3):289-296, 1996.
51. Martyn Plummer. JAGS: a program for analysis of Bayesian graphical models using Gibbs sampling. In International Workshop on Distributed Statistical Computing, 2003
52. Thomas Pock, Michael Pock, and Horst Bischof. Algorithmic differentiation: application to variational problems in computer vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(7):1180-1193, 2007.
53. Rajesh Ranganath, Chong Wang, Blei David, and Eric Xing. An adaptive learning rate for stochastic variational inference. In International Conference on Machine Learning, 2013.
54. Rajesh Ranganath, Sean Gerrish, and David M Blei. Black box variational inference. In Artificial Intelligence and Statistics, 2014.
55. Rajesh Ranganath, Dustin Tran, and David M Blei. Hierarchical variational models. In International Conference on Machine Learning, 2016.
56. Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, 2015.
57. Danilo J Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, 2014.
58. Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 1951.
59. Christian P Robert and George Casella. Monte Carlo Statistical Methods. Springer, 1999.
60. Francisco JR Ruiz, Michalis K Titsias, and David M Blei. The generalized reparameterization gradient. In Neural Information Processing Systems, 2016a.
61. Francisco JR Ruiz, Michalis K Titsias, and David M Blei. Overdispersed black-box variational inference. In Uncertainty in Artificial Intelligence, 2016b.
62. Tim Salimans and David Knowles. On using control variates with stochastic approximation for variational Bayes. arXiv:1401.1022, 2014.
63. John Salvatier, Thomas V Wiecki, and Christopher Fonnesbeck. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2:e55, 2016.
64. Matthias Seeger. Gaussian covariance and scalable variational inference. In International Conference on Machine Learning, 2010.
65. David J Spiegelhalter, Andrew Thomas, Nicky G Best, and Wally R Gilks. BUGS: Bayesian inference using Gibbs sampling, version 0.50. MRC Biostatistics Unit, Cambridge, 1995.
66. Stan Development Team. Stan Modeling Language Users Guide and Reference Manual, 2016.
67. Theano Development Team. Theano: a Python framework for fast computation of mathematical expressions. arXiv:1605.02688, 2016.
68. Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4, 2012.
69. Michalis Titsias and Miguel Lazaro-Gredilla. Doubly stochastic variational Bayes for non-conjugate inference. In International Conference on Machine Learning, 2014.
70. Dustin Iran, Alp Kucukelbir, Adji B Dieng, Maja Rudolph, Dawen Liang, and David M Blei. Edward: a library for probabilistic modeling, inference, and criticism. arXiv:1610.09787, 2016a.
71. Dustin Tran, Rajesh Ranganath, and David M Blei. The variational Gaussian process. In International Conference on Learning Representations, 2016b.
72. Richard E Turner and Maneesh Sahani. Two problems with variational expectation maximisation for time-series models. In Workshop on Inference and Estimation in Probabilistic Time-Series Models, 2008.
73. Mauricio Villegas, Roberto Paredes, and Bart Thomee. Overview of the ImageCLEF 2013 Scalable Concept Image Annotation Subtask. In CLEF Evaluation Labs and Workshop, 2013.
74. Martin J Wainwright and Michael I Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2):1-305, 2008.
75. Bernard Widrow and Michael A Lehr. 30 years of adaptive neural networks: perceptron, madaline, and backpropagation. Proceedings of the IEEE, 78(9):1415-1442, 1990.
76. David Wingate and Theophane Weber. Automated variational inference in probabilistic programming. arXiv:1301.1299, 2013.
77. John Winn and Christopher M Bishop. Variational message passing. Journal of Machine Learning Research, 6:661-694, 2005.
78. Frank Wood, Jan Willem van de Meent, and Vikash Mansinghka. A new approach to probabilistic programming inference. In Artificial Intelligence and Statistics, 2014.