Fast second-order stochastic backpropagation for variational inference

Advances in Neural Information Processing Systems

Volumn 2015-January, Issue , 2015, Pages 1387-1395

  Fan, Kai ^a   Wang, Ziteng ^b   Beck, Jeffrey ^a   Kwok, James T ^b   Heller, Katherine ^a  

^a DUKE UNIVERSITY   (United States)

^b HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

[No Author keywords available]

Indexed keywords

BACKPROPAGATION; GRADIENT METHODS; INFORMATION SCIENCE;

GRADIENT COMPUTATION; LIPSCHITZ CONTINUOUS; LOGISTIC REGRESSIONS; OPTIMIZATION METHOD; QUASI-NEWTON OPTIMIZATION; REAL-WORLD DATASETS; STOCHASTIC GRADIENT METHODS; VARIATIONAL INFERENCE;

STOCHASTIC SYSTEMS;

EID: 84965120997     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. Matthew James Beal. Variational algorithms for approximate Bayesian inference. PhD thesis, 2003.
2. David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3, 2003.
3. Joseph-Frédéric Bonnans, Jean Charles Gilbert, Claude Lemaréchal, and Claudia A Sagastizábal. Numerical optimization: theoretical and practical aspects. Springer Science & Business Media, 2006.
4. Georges Bonnet. Transformations des signaux aléatoires a travers les systèmes non linéaires sans mémoire. Annals of Telecommunications, 19(9):203-220, 1964.
5. George E Dahl, Tara N Sainath, and Geoffrey E Hinton. Improving deep neural networks for lvcsr using rectified linear units and dropout. In ICASSP, 2013.
6. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, 2011.
7. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11:625-660, 2010.
8. Thomas S Ferguson. Location and scale parameters in exponential families of distributions. Annals of Mathematical Statistics, pages 986-1001, 1962.
9. Zhe Gan, Chunyuan Li, Ricardo Henao, David Carlson, and Lawrence Carin. Deep temporal sigmoid belief networks for sequence modeling. In NIPS, 2015.
10. Karol Gregor, Ivo Danihelka, Alex Graves, and Daan Wierstra. Draw: A recurrent neural network for image generation. In ICML, 2015.
11. James Hensman, Magnus Rattray, and Neil D Lawrence. Fast variational inference in the conjugate exponential family. In NIPS, 2012.
12. Geoffrey E Hinton, Peter Dayan, Brendan J Frey, and Radford M Neal. The "wake-sleep" algorithm for unsupervised neural networks. Science, 268(5214):1158-1161, 1995.
13. Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507, 2006.
14. Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14(1):1303-1347, 2013.
15. Mohammad E Khan. Decoupled variational gaussian inference. In NIPS, 2014.
16. Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In NIPS, 2014.
17. Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
18. James Martens. Deep learning via hessian-free optimization. In ICML, 2010.
19. Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In ICML, 2014.
20. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.
21. Jiquan Ngiam, Adam Coates, Ahbik Lahiri, Bobby Prochnow, Quoc V Le, and Andrew Y Ng. On optimization methods for deep learning. In ICML, 2011.
22. Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013.
23. Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 6(1):147-160, 1994.
24. Robert Price. A useful theorem for nonlinear devices having gaussian inputs. Information Theory, IRE Transactions on, 4(2):69-72, 1958.
25. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014.
26. Tim Salimans. Markov chain monte carlo and variational inference: Bridging the gap. In ICML, 2015.
27. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural networks. In NIPS, 2014.
28. Michalis Titsias and Miguel Lázaro-Gredilla. Doubly stochastic variational bayes for non-conjugate inference. In ICML, 2014.