High-Order stochastic gradient thermostats for Bayesian learning of deep models

30th AAAI Conference on Artificial Intelligence, AAAI 2016

Volumn , Issue , 2016, Pages 1795-1801

  Li, Chunyuan ^a   Chen, Changyou ^a   Fan, Kai ^b   Carin, Lawrence ^a  

^a Duke University   (United States)

^b DUKE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; BAYESIAN NETWORKS; CONTINUOUS TIME SYSTEMS; STOCHASTIC SYSTEMS; THERMOSTATS;

BAYESIAN LEARNING; MEASURE OF UNCERTAINTY; NUMERICAL APPROXIMATIONS; NUMERICAL INTEGRATOR; POSTERIOR DISTRIBUTIONS; STATIONARY DISTRIBUTION; STOCHASTIC GRADIENT; SYMMETRIC SPLITTING;

STOCHASTIC MODELS;

EID: 85007273869     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (45)

1. Bengio, Y.; Simard, P.; and Frasconi, P. 1994. Learning long-term dependencies with gradient descent is difficult. TNN.
2. Blei, D. M.; Ng, A. Y.; and Jordan, M. I. 2003. Latent Dirichlet allocation. JMLR.
3. Blundell, C.; Cornebise, J.; Kavukcuoglu, K.; and Wierstra, D. 2015. Weight uncertainty in neural networks. In ICML.
4. Chao, W.-L.; Solomon, J.; Michels, D.; and Sha, F. 2015. Exponential integration for Hamiltonian Monte Carlo.
5. Chen, C.; Buntine, W.; Ding, N.; Xie, L.; and Du, L. 2015. Differential topic models. TPAMI.
6. Chen, C.; Ding, N.; and Carin, L. 2015. On the convergence of stochastic gradient MCMC algorithms with high-order integrators. In NIPS.
7. Chen, T.; Fox, E. B.; and Guestrin, C. 2014. Stochastic gradient Hamiltonian Monte Carlo. In ICML.
8. Ding, N.; Fang, Y.; Babbush, R.; Chen, C.; Skeel, R. D.; and Neven, H. 2014. Bayesian sampling using stochastic gradient thermostats. In NIPS.
9. Fan, K.; Wang, Z.; Beck, J.; Kwok, J.; and Heller, K. 2015. Fast second-order stochastic backpropagation for variational inference. In NIPS.
10. Gal, Y., and Ghahramani, Z. 2015. Bayesian convolutional neural networks with Bernoulli approximate variational inference. arXiv:1506.02158.
11. Gan, Z.; Chen, C.; Henao, R.; Carlson, D.; and Carin, L. 2015a. Scalable deep Poisson factor analysis for topic modeling. In ICML.
12. Gan, Z.; Henao, R.; Carlson, D.; and Carin, L. 2015b. Learning deep sigmoid belief networks with data augmentation. In AISTATS.
13. Gan, Z.; Li, C.; Henao, R.; Carlson, D.; and Carin, L. 2015c. Deep temporal sigmoid belief networks for sequence modeling. NIPS.
14. Glorot, X.; Bordes, A.; and Bengio, Y. 2011. Deep sparse rectifier neural networks. In AISTATS.
15. Guhaniyogi, R.; Qamar, S.; and Dunson, D. B. 2014. Bayesian conditional density filtering. arXiv:1401.3632.
16. Hernández-Lobato, J. M., and Adams, R. P. 2015. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In ICML.
17. Hinton, G. E. 2002. Training products of experts by minimizing contrastive divergence. In Neural computation.
18. Hoffman, M.; Bach, F. R.; and Blei, D. M. 2010. Online learning for latent Dirichlet allocation. In NIPS.
19. Kemp, C.; Perfors, A.; and Tenenbaum, J. B. 2007. Learning overhypotheses with hierarchical Bayesian models. Developmental science.
20. Kingma, D. P., and Welling, M. 2014. Auto-encoding variational Bayes. In ICLR.
21. Knapp, A. W. 2005. Basic real analysis. Sci. & B. M.
22. Korattikara, A.; Rathod, V.; Murphy, K.; and Welling, M. 2015. Bayesian dark knowledge. NIPS.
23. Leimkuhler, B., and Matthews, C. 2013. Rational construction of stochastic numerical methods for molecular sampling. Applied Math. Research Express.
24. Leimkuhler, B., and Shang, X. 2015. Adaptive thermostats for noisy gradient systems. arXiv:1505.06889.
25. Li, C.; Chen, C.; Carlson, D.; and Carin, L. 2016. Preconditioned stochastic gradient Langevin dynamics for deep neural networks. In AAAI.
26. Lin, C.-J.;Weng, R. C.; and Keerthi, S. S. 2008. Trust region newton method for logistic regression. JMLR.
27. Ma, Y. A.; Chen, T.; and Fox, E. B. 2015. A complete recipe for stochastic gradient MCMC. NIPS.
28. MacKay, D. 1992. A practical Bayesian framework for backpropagation networks. Neural computation.
29. Martens, J. 2010. Deep learning via Hessian-free optimization. In ICML.
30. Mnih, A., and Gregor, K. 2014. Neural variational inference and learning in belief networks. In ICML.
31. Neal, R. M. 1995. Bayesian learning for neural networks. PhD thesis, University of Toronto.
32. Neal, R. M. 2011. MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo.
33. Patterson, S., and Teh, Y. W. 2013. Stochastic gradient Riemannian Langevin dynamics on the probability simplex. In NIPS.
34. Pu, Y.; Yuan, X.; and Carin, L. 2015. A generative model for deep convolutional learning. In ICLR workshop.
35. Ranganath, R.; Tang, L.; Charlin, L.; and Blei, D. M. 2015. Deep exponential families. AISTATS.
36. Risken, H. 1989. The Fokker-Planck equation. Springer- Verlag, New York.
37. Rossmann, W. 2002. Lie Groups-An Introduction Through Linear Groups. Oxford Graduate Texts in Mathematics, Oxford Science Publications.
38. Rumelhart, D. E.; E., H. G.; andWilliams, R. J. 1986. Learning representations by back-propagating errors. Nature.
39. Salakhutdinov, R.; Tenenbaum, J. B.; and Torralba, A. 2013. Learning with hierarchical-deep models. TPAMI.
40. Shahbaba, B.; Lan, S.; Johnson, W. O.; and Neal, R. M. 2014. Split Hamiltonian Monte Carlo. Stat. and Comp.
41. Sutskever, I.; Martens, J.; Dahl, G.; and Hinton, G. 2013. On the importance of initialization and momentum in deep learning. In ICML.
42. Titsias, M., and Ĺazaro-Gredilla, M. 2014. Doubly stochastic variational Bayes for non-conjugate inference. In ICML.
43. Welling, M., and Teh, Y. W. 2011. Bayesian learning via stochastic gradient Langevin dynamics. In ICML.
44. Xu, F., and Tenenbaum, J. B. 2007. Word learning as Bayesian inference. Psychological review.
45. Zhou, M., and Carin, L. 2015. Negative binomial process count and mixture modeling. TPAMI.