Bridging the gap between stochastic gradient MCMC and stochastic optimization

Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016

Volumn , Issue , 2016, Pages 1051-1060

  Chen, Changyou ^a   Carlson, David ^b   Gan, Zhe ^a   Li, Chunyuan ^a   Carin, Lawrence ^a  

^a Duke University   (United States)

^b Columbia University ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; DEEP NEURAL NETWORKS; MARKOV PROCESSES; MOMENTUM; MONTE CARLO METHODS; SIMULATED ANNEALING; STOCHASTIC SYSTEMS; TEMPERATURE;

CONVENTIONAL OPTIMIZATION; MARKOV CHAIN MONTE-CARLO; NEURAL NETWORK MODEL; STOCHASTIC GRADIENT; STOCHASTIC OPTIMIZATION ALGORITHM; STOCHASTIC OPTIMIZATION METHODS; STOCHASTIC OPTIMIZATIONS; ZERO TEMPERATURES;

STOCHASTIC MODELS;

EID: 84986265678     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (39)

1. C. Andrieu, N. de Freitas, and A. Doucet. Reversible jump mcmc simulated annealing for neural networks. In UAI, 2000.
2. C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural networks. In ICML, 2015.
3. L. Bottou. Large-scale machine learning with stochastic gradient descent. In Proc. COMPSTAT, 2010.
4. N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent. Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. In ICML, 2012.
5. D. E. Carlson, E. Collins, Y.-P. Hsieh, L. Carin, and V. Cevher. Preconditioned spectral descent for deep learning. In Advances in Neural Information Processing Systems, pages 2953–2961, 2015.
6. C. Chen, N. Ding, and L. Carin. On the convergence of stochastic gradient mcmc algorithms with high-order integrators. In NIPS, 2015.
7. T. Chen, E. B. Fox, and C. Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In ICML, 2014.
8. K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In arXiv:1406.1078, 2014.
9. Y. N. Dauphin, H. de Vries, and Y. Bengio. Equilibrated adaptive learning rates for non-convex optimization. In NIPS, 2015.
10. N. Ding, Y. Fang, R. Babbush, C. Chen, R. D. Skeel, and H. Neven. Bayesian sampling using stochastic gradient thermostats. In NIPS, 2014.
11. J. Duchi, E. Hazan, and Y. Singer. Adaptive sub-gradient methods for online learning and stochastic optimization. In JMLR, 2011.
12. Z. Gan, C. Chen, R. Henao, D. Carlson, and L. Carin. Scalable deep Poisson factor analysis for topic modeling. In ICML, 2015.
13. S. Geman and D. Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. In PAMI, 1984.
14. M. Girolami and B. Calderhead. Riemann manifold Langevin and Hamiltonian Monte Carlo methods. In JRSS, 2011.
15. I. Goodfellow, D. Warde-farley, M. Mirza, A. Courville, and Y. Bengio. Maxout networks. In ICML, 2013.
16. K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. Le-Cun. What is the best multi-stage architecture for object recognition? In ICCV, 2009.
17. D. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
18. S. Kirkpatrick, C. D. G. Jr, and M. P. Vecchi. Optimization by simulated annealing. In Science, 1983.
19. C. Li, C. Chen, D. Carlson, and L. Carin. Preconditioned stochastic gradient Langevin dynamics for deep neural networks. In AAAI, 2016a.
20. C. Li, C. Chen, K. Fan, and L. Carin. High-order stochastic gradient thermostats for Bayesian learning of deep models. In AAAI, 2016b.
21. Y. Li, V. A. Protopopescu, N. Arnold, X. Zhang, and A. Gorin. Hybrid parallel tempering and simulated annealing method. In Applied Mathematics and Computation, 2009.
22. M. Lin, Q. Chen, and S. Yan. Network in network. In ICLR, 2014.
23. J. C. Mattingly, A. M. Stuart, and M. V. Tretyakov. Construction of numerical time-average and stationary measures via Poisson equations. In SIAM J. NUMER. ANAL., 2010.
24. R. M. Neal. Annealed importance sampling. In Statistics and Computing, 2001.
25. R. M. Neal. Mcmc using hamiltonian dynamics. In Handbook of Markov Chain Monte Carlo, 2011.
26. F. Obermeyer, J. Glidden, and E. Jonas. Scaling nonparametric bayesian inference via subsample-annealing. In AISTATS, 2014.
27. S. Patterson and Y. W. Teh. Stochastic gradient Riemannian Langevin dynamics on the probability simplex. In NIPS, 2013.
28. H. Risken. The Fokker-Planck equation. Springer-Verlag, New York, 1989.
29. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. In Nature, 1986.
30. E. Schechter. Handbook of Analysis and Its Foundations. Elsevier, 1997.
31. I. Sutskever, J. Martens, G. Dahl, and G. E. Hinton. On the importance of initialization and momentum in deep learning. In ICML, 2013.
32. Y. W. Teh, A. H. Thiery, and S. J. Vollmer. Consistency and fluctuations for stochastic gradient Langevin dynamics. In arXiv:1409.0578, 2014.
33. T. Tieleman and G. E. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. In Coursera: Neural Networks for Machine Learning, 2012.
34. J. W. van de Meent, B. Paige, and F. Wood. Tempering by subsampling. In arXiv:1401.7145, 2014.
35. V. Černý. Thermodynamical approach to the traveling salesman problem: An efficient simulation algorithm. In J. Optimization Theory and Applications, 1985.
36. S. J. Vollmer, K. C. Zygalakis, and Y. W. Teh. (Non-)asymptotic properties of stochastic gradient Langevin dynamics. In arXiv:1501.00438, 2015.
37. M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient Langevin dynamics. In ICML, 2011.
38. M. Zeiler and R. Fergus. Stochastic pooling for regularization of deep convolutional neural networks. In ICLR, 2013.
39. M. D. Zeiler. Adadelta: An adaptive learning rate method. In arXiv:1212.5701, 2012.