Gradient estimation using stochastic computation graphs

Advances in Neural Information Processing Systems

Volumn 2015-January, Issue , 2015, Pages 3528-3536

  Schulman, John ^a,b   Heess, Nicolas ^a   Weber, Theophane ^a   Abbeel, Pieter ^b  

^a DEEPMIND   (United Kingdom)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BACKPROPAGATION ALGORITHMS; DIRECTED GRAPHS; ESTIMATION; INFORMATION SCIENCE; LEARNING ALGORITHMS; PROBABILITY DISTRIBUTIONS; REINFORCEMENT LEARNING; STOCHASTIC MODELS;

CONDITIONAL PROBABILITY DISTRIBUTIONS; DETERMINISTIC FUNCTIONS; DIRECTED ACYCLIC GRAPH (DAG); GRADIENT-BASED LEARNING; PROBABILISTIC MODELING; SIMPLE MODIFICATIONS; STOCHASTIC COMPUTATIONS; VARIANCE REDUCTION TECHNIQUES;

STOCHASTIC SYSTEMS;

EID: 84965157716     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (29)

1. J. Baxter and P. L. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research, pages 319-350, 2001.
2. Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.
3. M. C. Fu. Gradient estimation. Handbooks in operations research and management science, 13:575-616, 2006.
4. P. Glasserman. Monte Carlo methods in financial engineering, volume 53. Springer Science & Business Media, 2003.
5. P. W. Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10):75-84, 1990.
6. E. Greensmith, P. L. Bartlett, and J. Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. The Journal of Machine Learning Research, 5:1471-1530, 2004.
7. K. Gregor, I. Danihelka, A. Mnih, C. Blundell, and D. Wierstra. Deep autoregressive networks. arXiv preprint arXiv:1310.8499, 2013.
8. A. Griewank and A. Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation. Siam, 2008.
9. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.
10. D. P. Kingma and M. Welling. Auto-encoding variational Bayes. arXiv:1312.6114, 2013.
11. D. P. Kingma and M. Welling. Efficient gradient-based inference through transformations between bayes nets and neural nets. arXiv preprint arXiv:1402.0480, 2014.
12. Y. Le Cun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
13. J. Martens. Deep learning via Hessian-free optimization. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 735-742, 2010.
14. A. Mnih and K. Gregor. Neural variational inference and learning in belief networks. arXiv:1402.0030, 2014.
15. V. Mnih, N. Heess, A. Graves, and K. Kavukcuoglu. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pages 2204-2212, 2014.
16. R. Munos. Policy gradient in continuous time. The Journal of Machine Learning Research, 7:771-791, 2006.
17. R. M. Neal. Learning stochastic feedforward networks. Department of Computer Science, University of Toronto, 1990.
18. R. M. Neal and G. E. Hinton. A view of the em algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, pages 355-368. Springer, 1998.
19. J. Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kaufmann, 2014.
20. R. Ranganath, S. Gerrish, and D. M. Blei. Black box variational inference. arXiv preprint arXiv:1401.0118, 2013.
21. D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv:1401.4082, 2014.
22. D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.
23. R. S. Sutton, D. A. McAllester, S. P. Singh, Y. Mansour, et al. Policy gradient methods for reinforcement learning with function approximation. In NIPS, volume 99, pages 1057-1063. Citeseer, 1999.
24. N. Vlassis, M. Toussaint, G. Kontes, and S. Piperidis. Learning model-free robot control by a Monte Carlo EM algorithm. Autonomous Robots, 27(2):123-130, 2009.
25. D. Wierstra, A. Förster, J. Peters, and J. Schmidhuber. Recurrent policy gradients. Logic Journal of IGPL, 18(5):620-634, 2010.
26. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
27. D. Wingate and T. Weber. Automated variational inference in probabilistic programming. arXiv preprint arXiv:1301.1299, 2013.
28. S. J. Wright and J. Nocedal. Numerical optimization, volume 2. Springer New York, 1999.
29. W. Zaremba and I. Sutskever. Reinforcement learning neural Turing machines. arXiv preprint arXiv:1505.00521, 2015.