Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation

Advances in Neural Information Processing Systems

Volumn 2017-December, Issue , 2017, Pages 5280-5289

  Wu, Yuhuai ^a   Mansimov, Elman ^b   Liao, Shun ^a   Grosse, Roger ^a   Ba, Jimmy ^a  

^a UNIVERSITY OF TORONTO   (Canada)

^b NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CURVE FITTING; GRADIENT METHODS; REINFORCEMENT LEARNING;

ACTOR-CRITIC METHODS; CONTINUOUS CONTROL; CONTINUOUS DOMAIN; DISCRETE CONTROL; NATURAL GRADIENT METHODS; NON-TRIVIAL TASKS; STATE OF THE ART; TRUST-REGION METHODS;

DEEP LEARNING;

EID: 85046992971     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. S. I. Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2): 251-276, 1998.
2. J. Ba, R. Grosse, and J. Martens. Distributed second-order optimization using Kronecker-factored approximations. In ICLR, 2017.
3. J. A. Bagnell and J. G. Schneider. Covariant policy search. In IJCAI, 2003.
4. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
5. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. arXiv preprint arXiv: 1606.01540, 2016.
6. R. Grosse and J. Martens. A Kronecker-factored approximate Fisher matrix for convolutional layers. In ICML, 2016.
7. S. Gu, T. Lillicrap, Z. Ghahramani, R. E. Turner, and S. Levine. Q-prop: Sample-efficient policy gradient with an off-policy critic. In ICLR, 2017.
8. N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. M. A. Eslami, M. Riedmiller, and D. Silver. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: 1707.02286, 2017.
9. M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR, 2017.
10. S. Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems, 2002.
11. D. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015.
12. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
13. J. Martens. Deep learning via Hessian-free optimization. In ICML-10, 2010.
14. J. Martens. New insights and perspectives on the natural gradient method. arXiv preprint arXiv: 1412.1193, 2014.
15. J. Martens and R. Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In ICML, 2015.
16. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540): 529-533, 2015.
17. V. Mnih, A. Puigdomenech Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, 2016.
18. J. Nocedal and S. Wright. Numerical Optimization. Springer, 2006.
19. J. Peters and S. Schaal. Natural actor-critic. Neurocomputing, 71(7-9): 1180-1190, 2008.
20. N. N. Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural Computation, 2002.
21. J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015.
22. J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
23. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
24. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587): 484-489, 2016.
25. R. S. Sutton, D. A. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12, 2000.
26. E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
27. Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, and N. de Freitas. Sample efficient actor-critic with experience replay. In ICLR, 2016.
28. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3): 229-256, 1992.