Interpolated policy gradient: Merging on-policy and off-policy gradient estimation for deep reinforcement learning

Advances in Neural Information Processing Systems

Volumn 2017-December, Issue , 2017, Pages 3847-3856

  Gu, Shixiang ^a   Lillicrap, Timothy ^b   Ghahramani, Zoubin ^a   Turner, Richard E ^a   Schölkopf, Bernhard ^a   Levine, Sergey ^c  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b DEEPMIND   (United Kingdom)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

MERGING; REINFORCEMENT LEARNING;

CONTINUOUS CONTROL; CONTROL VARIATES; EMPIRICAL - COMPARISONS; EMPIRICAL PERFORMANCE; PERFORMANCE BOUNDS; REINFORCEMENT LEARNING METHOD; STATE OF THE ART; THEORETICAL GUARANTEES;

DEEP LEARNING;

EID: 85047014445     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (31)

1. Bagnell, J Andrew and Schneider, Jeff. Covariant policy search. IJCAI, 2003.
2. Brockman, Greg, Cheung, Vicki, Pettersson, Ludwig, Schneider, Jonas, Schulman, John, Tang, Jie, and Zaremba, Wojciech. Openai gym. arXiv preprint arXiv:1606.01540 2016.
3. Degris, Thomas, White, Martha, and Sutton, Richard S. Off-policy actor-critic. arXiv preprint arXiv:1205.4839 2012.
4. Duan, Yan, Chen, Xi, Houthooft, Rein, Schulman, John, and Abbeel, Pieter. Benchmarking deep reinforcement learning for continuous control. International Conference on Machine Learning (ICML), 2016.
5. Gu, Shixiang, Lillicrap, Timothy, Ghahramani, Zoubin, Turner, Richard E, and Levine, Sergey. Q-prop: Sample-efficient policy gradient with an off-policy critic. ICLR, 2017.
6. Heess, Nicolas, Wayne, Gregory, Silver, David, Lillicrap, Tim, Erez, Tom, and Tassa, Yuval. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944-2952, 2015.
7. Jiang, Nan and Li, Lihong. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 652-661, 2016.
8. Jie, Tang and Abbeel, Pieter. On a connection between importance sampling and the likelihood ratio policy gradient. In Advances in Neural Information Processing Systems, pp. 1000-1008, 2010.
9. Kakade, Sham and Langford, John. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning (ICML), Volume 2, pp. 267-274, 2002.
10. Levine, Sergey and Koltun, Vladlen. Guided policy search. In International Conference on Machine Learning (ICML), pp. 1-9, 2013.
11. Levine, Sergey, Finn, Chelsea, Darrell, Trevor, and Abbeel, Pieter. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
12. Lillicrap, Timothy P, Hunt, Jonathan J, Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, and Wierstra, Daan. Continuous control with deep reinforcement learning. ICLR, 2016.
13. Mahmood, A Rupam, van Hasselt, Hado P, and Sutton, Richard S. Weighted importance sampling for off-policy learning with linear function approximation. In Advances in Neural Information Processing Systems, pp. 3014-3022, 2014.
14. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, et al. Humanlevel control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.
15. Munos, Rémi, Stepleton, Tom, Harutyunyan, Anna, and Bellemare, Marc G. Safe and efficient off-policy reinforcement learning. arXiv preprint arXiv:1606.02647 2016.
16. O'Donoghue, Brendan, Munos, Remi, Kavukcuoglu, Koray, and Mnih, Volodymyr. Pgq: Combining policy gradient and q-learning. ICLR, 2017.
17. Peshkin, Leonid and Shelton, Christian R. Learning from scarce experience. In Proceedings of the Nineteenth International Conference on Machine Learning, 2002.
18. Peters, Jan, Mülling, Katharina, and Altun, Yasemin. Relative entropy policy search. In AAAI. Atlanta, 2010.
19. Precup, Doina. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
20. Riedmiller, Martin. Neural fitted q iteration-first experiences with a data efficient neural reinforcement learning method. In European Conference on Machine Learning, pp. 317-328. Springer, 2005.
21. Ross, Sheldon M. Simulation. Burlington, MA: Elsevier, 2006.
22. Schulman, John, Levine, Sergey, Abbeel, Pieter, Jordan, Michael I, and Moritz, Philipp. Trust region policy optimization. In ICML, pp. 1889-1897, 2015.
23. Schulman, John, Moritz, Philipp, Levine, Sergey, Jordan, Michael, and Abbeel, Pieter. High-dimensional continuous control using generalized advantage estimation. International Conference on Learning Representations (ICLR), 2016.
24. Silver, David, Lever, Guy, Heess, Nicolas, Degris, Thomas, Wierstra, Daan, and Riedmiller, Martin. Deterministic policy gradient algorithms. In International Conference on Machine Learning (ICML), 2014.
25. Silver, David, Huang, Aja, Maddison, Chris J, Guez, Arthur, Sifre, Laurent, Van Den Driessche, George, Schrittwieser, Julian, Antonoglou, Ioannis, Panneershelvam, Veda, Lanctot, Marc, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.
26. Sutton, Richard S, McAllester, David A, Singh, Satinder P, Mansour, Yishay, et al. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NIPS), Volume 99, pp. 1057-1063, 1999.
27. Thomas, Philip. Bias in natural actor-critic algorithms. In ICML, pp. 441-448, 2014.
28. Thomas, Philip and Brunskill, Emma. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 2139-2148, 2016.
29. Todorov, Emanuel, Erez, Tom, and Tassa, Yuval. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
30. Wang, Ziyu, Bapst, Victor, Heess, Nicolas, Mnih, Volodymyr, Munos, Remi, Kavukcuoglu, Koray, and de Freitas, Nando. Sample efficient actor-critic with experience replay. ICLR, 2017.
31. Williams, Ronald J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.