The mirage of action-dependent baselines in reinforcement learning

6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings

Volumn , Issue , 2018, Pages

  Tucker, George ^a   Bhupatiraju, Surya ^a   Gu, Shixiang ^a,b,c   Turner, Richard E ^b   Ghahramani, Zoubin ^b,e   Levine, Sergey ^a,d  

^a GOOGLE INC   (United States)

^b UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^c MAX PLANCK INSTITUTE FOR INTELLIGENT SYSTEMS   (Germany)

^d UNIVERSITY OF CALIFORNIA   (United States)

^e Uber AI Labs   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DECISION MAKING; GRADIENT METHODS; MACHINE LEARNING; OPEN SOURCE SOFTWARE; OPEN SYSTEMS;

BENCHMARK DOMAINS; FUNCTION APPROXIMATORS; GRADIENT ESTIMATES; GRADIENT ESTIMATOR; OPEN-SOURCE CODE; POLICY GRADIENT METHODS; SEQUENTIAL DECISION MAKING; VARIANCE REDUCTIONS;

REINFORCEMENT LEARNING;

EID: 85083951605     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1329–1338, 2016.
2. Will Grathwohl, Dami Choi, Yuhuai Wu, Geoff Roeder, and David Duvenaud. Backpropagation through the void: Optimizing control variates for black-box gradient estimation. International Conference on Learning Representations (ICLR), 2018.
3. Audrunas Gruslys, Mohammad Gheshlaghi Azar, Marc G Bellemare, and Remi Munos. The reactor: A sample-efficient actor-critic architecture. arXiv preprint arXiv:1704.04651, 2017.
4. Shixiang Gu, Tim Lillicrap, Richard E Turner, Zoubin Ghahramani, Bernhard Schölkopf, and Sergey Levine. Interpolated policy gradient: Merging on-policy and off-policy gradient estimation for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3849–3858, 2017a.
5. Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E Turner, and Sergey Levine. Q-prop: Sample-efficient policy gradient with an off-policy critic. International Conference on Learning Representations (ICLR), 2017b.
6. Tang Jie and Pieter Abbeel. On a connection between importance sampling and the likelihood ratio policy gradient. In Advances in Neural Information Processing Systems, pp. 1000–1008, 2010.
7. Sham M Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems, pp. 1531–1538, 2002.
8. Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
9. Hao Liu, Yihao Feng, Yi Mao, Dengyong Zhou, Jian Peng, and Qiang Liu. Action-dependent control variates for policy optimization via stein identity. International Conference on Learning Representations (ICLR), 2018.
10. Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. arXiv preprint arXiv:1402.0030, 2014.
11. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pp. 1928–1937, 2016.
12. Art B Owen. Monte carlo theory, methods and examples. Monte Carlo Theory, Methods and Examples. Art Owen, 2013.
13. Jan Peters and Stefan Schaal. Policy gradient methods for robotics. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on, pp. 2219–2225. IEEE, 2006.
14. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.
15. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889–1897, 2015a.
16. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015b.
17. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
18. David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.
19. Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction, volume 1. MIT Press Cambridge, 1998.
20. Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, pp. 1057–1063, 2000.
21. Philip Thomas. Bias in natural actor-critic algorithms. In International Conference on Machine Learning, pp. 441–448, 2014.
22. Philip S Thomas and Emma Brunskill. Policy gradient methods for reinforcement learning with function approximation and action-dependent baselines. arXiv preprint arXiv:1706.06643, 2017.
23. Lex Weaver and Nigel Tao. The optimal reward baseline for gradient-based reinforcement learning. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, pp. 538–545. Morgan Kaufmann Publishers Inc., 2001.
24. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Reinforcement Learning, pp. 5–32. Springer, 1992.
25. Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. International Conference on Learning Representations (ICLR), 2018.