Continuous deep q-learning with model-based acceleration

33rd International Conference on Machine Learning, ICML 2016

Volumn 6, Issue , 2016, Pages 4135-4148

  Gu, Shixiang ^a,b,c   Lillicrap, Timothy ^d   Sutskever, Uya ^c   Levine, Sergey ^c  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b MAX PLANCK INSTITUTE FOR INTELLIGENT SYSTEMS   (Germany)

^c GoogIe Brain   (United States)

^d DEEPMIND   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPLEX NETWORKS; DEEP LEARNING; EFFICIENCY; ITERATIVE METHODS; LEARNING ALGORITHMS;

ACTOR-CRITIC METHODS; COMPLEMENTARY TECHNIQUES; CONTINUOUS CONTROL; EXPERIENCE REPLAY; LOCAL LINEAR MODELS; MODEL-FREE ALGORITHMS; PHYSICAL SYSTEMS; SAMPLE COMPLEXITY;

REINFORCEMENT LEARNING;

EID: 84998579328     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (37)

1. Atkeson, Christopher G, Moore, Andrew W, and Schaal, Stefan. Locally weighted learning for control. In Lazy learning, pp. 75-113. Springer, 1997.
2. Baird III, Leemon C. Advantage updating. Technical report, DTIC Document, 1993.
3. de Bruin, Tim, Kober, Jens, Tuyls, Karl, and Babuska, Robert. The importance of experience replay database composition in deep reinforcement learning. Deep Reinforcement Learning Workshop, NIPS, 2015.
4. Deisenroth, Marc and Rasmussen, Carl E. Pilco: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning (ICML), pp. 465-172, 2011.
5. Deisenroth, Marc Peter, Neumann, Gerhard, Peters, Jan, et al. A survey on policy search for robotics. Foundations and Trends in Robotics, 2(1-2): 1-142, 2013.
6. Fu, Justin, Levine, Sergey, and Abbeel, Pieter. One-shot learning of manipulation skills with online dynamics adaptation and neural network priors. arXiv preprint arXiv: 1509.06841, 2015.
7. Hafner, Roland and Riedmiller, Martin. Reinforcement learning in feedback control. Machine learning, 84(1-2): 137-169, 2011.
8. Harmon, Mance E and Baird III, Leemon C. Multi-player residual advantage learning with general function approximation. Wright Laboratory, WL/AACF, Wright-Patterson Air Force Base, OH, pp. 45433-7308, 1996.
9. Hausknecht, Matthew and Stone, Peter. Deep reinforcement learning in parameterized action space. arXiv preprint arXiv: 1511.04143, 2015.
10. Heess, Nicolas, Wayne, Gregory, Silver, David, Lillicrap, Tim, Erez, Tom, and Tassa, Yuval. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems (NIPS), pp. 2926-2934, 2015.
11. Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
12. Kober, Jens and Peters, Jan. Reinforcement learning in robotics: A survey. In Reinforcement Learning, pp. 579-610. Springer, 2012.
13. Konda, Vijay R and Tsitsiklis, John N. Actor-critic algorithms. In Advances in Neural Information Processing Systems (NIPS), volume 13, pp. 1008-1014, 1999.
14. Koutnfk, Jan, Cuccu, Giuseppe, Schmidhuber, Jiirgen, and Gomez, Faustino. Evolving large-scale neural networks for vision-based reinforcement learning. In Proceedings of the 15th annual conference on Genetic and evolutionary computation, pp. 1061-1068. ACM, 2013.
15. Lampe, Thomas and Riedmiller, Martin. Approximate model-assisted neural fitted q-iteration. In Neural Networks (IJCNN), 2014 International Joint Conference on, pp. 2698-2704. IEEE, 2014.
16. Levine, Sergey and Abbeel, Pieter. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems (NIPS), pp. 1071-1079, 2014.
17. Levine, Sergey and Koltun, Vladlen. Guided policy search. In International Conference on Machine Learning (ICML), pp. 1-9, 2013.
18. Levine, Sergey, Finn, Chelsea, Darrell, Trevor, and Abbeel, Pieter. End-to-end training of deep visuomotor policies. JMLR 17, 2016.
19. Li, Weiwei and Todorov, Emanuel. Iterative linear quadratic regulator design for nonlinear biological movement systems. In ICINCO (1), pp. 222-229, 2004.
20. Lillicrap, Timothy P, Hunt, Jonathan J, Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, and Wierstra, Daan. Continuous control with deep reinforcement learning. International Conference on Learning Representations (ICLR), 2016.
21. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Marc G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.
22. Nguyen, D and Widrow, B. The truck backer upper: An example of self learning in neural networks, 1989.
23. Peters, Jan and Schaal, Stefan. Policy gradient methods for robotics. In International Conference on Intelligent Robots and Systems (IROS), pp. 2219-2225. IEEE, 2006.
24. Peters, Jan, Mulling, Katharina, and Altun, Yasemin. Relative entropy policy search. In AAAI. Atlanta, 2010.
25. Rawlik, Konrad, Toussaint, Marc, and Vijayakumar, Sethu. On stochastic optimal control and reinforcement learning by approximate inference. Robotics, pp. 353, 2013.
26. Schaul, Tom, Quan, John, Antonoglou, Ioannis, and Silver, David. Prioritized experience replay. arXiv preprint arXiv: 1511.05952, 2015.
27. Schmidhuber, Jurgen. Reinforcement learning in marko-vian and non-markovian environments, pp. 500-506, 1991.
28. Schulman, John, Levine, Sergey, Abbeel, Pieter, Jordan, Michael I., and Moritz, Philipp. Trust region policy optimization. In International Conference on Machine Learning (ICML), pp. 1889-1897, 2015.
29. Schulman, John, Moritz, Philipp, Levine, Sergey, Jordan, Michael, and Abbeel, Pieter. High-dimensional continuous control using generalized advantage estimation. International Conference on Learning Representations (ICLR), 2016.
30. Silver, David, Lever, Guy, Heess, Nicolas, Degris, Thomas, Wierstra, Daan, and Riedmiller, Martin. Deterministic policy gradient algorithms. In International Conference on Machine Learning (ICML), 2014.
31. Sutton, Richard S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In International Conference on Machine Learning (ICML), pp. 216-224, 1990.
32. Sutton, Richard S, McAllester, David A, Singh, Satin-der P, Mansour, Yishay, et al. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NIPS), volume 99, pp. 1057-1063, 1999.
33. Tassa, Yuval, Erez, Tom, and Todorov, Emanuel. Synthesis and stabilization of complex behaviors through online trajectory optimization. In International Conference on Intelligent Robots and Systems (IROS), pp. 4906-4913. IEEE, 2012.
34. Todorov, Emanuel, Erez, Tom, and Tassa, Yuval. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems (IROS), pp. 5026-5033. IEEE, 2012.
35. Wahlstrom, Niklas, Schon, Thomas B, and Deisenroth, Marc Peter. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint arXiv: 1502.02251, 2015.
36. Wang, Ziyu, de Freitas, Nando, and Lanctot, Marc. Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv:1511.06581, 2015.
37. Watter, Manuel, Springenberg, Jost, Boedecker, Joschka, and Riedmiller, Martin. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in Neural Information Processing Systems (NIPS), pp. 2728-2736, 2015.