Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates

Proceedings - IEEE International Conference on Robotics and Automation

Volumn , Issue , 2017, Pages 3389-3396

  Gu, Shixiang ^a,b,c   Holly, Ethan ^a   Lillicrap, Timothy ^d   Levine, Sergey ^a,e  

^a GOOGLE INC   (United States)

^b UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^c Autonomous Vision Group   (United States)

^d DEEPMIND   (United Kingdom)

^e UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPLEX NETWORKS; DEEP LEARNING; DEEP NEURAL NETWORKS; PERSONNEL TRAINING; REINFORCEMENT LEARNING; ROBOTICS; ROBOTS;

3D MANIPULATION TASKS; EXPERIMENTAL EVALUATION; HUMAN INTERVENTION; LEARNING PROCESS; REAL PHYSICAL SYSTEMS; ROBOTIC APPLICATIONS; ROBOTIC MANIPULATION; SAMPLE COMPLEXITY;

LEARNING ALGORITHMS;

EID: 85027967014     PISSN: 10504729     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICRA.2017.7989385     Document Type: Conference Paper

References (42)

1. N. Kohl and P. Stone, "Policy gradient reinforcement learning for fast quadrupedal locomotion", in International Conference on Robotics and Automation (IROS), 2004.
2. G. Endo, J. Morimoto, T. Matsubara, J. Nakanishi, and G. Cheng, "Learning CPG-based biped locomotion with a policy gradient method: Application to a humanoid robot", International Journal of Robotic Research, vol. 27, no. 2, pp. 213-228, 2008.
3. J. Peters and S. Schaal, "Reinforcement learning of motor skills with policy gradients", Neural Networks, vol. 21, no. 4, pp. 682-697, 2008.
4. E. Theodorou, J. Buchli, and S. Schaal, "Reinforcement learning of motor skills in high dimensions", in ICRA, 2010.
5. J. Peters, K. Mülling, and Y. Altün, "Relative entropy policy search", in AAAI Conference on Artificial Intelligence, 2010.
6. M. Kalakrishnan, L. Righetti, P. Pastor, and S. Schaal, "Learning force control policies for compliant manipulation", in International Conference on Intelligent Robots and Systems (IROS), 2011.
7. P. Abbeel, A. Coates, M. Quigley, and A. Ng, "An application of reinforcement learning to aerobatic helicopter flight", in Advances in Neural Information Processing Systems (NIPS), 2006.
8. J. Kober, J. A. Bagnell, and J. Peters, "Reinforcement learning in robotics: A survey", International Journal of Robotic Research, vol. 32, no. 11, pp. 1238-1274, 2013.
9. P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal, "Learning and generalization of motor skills by learning from demonstration", in ICRA, 2009.
10. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, "Continuous control with deep reinforcement learning", ICLR, 2016.
11. S. Gu, T. Lillicrap, I. Sutskever, and S. Levine, "Continuous deep q-learning with model-based acceleration", in ICML, 2016.
12. M. Deisenroth, G. Neumann, and J. Peters, "A survey on policy search for robotics", Foundations and Trends in Robotics, vol. 2, no. 1-2, pp. 1-142, 2013.
13. K. J. Hunt, D. Sbarbaro, R. Żbikowski, and P. J. Gawthrop, "Neural networks for control systems: A survey", Automatica, vol. 28, no. 6, pp. 1083-1112, Nov. 1992.
14. M. Riedmiller, "Neural fitted q iteration-first experiences with a data efficient neural reinforcement learning method", in European Conference on Machine Learning. Springer, 2005, pp. 317-328.
15. R. Hafner and M. Riedmiller, "Neural reinforcement learning controllers for a real robot application", in ICRA, 2007.
16. M. Riedmiller, S. Lange, and A. Voigtlaender, "Autonomous reinforcement learning on raw visual input data in a real world application", in International Joint Conference on Neural Networks, 2012.
17. J. Koutník, G. Cuccu, J. Schmidhuber, and F. Gomez, "Evolving largescale neural networks for vision-based reinforcement learning", in Conference on Genetic and Evolutionary Computation, ser. GECCO'13, 2013.
18. J. Schulman, S. Levine, P. Moritz, M. Jordan, and P. Abbeel, "Trust region policy optimization", in ICML, 2015.
19. S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-end training of deep visuomotor policies", Journal of Machine Learning Research (JMLR), vol. 17, 2016.
20. M. Deisenroth and C. Rasmussen, "PILCO: a model-based and dataefficient approach to policy search", in ICML, 2011.
21. T. Moldovan, S. Levine, M. Jordan, and S. Abbeel, "Optimism-driven exploration for nonlinear systems", in ICRA, 2015.
22. R. Lioutikov, A. Paraschos, G. Neumann, and J. Peters, "Samplebased information-theoretic stochastic optimal control", in International Conference on Robotics and Automation, 2014.
23. M. P. Deisenroth, G. Neumann, J. Peters et al., "A survey on policy search for robotics." Foundations and Trends in Robotics, vol. 2, no. 1-2, pp. 1-142, 2013.
24. R. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning", Machine Learning, vol. 8, no. 3-4, pp. 229-256, May 1992.
25. C. J. Watkins and P. Dayan, "Q-learning", Machine learning, vol. 8, no. 3-4, pp. 279-292, 1992.
26. R. Sutton, D. McAllester, S. Singh, and Y. Mansour, "Policy gradient methods for reinforcement learning with function approximation", in Advances in Neural Information Processing Systems (NIPS), 1999.
27. J. Koutník, G. Cuccu, J. Schmidhuber, and F. Gomez, "Evolving largescale neural networks for vision-based reinforcement learning", in Proceedings of the 15th annual conference on Genetic and evolutionary computation. ACM, 2013, pp. 1061-1068.
28. V. Mnih et al., "Human-level control through deep reinforcement learning", Nature, vol. 518, no. 7540, pp. 529-533, 2015.
29. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, "Asynchronous methods for deep reinforcement learning", in ICML, 2016, pp. 1928-1937.
30. R. Hafner and M. Riedmiller, "Reinforcement learning in feedback control", Machine learning, vol. 84, no. 1-2, pp. 137-169, 2011.
31. M. Inaba, S. Kagami, F. Kanehiro, and Y. Hoshino, "A platform for robotics research based on the remote-brained robot approach", International Journal of Robotics Research, vol. 19, no. 10, 2000.
32. J. Kuffner, "Cloud-enabled humanoid robots", in IEEE-RAS International Conference on Humanoid Robotics, 2010.
33. B. Kehoe, A. Matsukawa, S. Candido, J. Kuffner, and K. Goldberg, "Cloud-based robot grasping with the google object recognition engine", in ICRA, 2013.
34. B. Kehoe, S. Patil, P. Abbeel, and K. Goldberg, "A survey of research on cloud robotics and automation", IEEE Transactions on Automation Science and Engineering, vol. 12, no. 2, April 2015.
35. A. Yahya, A. Li, M. Kalakrishnan, Y. Chebotar, and S. Levine, "Collective robot reinforcement learning with distributed asynchronous guided policy search", arXiv preprint arXiv:1610.00673, 2016.
36. A. Sahara, M. Imai, and Y. Anzai, "Cahra: Collision avoidance system for humanoid robot arms with potential field", in IEEE lntemational Conference on Systems, Man and Cybemetics, 2004.
37. E. Todorov, T. Erez, and Y. Tassa, "Mujoco: A physics engine for model-based control", in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2012, pp. 5026-5033.
38. D. Kingma and J. Ba, "Adam: A method for stochastic optimization", ICLR, 2015.
39. J. Kober and J. Peters, "Learning motor primitives for robotics", in ICRA, 2009.
40. R. Tedrake, T. W. Zhang, and H. S. Seung, "Learning to walk in 20 minutes."
41. S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift", ICML, 2015.
42. A. Rusu, S. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell, "Policy distillation", in ICLR, 2016.