Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

Proceedings - IEEE International Conference on Robotics and Automation

Volumn , Issue , 2018, Pages 5628-5635

  Zhang, Tianhao ^a,b   McCarthy, Zoe ^a   Jowl, Owen ^a   Lee, Dennis ^a   Chen, Xi ^a,b   Goldberg, Ken ^a   Abbeel, Pieter ^a,b,c,d  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b Embodied Intelligence   (United States)

^c OpenAI LLC   (United States)

^d NONE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPLEX NETWORKS; E-LEARNING; PIXELS; ROBOTICS; VIRTUAL REALITY;

COMPLEX TASK; HAND TRACKING; IMITATION LEARNING; MANIPULATION TASK; ROBOT SKILLS; VIRTUAL-REALITY HEADSETS; VISUOMOTORS;

DEEP NEURAL NETWORKS;

EID: 85059374758     PISSN: 10504729     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICRA.2018.8461249     Document Type: Conference Paper

References (51)

1. S. Schaal, "Is imitation learning the route to humanoid robots?" Trends in cognitive sciences, vol. 3, no. 6, pp. 233-242, 1999.
2. S. Calinon, Robot programming by demonstration. EPFL Press, 2009.
3. B. D. Argall, S. Chernova, M. Veloso, and B. Browning, "A survey of robot learning from demonstration," Robotics and autonomous systems, vol. 57, no. 5, pp. 469-483, 2009.
4. D. A. Pomerleau, "Alvinn: An autonomous land vehicle in a neural network," in Advances in Neural Information Processing Systems, 1989, pp. 305-313.
5. M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al., "End to end learning for self-driving cars," arXiv preprint arXiv:1604.07316, 2016.
6. A. Giusti, J. Guzzi, D. C. Cires?an, F.-L. He, J. P. Rodríguez, F. Fontana, M. Faessler, C. Forster, J. Schmidhuber, G. Di Caro, et al., "A machine learning approach to visual perception of forest trails for mobile robots," IEEE Robotics and Automation Letters, vol. 1, no. 2, pp. 661-667, 2016.
7. P. Abbeel, A. Coates, and A. Y. Ng, "Autonomous helicopter aerobatics through apprenticeship learning," The International Journal of Robotics Research, 2010.
8. S. Calinon, F. D'halluin, E. L. Sauser, D. G. Caldwell, and A. G. Billard, "Learning and reproduction of gestures by imitation," IEEE Robotics & Automation Magazine, vol. 17, no. 2, pp. 44-54, 2010.
9. B. Akgun, M. Cakmak, K. Jiang, and A. L. Thomaz, "Keyframe-based learning from demonstration," International Journal of Social Robotics, vol. 4, no. 4, pp. 343-355, 2012.
10. J. Schulman, J. Ho, C. Lee, and P. Abbeel, "Learning from demonstrations through the use of non-rigid registration," in Proceedings of the 16th International Symposium on Robotics Research (ISRR), 2013.
11. A. Dragan, K. C. Lee, and S. Srinivasa, "Teleoperation with intelligent and customizable interfaces," Journal of Human-Robot Interaction, vol. 1, no. 3, 2013.
12. A. E. Bryson, Applied optimal control: optimization, estimation and control. CRC Press, 1975.
13. J. T. Betts, Practical methods for optimal control and estimation using nonlinear programming. SIAM, 2010.
14. M. Posa, C. Cantu, and R. Tedrake, "A direct method for trajectory optimization of rigid bodies through contact," The International Journal of Robotics Research, vol. 33, no. 1, pp. 69-81, 2014.
15. S. Levine and P. Abbeel, "Learning neural network policies with guided policy search under unknown dynamics," in Advances in Neural Information Processing Systems, 2014, pp. 1071-1079.
16. J. Peters and S. Schaal, "Natural actor-critic," Neurocomputing, vol. 71, no. 7, pp. 1180-1190, 2008.
17. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., "Human-level control through deep reinforcement learning," Nature, vol. 518, no. 7540, pp. 529-533, 2015.
18. J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, "Trust region policy optimization," in Proceedings of the 32nd International Conference on Machine Learning (ICML-15), 2015, pp. 1889-1897.
19. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, "Continuous control with deep reinforcement learning," arXiv preprint arXiv:1509.02971, 2015.
20. S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-end training of deep visuomotor policies," Journal of Machine Learning Research, vol. 17, no. 39, pp. 1-40, 2016.
21. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, "Asynchronous methods for deep reinforcement learning," in International Conference on Machine Learning, 2016, pp. 1928-1937.
22. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv:1707.06347, 2017.
23. M. Talamini, K. Campbell, and C. Stanfield, "Robotic gastrointestinal surgery: early experience and system description," Journal of laparoendoscopic & advanced surgical techniques, vol. 12, no. 4, pp. 225-232, 2002.
24. S. Ross, G. J. Gordon, and D. Bagnell, "A reduction of imitation learning and structured prediction to no-regret online learning." in AISTATS, vol. 1, no. 2, 2011, p. 6.
25. A. Y. Ng, S. J. Russell, et al., "Algorithms for inverse reinforcement learning." in Icml, 2000, pp. 663-670.
26. P. Abbeel and A. Ng, "Apprenticeship learning via inverse reinforcement learning," in International Conference on Machine Learning (ICML), 2004.
27. B. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, "Maximum entropy inverse reinforcement learning," in AAAI Conference on Artificial Intelligence, 2008.
28. S. Levine, Z. Popovic, and V. Koltun, "Nonlinear inverse reinforcement learning with Gaussian processes," in Advances in Neural Information Processing Systems (NIPS), 2011.
29. C. Finn, S. Levine, and P. Abbeel, "Guided cost learning: Deep inverse optimal control via policy optimization," in Proceedings of the 33rd International Conference on Machine Learning, vol. 48, 2016.
30. J. Ho and S. Ermon, "Generative adversarial imitation learning," in Advances in Neural Information Processing Systems, 2016, pp. 4565-4573.
31. A. Billard, Y. Epars, S. Calinon, S. Schaal, and G. Cheng, "Discovering optimal imitation strategies," Robotics and autonomous systems, vol. 47, no. 2, pp. 69-77, 2004.
32. S. Schaal, J. Peters, J. Nakanishi, and A. Ijspeert, "Learning movement primitives," Robotics Research, pp. 561-572, 2005.
33. P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal, "Learning and generalization of motor skills by learning from demonstration," in Robotics and Automation, 2009. ICRA'09. IEEE International Conference on. IEEE, 2009, pp. 763-768.
34. N. Ratliff, J. A. Bagnell, and S. S. Srinivasa, "Imitation learning for locomotion and manipulation," in Humanoid Robots, 2007 7th IEEE-RAS International Conference on. IEEE, 2007, pp. 392-397.
35. T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, A. Sendonaris, G. Dulac-Arnold, I. Osband, J. Agapiou, et al., "Learning from demonstrations for real world reinforcement learning," arXiv preprint arXiv:1704.03732, 2017.
36. S. Ross, N. Melik-Barkhudarov, K. S. Shankar, A. Wendel, D. Dey, J. A. Bagnell, and M. Hebert, "Learning monocular reactive uav control in cluttered natural environments," in Robotics and Automation (ICRA), 2013 IEEE International Conference on. IEEE, 2013, pp. 1765-1772.
37. R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine, "Visionbased multi-task manipulation for inexpensive robots using end-to-end learning from demonstration," arXiv preprint arXiv:1707.02920, 2017.
38. B. C. Stadie, P. Abbeel, and I. Sutskever, "Third-person imitation learning," arXiv preprint arXiv:1703.01703, 2017.
39. Y. Liu, A. Gupta, P. Abbeel, and S. Levine, "Imitation from observation: Learning to imitate behaviors from raw video via context translation," arXiv preprint arXiv:1707.03374, 2017.
40. C. Stanton, A. Bogdanovych, and E. Ratanasena, "Teleoperation of a humanoid robot using full-body motion capture, example movements, and machine learning," in Proc. Australasian Conference on Robotics and Automation, 2012.
41. L. Fritsche, F. Unverzag, J. Peters, and R. Calandra, "First-person teleoperation of a humanoid robot," in Humanoid Robots (Humanoids), 2015 IEEE-RAS 15th International Conference on. IEEE, 2015, pp. 997-1002.
42. J. I. Lipton, A. J. Fay, and D. Rus, "Baxter's homunculus: Virtual reality spaces for teleoperation in manufacturing," IEEE Robotics and Automation Letters, vol. 3, no. 1, pp. 179-186, 2018.
43. V. Kumar and E. Todorov, "Mujoco haptix: A virtual reality system for hand manipulation," in Humanoid Robots (Humanoids), 2015 IEEERAS 15th International Conference on. IEEE, 2015, pp. 657-663.
44. E. Rosen, D. Whitney, E. Phillips, G. Chien, J. Tompkin, G. Konidaris, and S. Tellex, "Communicating robot arm motion intent through mixed reality head-mounted displays," arXiv preprint arXiv:1708.03655, 2017.
45. X. Yan, M. Khansari, Y. Bai, J. Hsu, A. Pathak, A. Gupta, J. Davidson, and H. Lee, "Learning grasping interaction with geometry-aware 3d representations," arXiv preprint arXiv:1708.07303, 2017.
46. V. Kumar, A. Gupta, E. Todorov, and S. Levine, "Learning dexterous manipulation policies from experience and imitation," in ICRA, 2016.
47. A. Y. Ng, A. Coates, M. Diel, V. Ganapathi, J. Schulte, B. Tse, E. Berger, and E. Liang, "Autonomous inverted helicopter flight via reinforcement learning," in Experimental Robotics IX. Springer Berlin Heidelberg, 2006, pp. 363-372.
48. J. Peters and S. Schaal, "Reinforcement learning of motor skills with policy gradients," Neural networks, vol. 21, no. 4, pp. 682-697, 2008.
49. R. Tedrake, T. W. Zhang, and H. S. Seung, "Stochastic policy gradient reinforcement learning on a simple 3d biped," in Intelligent Robots and Systems, 2004.(IROS 2004). Proceedings. 2004 IEEE/RSJ International Conference on, vol. 3. IEEE, 2004, pp. 2849-2854.
50. A. Y. Ng and S. J. Russell, "Algorithms for inverse reinforcement learning." in Icml, 2000, pp. 663-670.
51. M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu, "Reinforcement learning with unsupervised auxiliary tasks," arXiv preprint arXiv:1611.05397, 2016.