Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning

32nd AAAI Conference on Artificial Intelligence, AAAI 2018

Volumn , Issue , 2018, Pages 3199-3206

  Henderson, Peter ^a   Chang, Wei Di ^a   Bacon, Pierre Luc ^a   Meger, David ^a   Pineau, Joelle ^a   Precup, Doina ^a  

^a MCGILL UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; INVERSE PROBLEMS;

COMPLEX PROBLEMS; CONTINUOUS CONTROL; INVERSE REINFORCEMENT LEARNING; JOINT REWARDS; LEARNING POLICY; POLICY OPTIONS; REWARD FUNCTION; TRANSFER LEARNING;

REINFORCEMENT LEARNING;

EID: 85060430951     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (32)

1. Abbeel, P., and Ng, A. Y. 2004. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-first International Conference on Machine Learning, 1-. New York, NY, USA: ACM.
2. Babes, M.; Marivate, V.; Subramanian, K.; and Littman, M. L. 2011. Apprenticeship learning about multiple intentions. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), 897-904.
3. Bacon, P.-L.; Harb, J.; and Precup, D. 2017. The option-critic architecture. In AAAI, 1726-1734.
4. Bengio, E.; Bacon, P.-L.; Pineau, J.; and Precup, D. 2015. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297.
5. Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; and Zaremba, W. 2016. OpenAI Gym.
6. Choi, J., and eung Kim, K. 2012. Nonparametric bayesian inverse reinforcement learning for multiple reward functions. In Pereira, F.; Burges, C. J. C.; Bottou, L.; and Wein-berger, K. Q., eds., Advances in Neural Information Processing Systems 25. Curran Associates, Inc. 305-313.
7. Christiano, P.; Shah, Z.; Mordatch, I.; Schneider, J.; Blackwell, T.; Tobin, J.; Abbeel, P.; and Zaremba, W. 2016. Transfer from simulation to real world through learning deep inverse dynamics model. arXiv preprint arXiv:1610.03518.
8. Daniel, C.; Neumann, G.; and Peters, J. R. 2012. Hierarchical relative entropy policy search. In International Conference on Artificial Intelligence and Statistics, 273-281.
9. Dayan, P., and Hinton, G. E. 1993. Feudal reinforcement learning. In Advances in neural information processing systems, 271-278.
10. Dietterich, T. G. 2000. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research 13:227-303.
11. Goodfellow, I. J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2672-2680.
12. Hausman, K.; Chebotar, Y.; Schaal, S.; Sukhatme, G.; and Lim, J. 2017. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. arXiv preprint arXiv:1705.10479.
13. Henderson, P.; Chang, W.-D.; Shkurti, F.; Hansen, J.; Meger, D.; and Dudek, G. 2017. Benchmark environments for multitask learning in continuous domains. ICML Lifelong Learning: A Reinforcement Learning Approach Workshop.
14. Ho, J., and Ermon, S. 2016. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, 4565-4573.
15. Jacobs, R. A.; Jordan, M. I.; Nowlan, S. J.; and Hinton, G. E. 1991. Adaptive mixtures of local experts. Neural computation 3(1):79-87.
16. Krishnan, S.; Garg, A.; Liaw, R.; Miller, L.; Pokorny, F. T.; and Goldberg, K. 2016. Hirl: Hierarchical inverse reinforcement learning for long-horizon tasks with delayed rewards. arXiv preprint arXiv:1604.06508.
17. Li, Y.; Song, J.; and Ermon, S. 2017. InfoGAIL: Interpretable imitation learning from visual demonstrations. arXiv preprint arXiv:1703.08840.
18. Liu, Y., and Yao, X. 2002. Learning and evolution by minimization of mutual information. In International Conference on Parallel Problem Solving from Nature, 495-504. Springer.
19. Merel, J.; Tassa, Y.; Srinivasan, S.; Lemmon, J.; Wang, Z.; Wayne, G.; and Heess, N. 2017. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201.
20. Mnih, V.; Badia, A. P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K. 2016. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, 1928-1937.
21. Ng, A. Y., and Russell, S. J. 2000. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, 663-670. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.
22. Schulman, J.; Levine, S.; Abbeel, P.; Jordan, M.; and Moritz, P. 2015. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), 1889-1897.
23. Schulman, J.; Moritz, P.; Levine, S.; Jordan, M.; and Abbeel, P. 2016. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR).
24. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
25. Sermanet, P.; Xu, K.; and Levine, S. 2016. Unsupervised perceptual rewards for imitation learning. arXiv preprint arXiv:1612.06699.
26. Shazeer, N.; Mirhoseini, A.; Maziarz, K.; Davis, A.; Le, Q.; Hinton, G.; and Dean, J. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.
27. Sutton, R. S.; McAllester, D. A.; Singh, S. P.; and Mansour, Y. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, 1057-1063.
28. Sutton, R. S.; Precup, D.; and Singh, S. 1999. between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence 112(1-2):181-211.
29. Todorov, E.; Erez, T.; and Tassa, Y. 2012. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, 5026-5033.
30. van Seijen, H.; Fatemi, M.; Romoff, J.; Laroche, R.; Barnes, T.; and Tsang, J. 2017. Hybrid reward architecture for reinforcement learning. arXiv preprint arXiv:1706.04208.
31. Wang, Z.; Merel, J.; Reed, S.; Wayne, G.; de Freitas, N.; and Heess, N. 2017. Robust imitation of diverse behaviors. arXiv preprint arXiv:1707.02747.
32. Ziebart, B. D.; Maas, A.; Bagnell, J. A.; and Dey, A. K. 2008. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3, 1433-1438. AAAI Press.