Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation

Advances in Neural Information Processing Systems

Volumn , Issue , 2016, Pages 3682-3690

  Kulkarni, Tejas D ^a   Narasimhan, Karthik R ^b   Saeedi, Ardavan ^b   Tenenbaum, Joshua B ^c  

^a DEEPMIND   (United Kingdom)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DEEP LEARNING; LEARNING ALGORITHMS; STOCHASTIC SYSTEMS;

DECISION PROCESS; DELAYED FEEDBACK; GOAL SPECIFICATIONS; INTRINSIC BEHAVIOR; INTRINSIC MOTIVATION; STOCHASTIC TRANSITIONS; TEMPORAL ABSTRACTION; VALUE FUNCTIONS;

REINFORCEMENT LEARNING;

EID: 85019246453     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (35)

1. A. Baranes and P.-Y. Oudeyer. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1): 49-73, 2013.
2. A. G. Barto and S. Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13(4): 341-379, 2003.
3. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2012.
4. L. C. Cobo, C. L. Isbell, and A. L. Thomaz. Object focused q-learning for autonomous agents. In Proceedings of AAMAS, pages 1061-1068, 2013.
5. P. Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4): 613-624, 1993.
6. T. G. Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. J. Artif. Intell. Res.(JAIR), 13: 227-303, 2000.
7. C. Diuk, A. Cohen, and M. L. Littman. An object-oriented representation for efficient reinforcement learning. In Proceedings of the International Conference on Machine learning, pages 240-247, 2008.
8. S. Eslami, N. Heess, T. Weber, Y. Tassa, K. Kavukcuoglu, and G. E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. arXiv preprint arXiv: 1603.08575, 2016.
9. M. Frank, J. Leitner, M. Stollenga, A. Förster, and J. Schmidhuber. Curiosity driven reinforcement learning for motion planning on humanoids. Intrinsic motivations and open-ended development in animals, humans, and robots, page 245, 2015.
10. S. Goel and M. Huber. Subgoal discovery for hierarchical reinforcement learning using learned policies. In FLAIRS conference, pages 346-350, 2003.
11. C. Guestrin, D. Koller, C. Gearhart, and N. Kanodia. Generalizing plans to new environments in relational mdps. In Proceedings of International Joint conference on Artificial Intelligence, pages 1003-1010, 2003.
12. C. Guestrin, D. Koller, R. Parr, and S. Venkataraman. Efficient solution algorithms for factored mdps. Journal of Artificial Intelligence Research, pages 399-468, 2003.
13. N. Hernandez-Gardiol and S. Mahadevan. Hierarchical memory-based reinforcement learning. In Advances in Neural Information Processing Systems, pages 1047-1053, 2001.
14. J. Koutník, J. Schmidhuber, and F. Gomez. Evolving deep unsupervised convolutional networks for vision-based reinforcement learning. In Proceedings of the 2014 conference on Genetic and evolutionary computation, pages 541-548. ACM, 2014.
15. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. arXiv preprint arXiv: 1602.01783, 2016.
16. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, et al. Human-level control through deep reinforcement learning. Nature, 518(7540): 529-533, 2015.
17. S. Mohamed and D. J. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems, pages 2116-2124, 2015.
18. A. Nair, P. Srinivasan, S. Blackwell, C. Alcicek, R. Fearon, A. De Maria, V. Panneershelvam, et al. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv: 1507.04296, 2015.
19. I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. arXiv preprint arXiv: 1602.04621, 2016.
20. P.-Y. Oudeyer and F. Kaplan. What is intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, 1: 6, 2009.
21. T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 1312-1320, 2015.
22. J. Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). Autonomous Mental Development, IEEE Transactions on, 2(3): 230-247, 2010.
23. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587): 484-489, 2016.
24. Ö. Şimşek, A. Wolfe, and A. Barto. Identifying useful subgoals in reinforcement learning by local graph partitioning. In Proceedings of the International conference on Machine learning, pages 816-823, 2005.
25. S. Singh, R. L. Lewis, and A. G. Barto. Where do rewards come from. In Proceedings of the annual conference of the cognitive science society, pages 2601-2606, 2009.
26. S. Singh, R. L. Lewis, A. G. Barto, and J. Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective. Autonomous Mental Development, IEEE Transactions on, 2(2): 70-82, 2010.
27. S. P. Singh, A. G. Barto, and N. Chentanez. Intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pages 1281-1288, 2004.
28. J. Sorg and S. Singh. Linear options. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, pages 31-38, Richland, SC, 2010.
29. E. S. Spelke and K. D. Kinzler. Core knowledge. Developmental science, 10(1): 89-96, 2007.
30. K. L. Stachenfeld, M. Botvinick, and S. J. Gershman. Design principles of the hippocampal cognitive map. In Advances in neural information processing systems, pages 2528-2536, 2014.
31. B. C. Stadie, S. Levine, and P. Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv: 1507.00814, 2015.
32. R. S. Sutton and A. G. Barto. Introduction to reinforcement learning. MIT Press Cambridge, 1998.
33. R. S. Sutton, J. Modayil, M. Delp, T. Degris, P. M. Pilarski, A. White, and D. Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems, pages 761-768, 2011.
34. R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1): 181-211, 1999.
35. C. Szepesvari, R. S. Sutton, J. Modayil, S. Bhatnagar, et al. Universal option models. In Advances in Neural Information Processing Systems, pages 990-998, 2014.