Transfer in reinforcement learning via shared features

Journal of Machine Learning Research

Volumn 13, Issue , 2012, Pages 1333-1371

  Konidaris, George ^a   Scheidwasser, Ilya ^b   Barto, Andrew G ^c  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b NORTHEASTERN UNIVERSITY   (United States)

^c University of Massachusetts Amherst   (United States)

Author keywords

Reinforcement learning; Shaping; Skills; Transfer

Indexed keywords

KNOWLEDGE TRANSFER; SHAPING; SKILL TRANSFER; SKILLS; TRANSFER;

KNOWLEDGE MANAGEMENT;

REINFORCEMENT LEARNING;

EID: 84862001711     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (55)

1. P. E. Agre and D. Chapman. Pengi: An implementation of a theory of activity. In Proceedings of the Sixth National Conference on Artificial Intelligence, pages 268-272, 1987.
2. B. Banerjee and P. Stone. General game learning using knowledge transfer. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, pages 672-677, 2007.
3. A. G. Barto and S. Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13:41-77, 2003. Special Issue on Reinforcement Learning.
4. D. S. Bernstein. Reusing old policies to accelerate learning on new MDPs. Technical Report UM-CS-1999-026, Department of Computer Science, University of Massachusetts at Amherst, April 1999.
5. J. Boyan and A. W. Moore. Learning evaluation functions to improve optimization by local search. Journal of Machine Learning Research, 1:77-112, 2000.
6. T. Croonenborghs, K. Driessens, and M. Bruynooghe. Learning relational options for inductive transfer in relational reinforcement learning. In Proceedings of the Seventeenth International Conference on Inductive Logic Programming, pages 88-97, 2007.
7. T. G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research, 13:227-303, 2000. (Pubitemid 33682087)
8. B. L. Digney. Learning hierarchical control structures for multiple tasks and changing environments. In R. Pfeifer, B. Blumberg, J. Meyer, and S. W. Wilson, editors, From Animals to Animats 5: Proceedings of the Fifth International Conference on Simulation of Adaptive Behavior, Zurich, Switzerland, August 1998. MIT Press.
9. M. Dorigo and M. Colombetti. Robot Shaping: An Experiment in Behavior Engineering. MIT Press/Bradford Books, 1998.
10. K. Ferguson and S. Mahadevan. Proto-transfer learning in Markov Decision Processes using spectral methods. In Proceedings of the ICML Workshop on Structural Knowledge Transfer for Machine Learning, Pittsburgh, June 2006.
11. K. Ferguson and S. Mahadevan. Proto-transfer learning in Markov Decision Processes using spectral methods. Technical Report TR-08-23, University of Massachusetts Amherst, 2008.
12. F. Fernández and M. Veloso. Probabilistic policy reuse in a reinforcement learning agent. In Proceedings of the 5th International Joint Conference on Autonomous Agents and Multiagent Systems, pages 720-727, 2006. (Pubitemid 46609543)
13. E. Ferrante, A. Lazaric, and M. Restelli. Transfer of task representation in reinforcement learning using policy-based proto-value functions (short paper). In Proceedings of the Seventh International Conference on Autonomous Agents and Multiagent Systems, pages 1329-1332, 2008.
14. L. Frommberger. Learning to behave in space: A qualitative spatial representation for robot navigation with reinforcement learning. International Journal on Artificial Intelligence Tools, 17(3):465-482, 2008.
15. A. Guazzelli, F. J. Corbacho, M. Bota, and M. A. Arbib. Affordances, motivations, and the world graph theory. Adaptive Behavior, 6(3/4):433-471, 1998.
16. B. Hengst. Discovering hierarchy in reinforcement learning with HEXQ. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 243-250, 2002.
17. A. Jonsson and A. G. Barto. Automated state abstraction for options using the U-Tree algorithm. In Advances in Neural Information Processing Systems 13, pages 1054-1060, 2001.
18. A. Jonsson and A. G. Barto. A causal approach to hierarchical decomposition of factored MDPs. In Proceedings of the Twenty Second International Conference on Machine Learning, 2005.
19. G. D. Konidaris and G. M. Hayes. Estimating future reward in reinforcement learning animats using associative learning. In From Animals to Animats 8: Proceedings of the 8th International Conference on the Simulation of Adaptive Behavior, pages 297-304, July 2004.
20. A. Laud and G. De Jong. The influence of reward on the speed of reinforcement learning: an analysis of shaping. In Proceedings of the Twentieth International Conference on Machine Learning, pages 440-447, 2003.
21. A. Lazaric, M. Restelli, and A. Bonarini. Transfer of samples in batch reinforcement learning. In Proceedings of the Twenty-Fifth International Conference on Machine Learning, pages 544-551, 2008.
22. S. Mannor, I. Menache, A. Hoze, and U. Klein. Dynamic abstraction in reinforcement learning via clustering. In Proceedings of the Twenty First International Conference on Machine Learning, pages 560-567, 2004. (Pubitemid 40290853)
23. M. J. Matarić. Reinforcement learning in the multi-robot domain. Autonomous Robots, 4(1):73-83, 1997. (Pubitemid 127508276)
24. A. McGovern and A. G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 361-368, 2001.
25. N. Mehta, S. Ray, P. Tadepalli, and T. Dietterich. Automatic discovery and transfer of MAXQ hierarchies. In Proceedings of the Twenty Fifth International Conference on Machine Learning, 2008.
26. A. W. Moore and C. G. Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning, 13(1):103-130, 1993.
27. A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: theory and application to reward shaping. In Proceedings of the 16th International Conference on Machine Learning, pages 278-287, 1999.
28. R. Parr and S. Russell. Reinforcement learning with hierarchies of machines. In Advances in Neural Information Processing Systems 10, pages 1043-1049, 1997.
29. T. J. Perkins and D. Precup. Using options for knowledge transfer in reinforcement learning. Technical Report UM-CS-1999-034, Department of Computer Science, University of Massachusetts, Amherst, 1999.
30. M. Pickett and A. G. Barto. Policyblocks: An algorithm for creating useful macro-actions in reinforcement learning. In Proceedings of the Nineteenth International Conference of Machine Learning, pages 506-513, 2002.
31. D. Precup, R. S. Sutton, and S. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 759-766, 2000.
32. M. L. Puterman. Markov Decision Processes. Wiley, 1994.
33. J. Randløv and P. Alstrøm. Learning to drive a bicycle using reinforcement learning and shaping. In Proceedings of the 15th International Conference on Machine Learning, pages 463-471, 1998.
34. B. Ravindran and A. G. Barto. Relativized options: Choosing the right transformation. In Proceedings of the Twentieth International Conference on Machine Learning, pages 608-615, 2003a.
35. B. Ravindran and A. G. Barto. SMDP homomorphisms: An algebraic approach to abstraction in semi markov decision processes. In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, pages 1011-1016, 2003b.
36. O. Selfridge, R. S. Sutton, and A. G. Barto. Training and tracking in robotics. In Proceedings of the Ninth International Joint Conference on Artificial Intelligence, pages 670-672, 1985.
37. Ö. Şimşek and A. G. Barto. Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proceedings of the Twenty-First International Conference on Machine Learning, pages 751-758, 2004. (Pubitemid 40290877)
38. Ö. Şimşek, A. P. Wolfe, and A. G. Barto. Identifying useful subgoals in reinforcement learning by local graph partitioning. In Proceedings of the Twenty-Second International Conference on Machine Learning, 2005.
39. S. Singh, A. G. Barto, and N. Chentanez. Intrinsically motivated reinforcement learning. In Proceedings of the 18th Annual Conference on Neural Information Processing Systems, 2004.
40. B. F. Skinner. The Behavior of Organisms: An Experimental Analysis. Appleton-Century-Crofts, New York, 1938.
41. M. Snel and S. Whiteson. Multi-task evolutionary shaping without pre-specified representations. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1031-1038, 2010.
42. P. Stone, R. S. Sutton, and G. Kuhlmann. Reinforcement learning for robocup soccer keepaway. Adaptive Behavior, 13(3):165-188, 2005. (Pubitemid 41546119)
43. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998.
44. R. S. Sutton, D. Precup, and S. P. Singh. Intra-option learning about temporally abstract actions. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 556-564, 1998.
45. R. S. Sutton, D. Precup, and S. P. Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2):181-211, 1999. (Pubitemid 32079890)
46. M. E. Taylor and P. Stone. Value functions for RL-based behavior transfer: a comparative study. In Proceedings of the Twentieth National Conference on Artificial Intelligence, 2005.
47. M. E. Taylor, P. Stone, and Y. Liu. Transfer learning via inter-task mappings for temporal difference learning. Journal of Machine Learning Research, 8:2125-2167, 2007. (Pubitemid 47510988)
48. M. E. Taylor, G. Kuhlmann, and P. Stone. Autonomous transfer for reinforcement learning. In Proceedings of the Seventh International Conference on Autonomous Agents and Multiagent Systems, 2008.
49. S. Thrun and A. Schwartz. Finding structure in reinforcement learning. In Advances in Neural Information Processing Systems, volume 7, pages 385-392. The MIT Press, 1995.
50. L. Torrey, J. Shavlik, T. Walker, and R. Maclin. Skill acquisition via transfer learning and advice taking. In Proceedings of the Seventeenth European Conference on Machine Learning, pages 425-436, 2006. (Pubitemid 44618851)
51. J. Weng, J. McClelland, A. Pentland, O. Sporns, I. Stockman, M. Sur, and E. Thelen. Autonomous mental development by robots and animals. Science, 291(5504):599-600, 2000.
52. E. Wiewiora. Potential-based shaping and Q-value initialization are equivalent. Journal of Artificial Intelligence Research, 19:205-208, 2003. (Pubitemid 41525920)
53. E. Wiewiora, G. Cottrell, and C. Elkan. Principled methods for advising reinforcement learning agents. In Proceedings of the Twentieth International Conference on Machine Learning, pages 792-799, 2003.
54. A. Wilson, A. Fern, S. Ray, and P. Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th International Conference on Machine Learning, pages 1015-1022, 2007.
55. W. Zhang and T. G. Dietterich. A reinforcement learning approach to job-shop scheduling. In Proceedings of the Fourteenth International Conference on Artificial Intelligence, pages 1114-1120, 1995.