Online learning of shaping rewards in reinforcement learning

Neural Networks

Volumn 23, Issue 4, 2010, Pages 541-550

  Grześ, Marek ^a   Kudenko, Daniel ^a  

^a UNIVERSITY OF YORK   (United Kingdom)

Author keywords

Learning heuristic; Potential based reward shaping; Reinforcement learning

Indexed keywords

BACKGROUND KNOWLEDGE; CONVERGENCE RATES; DISCRETISATION; FREE SPACE; LEARNING AGENTS; LEARNING HEURISTIC; MODEL FREE; MODEL-BASED; MULTI-GRID; NOVEL ALGORITHM; ONLINE LEARNING; POTENTIAL FUNCTION; REINFORCEMENT LEARNING AGENT; TEMPORAL DIFFERENCE LEARNING;

ALGORITHMS; INTELLIGENT AGENTS; REINFORCEMENT LEARNING;

REINFORCEMENT;

ALGORITHM; ARTICLE; KNOWLEDGE; MATHEMATICAL COMPUTING; MATHEMATICAL MODEL; PRIORITY JOURNAL; PROBABILITY; PROBLEM SOLVING; REINFORCEMENT; REWARD; SIMULATION;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; DECISION SUPPORT TECHNIQUES; MARKOV CHAINS; MODELS, NEUROLOGICAL; REINFORCEMENT SCHEDULE; REWARD;

EID: 77950298151     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neunet.2010.01.001     Document Type: Article

References (40)

1. Anderson, C., & Crawford-Hines, S. (1994). Multigrid Q-learning. Technical report CS-94-121, Colorado State University.
2. Asmuth, J., Littman, M. L., & Zinkov, R. (2008). Potential-based shaping in model-based reinforcement learning. In Proceedings of AAAI conference on artificial intelligence.
3. Bertsekas D.P. Dynamic programming and optimal control (2 Vol Set) 2007, Athena Scientific. 3rd ed.
4. Bertsekas D.P., Tsitsiklis J.N. Neuro-dynamic programming 1996, Athena Scientific.
5. Bishop C.M. Neural networks for pattern recognition 1996, Oxford University Press.
6. Böhm, N., Kókai, G., & Mandl, S. (2005). An evolutionary approach to tetris. In Proceedings of the sixth metaheuristics international conference.
7. Boutilier C., Dean T., Hanks S. Decision-theoretic planning: Structural assumptions and computational leverage. Journal of Artificial Intelligence Research 1999, 11:1-94.
8. Brafman R.I., Tennenholtz M. R-max - A general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research 2002, 213-231.
9. Chow C.S., Tsitsiklis J.N. An optimal one-way multigrid algorithm for discrete-time stochastic control. IEEE Transactions on Automatic Control 1991, 36(8):898-914.
10. Cohen P.R. Empirical methods for artificial intelligence 1995, MIT Press.
11. Dayan, P., & Hinton, G. E. (1993). Feudal reinforcement learning. In Proceedings of advances in neural information processing systems.
12. Dietterich T.G. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research 2000, 13:227-303.
13. Epshteyn, A., & DeJong, G. (2006). Qualitative reinforcement learning. In Proceedings of the 23rd international conference on machine learning (pp. 305-312).
14. Ghallab M., Nau D., Traverso P. Automated planning, theory and practice 2004, Elsevier, Morgan Kaufmann Publishers.
15. Grzes M., Kudenko D. Multigrid reinforcement learning with reward shaping. LNCS 2008, Springer-Verlag.
16. Grzes M., Kudenko D. Plan-based reward shaping for reinforcement learning. Proceedings of the 4th IEEE international conference on intelligent systems 2008, 22-29. IEEE.
17. Gullapalli, V., & Barto, A. G. (1992). Shaping as a method for accelerating reinforcement learning. In Proceedings of the 1992 IEEE international symposium on intelligent control (pp. 554-559).
18. Kaelbling, L. P. (1993). Hierarchical learning in stochastic domains: Preliminary results. In Proceedings of international conference on machine learning (pp. 167-173).
19. Kearns M., Singh S. Near-optimal reinforcement learning in polynomial time. Machine Learning 2002, 209-232.
20. Konidaris, G., & Barto, A. (2006). Autonomous shaping: Knowledge transfer in reinforcement learning. In The 23th international conference on machine learning.
21. Marthi, B. (2007). Automatic shaping and decomposition of reward functions. In Proceedings of the 24th international conference on machine learning (pp. 601-608).
22. Mataric, M. J. (1994). Reward functions for accelerated learning. In Proceedings of the 11th international conference on machine learning (pp. 181-189).
23. Mitchell T.M. Machine learning 1997, McGraw-Hill.
24. Moore, A., Baird, L., & Kaelbling, L. P. (1999). Multi-value-functions: Efficient automatic action hierarchies for multiple goal MDPs. In Proceedings of the international joint conference on artificial intelligence (pp. 1316-1323).
25. Ng, A. Y., Harada, D., & Russell, S. J. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the 16th international conference on machine learning (pp. 278-287).
26. Ng, A. Y., & Jordan, M. (2000). PEGASUS: A policy search method for large MDPs and POMDPs. In Proceedings of uncertainty in artificial intelligence (pp. 406-415).
27. Parr, R., & Russell, S. (1997). Reinforcement learning with hierarchies of machines. In Proceedings of advances in neural information processing systems. Vol. 10.
28. Puterman M.L. Markov decision processes: Discrete stochastic dynamic programming 1994, John Wiley & Sons, Inc, New York, NY, USA.
29. Randløv, J., & Alstrom, P. (1998). Learning to drive a bicycle using reinforcement learning and shaping. In Proceedings of the 15th international conference on machine learning (pp. 463-471).
30. Rayner, D. C., Davison, K., Bulitko, V., Anderson, K., & Lu, J. (2007). Real-time heuristic search with a priority queue. In Proceedings of the 2007 international joint conference on artificial intelligence (pp. 2372-2377).
31. Russell S.J., Norvig P. Artificial intelligence: A modern approach 2002, Prentice Hall. 2nd ed.
32. Stone, P., & Veloso, M. (2000). Layered learning. In Proceedings of the 11th European conference on machine learning.
33. Strehl, A. L., Li, L., & Littman, M. L. (2006). Pac reinforcement learning bounds for rtdp and rand-rtdp. In Proceedings of AAAI workshop on learning for search.
34. Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proceedings of the 7th international conference on machine learning (pp. 216-224).
35. Sutton R.S., Barto A.G. Reinforcement learning: An introduction 1998, MIT Press.
36. Sutton R.S., Precup D., Singh S.P. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence 1999, 112(1-2):181-211.
37. Taylor, M. E., & Stone, P. (2005). Behavior transfer for value-function-based reinforcement learning. In Proceedings of the 4th international joint conference on autonomous agents and multiagent systems (pp. 53-59).
38. Tesauro G.J. TD-gammon, a self-teaching backgammon program, achieves master-level play. Neural Computation 1994, 6(2):215-219.
39. Wiewiora E. Potential-based shaping and q-value initialisation are equivalent. Journal of Artificial Intelligence Research 2003, 19:205-208.
40. Wingate D., Seppi K.D. Prioritization methods for accelerating MDP solvers. Journal of Machine Learning Research 2005, 6:851-881.