An empirical study of potential-based reward shaping and advice in complex, multi-agent systems

Advances in Complex Systems

Volumn 14, Issue 2, 2011, Pages 251-278

  Devlin, Sam ^a   Kudenko, Daniel ^a   Grze, Marek ^b  

^a UNIVERSITY OF YORK   (United Kingdom)

^b UNIVERSITY OF WATERLOO   (Canada)

Author keywords

multi agent; Reinforcement learning; reward shaping

Indexed keywords

EID: 79955403826     PISSN: 02195259     EISSN: None     Source Type: Journal    
DOI: 10.1142/S0219525911002998     Document Type: Conference Paper

References (37)

1. Babes, M., de Cote, E. and Littman, M., Social reward shaping in the prisoner's dilemma, in Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, Vol. 3 (2008), pp. 1389-1392.
2. Balch, T., Learning roles: Behavioral diversity in robot teams, in AAAI Workshop on Multiagent Learning (1997).
3. Bertsekas, D. P., Dynamic Programming and Optimal Control (2 Vol Set) (Athena Scientific, 3rd edn., 2007).
4. Binmore, K., Fun and Games - A Text on Game Theory (D. C. Heath & Co., 1991).
5. Busoniu, L., Babuska, R. and De Schutter, B., A comprehensive survey of multi-agent reinforcement learning, IEEE Trans. Syst. Man Cyb. C. 38 (2008) 156.
6. Claus, C. and Boutilier, C., The dynamics of reinforcement learning in cooperative multiagent systems, in Proceedings of the National Conference on Artificial Intelligence (1998), pp. 746-752.
7. Devlin, S., Grzés, M. and Kudenko, D., Reinforcement learning in robocup keepaway with partial observability, in IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, 2009. WI-IAT'09 (2009).
8. Devlin, S. and Kudenko, D., Theoretical considerations of potential-based reward shaping for multi-agent systems, in Proceedings of The Tenth Annual International Conference on Autonomous Agents and Multiagent Systems (AAMAS) (2011).
9. Eric Wiewiora, G. C. and Elkan, C., Principled methods for advising reinforcement learning agents, in Proceedings of the Twentieth International Conference on Machine Learning (2003).
10. Fudenberg, D. and Tirole, J., Game Theory (MIT Press, Cambridge, MA, 1991).
11. Grzés, M. and Kudenko, D., Multigrid reinforcement learning with reward shaping, Artificial Neural Networks-ICANN 2008 (2008), pp. 357-366.
12. Grzés, M. and Kudenko, D., Plan-based reward shaping for reinforcement learning, in Proceedings of the 4th IEEE International Conference on Intelligent Systems (IS'08) (IEEE, 2008), pp. 22-29.
13. Hu, J. andWellman, M., Nash Q-learning for general-sum stochastic games, J. Mach. Learn. Res. 4 (2003) 1039-1069.
14. Iscen, A. and Erogul, U., A new perspective to the keepaway soccer: the takers, in Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, Vol. 3 (2008), pp. 1341-1344.
15. Kalyanakrishnan, S. and Stone, P., Learning complementary multiagent behaviors: A case study, in RoboCup 2009: Robot Soccer World Cup XIII, eds. Baltes, J., Lagoudakis, M., Naruse, T. and Ghidary, S., Lecture Notes in Computer Science, Vol. 5949 (Springer Berlin/Heidelberg, 2010), pp. 153-165.
16. Maclin, R. and Shavlik, J., Creating advice-taking reinforcement learners, Lect. Notes Artif. Int. (1996) 251-281. (Pubitemid 126724368)
17. Marthi, B., Automatic shaping and decomposition of reward functions, in Proceedings of the 24th International Conference on Machine Learning (ACM, 2007), p. 608.
18. Mihaylov, M., Tuyls, K. and Noẃe, A., Decentralized learning in wireless sensor networks, Adaptive and Learning Agents (2009), pp. 60-73.
19. Min, H., Zeng, J., Chen, J. and Zhu, J., A Study of reinforcement learning in a new multiagent domain, in IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, 2008. WI-IAT'08, Vol. 2 (2008).
20. Nash, J., Non-cooperative games, Ann. Math. 54 (1951) 286-295.
21. Ng, A. Y., Harada, D. and Russell, S. J., Policy invariance under reward transformations: Theory and application to reward shaping, in Proceedings of the 16th International Conference on Machine Learning (1999), pp. 278-287.
22. Peters, J., Vijayakumar, S. and Schaal, S., Reinforcement learning for humanoid robotics, in Proceedings of Humanoids2003, Third IEEE-RAS International Conference on Humanoid Robots (2003).
23. Puterman, M. L., Markov Decision Processes: Discrete Stochastic Dynamic Programming (John Wiley & Sons, Inc., New York, NY, USA, 1994).
24. Randløv, J. and Alstrom, P., Learning to drive a bicycle using reinforcement learning and shaping, in Proceedings of the 15th International Conference on Machine Learning (1998), pp. 463-471.
25. Shoham, Y., Powers, R. and Grenager, T., If multi-agent learning is the answer, what is the question? Artif. Intell. 171 (2007) 365-377. (Pubitemid 46802421)
26. Stone, P., Kuhlmann, G., Taylor, M. E. and Liu, Y., Keepaway soccer: From machine learning testbed to benchmark, in RoboCup-2005: Robot Soccer World Cup IX, eds. Noda, I., Jacoff, A., Bredenfeld, A. and Takahashi, Y., Vol. 4020 (Springer-Verlag, Berlin, 2006), pp. 93-105. (Pubitemid 350278772)
27. Stone, P., Sutton, R. S. and Kuhlmann, G., Reinforcement learning for RoboCupsoccer keepaway, Adapt. Behav. 13 (2005) 165-188.
28. Sutton, R., Generalization in reinforcement learning: Successful examples using sparse coarse coding, Adv. Neur. In. (1996) 1038-1044.
29. Sutton, R. S., Temporal Credit Assignment in Reinforcement Learning, Ph.D. Thesis, Department of Computer Science, University of Massachusetts, Amherst (1984).
30. Sutton, R. S. and Barto, A. G., Reinforcement Learning: An Introduction (MIT Press, 1998).
31. Tan, M., Multi-agent reinforcement learning: Independent vs. cooperative agents, in Proceedings of the Tenth International Conference on Machine Learning, Vol. 337 (1993).
32. Tumer, K. and Khani, N., Learning from actions not taken in multiagent systems, Adv. Complex Syst. 12 (2009) 455-473.
33. Tumer, K. and Wolpert, D., Collective intelligence and Braess' paradox, in Proceedings of the National Conference on Artificial Intelligence (2000), pp. 104-109.
34. Wang, X. and Sandholm, T., Reinforcement learning to play an optimal Nash equilibrium in team Markov games, Adv. Neur. In. (2003) 1603-1610.
35. Wiewiora, E., Potential-based shaping and Q-value initialization are equivalent, J. Artif. Intell. Res. 19 (2003) 205-208. (Pubitemid 41525920)
36. Wolpert, D. and Tumer, K., An introduction to collective intelligence, Technical Report cs.LG/9908014, NASA Ames Research Center (1999).
37. Wooldridge, M., An Introduction to MultiAgent Systems (John Wiley and Sons, 2002).