An evolutionary dynamical analysis of multi-agent learning in iterated games

Autonomous Agents and Multi-Agent Systems

Volumn 12, Issue 1, 2006, Pages 115-153

  Tuyls, Karl ^a   T Hoen, Pieter Jan ^b   Vanschoenwinkel, Bram ^c  

^a HASSELT UNIVERSITY   (Belgium)

^b CWI   (Netherlands)

^c VRIJE UNIVERSITEIT BRUSSEL   (Belgium)

Author keywords

COllective INtelligence; Evolutionary Game Theory; Iterated games; Multi agent systems; Reinforcement learning

Indexed keywords

EID: 31344450384     PISSN: 13872532     EISSN: 15737454     Source Type: Journal    
DOI: 10.1007/s10458-005-3783-9     Document Type: Review

References (56)

1. B. Banerjee and J. Peng, "Adaptive policy gradient in multiagent learning," in Proceedings of the Third International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2003.
2. A. Barto and S. Mahadevan, "Recent advances in hierarchical reinforcement learning," Discrete-Event Syst. J. vol. 13, pp. 41-77, 2003.
3. A. L. C. Bazzan, A game-theoretic approach to coordination of traffic signal agents, PhD thesis, University of Karlsruhe, 1997.
4. R. Becker, S. Zilberstein, V. Lesser, and C. V. Goldman, "Transition independent decentralized Markov decision problem," in Proceedings of the Third International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2003.
5. T. Börgers and R. Sarin, "Reinforcement and replicator dynamics," J. Econ. Theory, vol. 77, no.1, pp. 1-14, 1997.
6. R. Boyd and P. J. Richerson, Culture and the Evolutionary Process, The University of Chicago Press, 1985.
7. R. R. Bush and F. Mosteller, "A Mathematical Model for Simple Learning," The Psychol. Rev. vol. 58, pp. 15-18, 1951.
8. R. R. Bush and F. Mosteller, Stochastic Models for Learning, Wiley: New York, 1955.
9. C. Claus and G. Boutilier, "The Dynamics of Reinforcement Learning in Cooperative Multi-Agent Systems," in Proceedings of the 15th International Conference on Artificial Intelligence, pp. 746-752, 1998.
10. J. G. Cross, "A stochastic learning model of economic behaviour," Quart. J. Econ., vol. 87, no.5, pp. 239-266, 1973.
11. C. M. Gintis, Game Theory Evolving, Princeton University Press, 2000.
12. T. Grenager, and R. Powers, and Y. Shoham, "Dispersion games: general definitions and some specific learning results," in Proceedings of the Eighteenth National Conference on Artificial Intelligence AAAI 02, 2002.
13. C. Guestrin, D. Koller, C. Gearhart, and N. Kanodia, "Generalizing plans to new environments in relational MDPs," in International Joint Conference on Artificial Intelligence (IJCAI-03), 2003.
14. M. W. Hrisch and S. Smale, Differential Equation, Dynamical Systems and Linear Algebra, Academic Press, Inc 1974.
15. J. Hofbauer and K. Sigmund, Evolutionary Games and Population Dynamics, Cambridge University Press, 1998.
16. J. Hu and M. P. Wellman, "Multiagent reinforcement learning in stochastic games," in Internal Report from the Laboratory for Information and Decision Systems and the Operation Research Center, 1999.
17. P. Huang and K. Sycara, "Multi-agent Learning in Extensive Games with complete information," in Proceedings of the Third International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2003.
18. C. Jafari, A. Greenwald, D. Gondek, and G. Ercal, "On no-regret learning fictitious play and nash equilibrium," in Proceedings of the Eighteenth International Conference on Machine Learning (ICML), Cambridge University Press, pp. 223-226, 2001.
19. H. Jung and M. Tambe, "Performance model for large scale multiagent systems" in Proceedings of the Third International Conference Autonomous Agents and Multiagent Systems (AAMAS), 2003.
20. L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Reinforcement Learning: A Survey," J. Artif. Intell. Res. vol. 4, pp. 237-285, 1996.
21. M. Lauer and M. Riedmiller, "An algorithm for distributed reinforcement learning in cooperative multi-agent systems," in Proc. 17th International Conf. on Machine Learning Morgan Kaufmann: San Francisco, CA, pp. 535-542, 2000.
22. M. L. Littman, "Markov games as a framework for multi-agent reinforcement learning," in Proceedings of the Eleventh International Conference on Machine Learning," Cambridge University Press, pp. 157-163, 1994.
23. J. Maynard Smith, Evolution and the Theory of the Games, Cambridge University Press, 1982.
24. J. Maynard Smith, and G. R. Price, "The logic of animal conflict," Nature, vol. 146, no. 2, pp. 15-18, 1973.
25. K. Narendra and M. Thathachar, "Learning automata: A survey," IEEE Trans. Syst. Man Cybernet, vol. 14, no.5, pp. 323-334, 1974.
26. K. Narendra and M. Thathachar, Learning Automata: An Introduction, Prentice-Hall, 1989.
27. A. Nowé, J. Parent, and K. Verbeeck, "Social agents playing a periodical policy," in Proceedings of the 12th European Conference on Machine Learning, Volume 2176 of Lecture Notes in Artificial Intelligence, Springer, pp. 382-393, 2001.
28. E. Nudelman, J. Wortman, K. Leyton-Brown, and Y. Shoham, "Run the GAMUT: A comprehensive approach to evaluating game-theoretic algorithms, algorithms," in Proceedings of the Fourth International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2004.
29. M. J. Osborne and A. Rubinstein, A Coruse in Game Theory, MIT Press, 1994.
30. J. Parent, K. Verbeeck, A. Nowé, K. Steenhaut, J. Lemeire, and E. Dirkx, "Adaptive load balancing of parallel applications with social reinforement learning on heterogeneous sysems," J. Sci. Program. 2004. to appear.
31. S. Phelps, S. Parsons, and P. McBurney, "An evolutionary game-theoretic comparison of two double-action market designs," in Workshop on Agent Medicated Electronic commerce VI: Theories for Engineering of Distributed Mechanisms and Systems (AMEC'04), Volume 2531 of Lecture Notes in Artificial Intelligence, Springer, pp. 109-118, 2004.
32. R. Powers and Y. Shoham, "New criteria and a new algorithm for learning in multi-agent system," in Proceedings of Eighteenth Annual Conference on Neural Information Processing Systems (NIPS), 2004.
33. F.V. Redondo, Game Theory and Economics, Cambridge University Press, 2001.
34. L. Samuelson, Evolutionary Games and Equilibrium Selection, MIT Press: Cambridge, MA, 1997.
35. T. D. Schneider, "Evolution of biological information," J. Nucl. Acid Res. vol. 28, no. 14, pp. 2794-2799, 2000.
36. D. Stauffer, Life Love and Death: Models of Biological Reproduction and Aging, Institute for Theoretical physics: Köln Euroland, 1999.
37. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. MIT Press: Cambridge MA, 1998.
38. P. J. 't Hoen and S. M. Bohte, "Collective INtelligence with sequence of actions," in 14th European conference on Machine Learning, Volume 2837 of Lecture Notes in Articifical Intelligence, Springer, 2003.
39. P. J. 't Hoen and S. M. Bohte, "Collective INtelligence with task assignment," in proceedings of the Workshop on Collectives and the Design of Complex Systems (CDOCS03), forthcoming.
40. Also available as Technical Rapport SEN-E0315, Lecture Notes in Artificial Intelligence, Springer, 2003.
41. P. J. 't Hoen and K. Tuyls, "Analyzing multi-agent reinforcement learning using evolutionary dynamics," in Proceedings of the 15th European Conference on Machine Learning (ECML), Lecture Notes in Artificial Intelligence, Springer, 2004.
42. P. S. Sastry and M. A. L. Thathacher, "Varieties of Learning Automata: An Overview," IEEE Trans. Sys. Man Cybernet, vol. 32, no. 6, pp. 323-334, 2002.
43. J. N. Tsitsiklis, "Asynchronous stochastic approximation and Q-learning," Machine Learn, vol. 16, pp. 185-202, 1994.
44. K. Tumer and D. Wolpert, "Collective INtelligence and Braess' Paradox," in Proceedings of the Sixteenth National Conference on Artificial Intelligence, Austin, pp. 104-109, August, 2000.
45. K. Tuyls, Learning in Multi-Agent Systems, An Evolutionary Game Theoretic Approach, Ph.D. dissertation, Computational Modeling Lab, Vrije Universiteit Brussel, Belgium, 2004.
46. K. Tuyls, D. Heytens, A. Nowé, and B. Manderick, "Extended Replicator Dynamics as a Key to Reinforcement Learning in Multi-Agent Systems," in Proceedings of the 14th European Conference on Machine Learning (ECML), Volume 2837, of Lecture Notes in Artificial Intelligence, Springer, 2003.
47. K. Tuyls, T. Lenaerts, K. Verbeeck, S. Maes and B. Manderick, "Towards a relation between learning agents and evolutionary dynamics", in Proceedings of the Belgian-Dutch Conference on Artificial Intelligence (BNAIC 2002), Cambridge University Press, pp. 223-226, 2002.
48. K. Tuyls, A. Nowe, T. Lenaerts, and B. Manderick, "An evolutionary game theoretic perspective on learning in multi-agent systems," in Synthese, Section Knowledge, Rationality and Action, Kluwer Academic Publishers, 2004, vol. 139, no. 2, pp. 297-330.
49. K. Tuyls, K. Verbeeck, T. Lenaerts, "A Selection-Mutation model for Q-learning in Multi-Agent Systems," in Proceedings of the Third Interational conference on Autonomous Agents and Multi-agent Systems (AAMAS), The ACM International Conference Proceedings Series, 2003.
50. K. Verbeeck, A. Nowé, and J. Parent, "Homo egualis reinforcement learning agents for load balancing," in Proceedings of the 1st NASA Workshop on Radical Agent Concepts, Volume 2564 of Lecture Notes in Artificial Intelligence, Springer, pp. 109-118, 2002.
51. J. von Neumann and O. Morgenstern, Theory of Games and Economic Behavior, Princeton University Press, 1944.
52. W. E. Walsh, R. Das, G. Tesauro, and J. O. Kephart, "Analyzing complex strategic interactions in multi-agent games," in Proceedings of the The Eighteenth National Conference on Artificial Intelligence (AAAI-02) Workshop on Game Theoretic, and Decision Theoretic Agents, Lecture Notes in Artificial Intelligence, Springer, pp. 109-118, 2002.
53. C. Watkins and P. Dayan, "Q-learning", Machine Learn., vol. 8, pp. 279-292, 1992.
54. J. W. Weibull, Evolutionary Game Theory, MIT Press, 1996.
55. David H. Wolpert, Kagan Turner, and Jeremy Frank, "Using Collective INtelligence to route internet traffic," in Advances in Neural Information Processing Systems-II, Denver, pp. 952-958, 1998.
56. David H. Wolpert, Kevin R. Wheler, and Kagan Turner, "General Principles of learning-based multi-agent systems", in Oren Etzioni and Jörg P. Müller and Jeffrey M. Bradshaw (ed.), Proceedings of the Third International Conference on Autonomous Agents (Agents'99), ACM Press: Seattle, WA, USA, pp. 77-83, 1999.