Individual Q-learning in normal form games

SIAM Journal on Control and Optimization

Volumn 44, Issue 2, 2006, Pages 495-514

  Leslie, David S ^a   Collins, E J ^b  

^a UNIVERSITY OF NEW SOUTH WALES   (Australia)

^b UNIVERSITY OF BRISTOL   (United Kingdom)

Author keywords

Multi agent learning; Normal form games; Player dependent learning rates; Reinforcement learning; Stochastic approximation

Indexed keywords

ALGORITHMS; GAME THEORY; ITERATIVE METHODS; PROBLEM SOLVING;

MULTI-AGENT LEARNING; NORMAL FORM GAMES; PLAYER-DEPENDENT LEARNING RATES; REINFORCEMENT LEARNING; STOCHASTIC APPROXIMATION;

LEARNING SYSTEMS;

EID: 33645029191     PISSN: 03630129     EISSN: None     Source Type: Journal    
DOI: 10.1137/S0363012903437976     Document Type: Article

References (38)

1. P. AUER, N. CESA-BIANCHI, Y. FREUND, AND R. E. SCHAPIRE (1995), Gambling in a rigged casino: The adversarial multi-armed bandit problem, in 36th Annual Symposium on Foundations of Computer Science, IEEE Computer Society Press, Los Alamitos, CA, pp. 322-331.
2. R. J. AUMANN (1974), Subjectivity and correlation in randomized strategies, J. Math. Econom., 1, pp. 67-96.
3. A. BAÑDS (1968), On pseudo-games, Ann. Math. Statist., 39, pp. 1932-1945.
4. M. BENAÏM (1999), Dynamics of stochastic approximation algorithms, in Le Séminaire de Probabilités, Lecture Notes in Math. 1709, Springer-Verlag, Berlin, pp. 1-68.
5. M. BENAÏM AND M. W. HIRSCH (1999), Mixed equilibria and dynamical systems arising from fictitious play in perturbed games, Games Econom. Behav., 29, pp. 36-72.
6. U. BERGER (2005), Fictitious play inlxn games, J. Econom. Theory, 120, pp. 139-154.
7. T. BÖRGERS AND R. SARIN (1997), Learning through reinforcement and replicator dynamics, J. Econom. Theory, 77, pp. 1-14.
8. V. S. BORKAR (2001), Reinforcement learning in Markovian evolutionary games, available at http://www.tcs.tifr.res.in/~borkar/game.ps.
9. M. BOWLING AND M. VELOSO (2002), Multiagent learning using a variable learning rate, Artificial Intelligence, 136, pp. 215-250.
10. J. A. BOYAN AND M. L. LITTMAN (1994), Packet routing in dynamically changing networks: A reinforcement learning approach, in Advances in Neural Information Processing Systems, Vol. 6, J. D. Cowan, G. Tesauro, and J. Alspector, eds., Morgan Kaufmann, San Francisco, pp. 671-678.
11. G. W. BROWN (1951), Iterative solution of games by fictitious play, in Activity Analysis of Production and Allocation, T. C. Koopmans, ed., John Wiley &; Sons, New York, pp. 374-376.
12. H.-F. CHEN AND Y.-M. ZHU (1986), Stochastic approximation procedures with randomly varying truncations, Sci. China Ser. A, 29, pp. 914-926.
13. S. COWAN (1992), Dynamical Systems Arising from Game Theory, Ph.D. thesis, University of California, Berkeley.
14. R. H. CRITES AND A. G. BARTO (1998), Elevator group control using multiple reinforcement learning agents, Machine Learning, 33, pp. 235-262.
15. D. P. FOSTER AND H. P. YOUNG (2003), Learning, hypothesis testing, and Nash equilibrium, Games Econom. Behav., 45, pp. 73-96.
16. D. FUDENBERG AND D. K. LEVINE (1998), The Theory of Learning in Games, MIT Press, Cambridge, MA.
17. D. FUDENBERG AND J. TIROLE (1991), Game Theory, MIT Press, Cambridge, MA.
18. P. W. GLIMCHER, M. C. DORRIS, AND H. M. BAYER (2005), Physiological utility theory and the neuroeconomics of choice, Games Econom. Behav., to appear.
19. S. GOVINDAN, P. J. RENY, AND A. J. ROBSON (2003), A short proof of Harsanyi's purification theorem, Games Econom. Behav., 45, pp. 369-374.
20. J. HANNAN (1957), Approximation to Bayes risk in repeated play, in Contributions to the Theory of Games, Vol. 3, Ann. of Math. Stud. 39, M. Drescher, A. W. Tucker, and P. Wolfe, eds., Princeton University Press, Princeton, NJ, pp. 97-139.
21. J. C. HARSANYI (1973), Games with randomly disturbed payoffs: A new rationale for mixed-strategy equilibrium points, Internat. J. Game Theory, 2, pp. 1-23.
22. S. HART AND A. MAS-COLELL (2000), A simple adaptive procedure leading to correlated equilibrium, Econometrica, 68, pp. 1127-1150.
23. S. HART AND A. MAS-COLELL (2001), A reinforcement procedure leading to correlated equilibrium, in Economic Essays: A Festschrift for Werner Hildenbrand, W. N. G. Debreu and W. Trockel, eds., Springer-Verlag, Berlin, pp. 181-200.
24. S. HART AND A. MAS-COLELL (2003), Uncoupled dynamics cannot lead to Nash equilibrium, Amer. Econom. Rev., 93, pp. 1830-1836.
25. J. HOFBAUER AND E. HOPKINS (2005), Learning in perturbed asymmetric games, Games Econom. Behav., 52, pp. 133-152.
26. J. S. JORDAN (1993), Three problems in learning mixed strategy equilibria, Games Econom. Behav., 5, pp. 368-386.
27. D. KOLLER AND B. MILCH (2003), Multi-agent influence diagrams for representing and solving games, Games Econom. Behav., 45, pp. 181-221.
28. H. J. KUSHNER AND G. G. YIN (1997), Stochastic Approximation Algorithms and Applications, Appl. Math. 35, Springer-Verlag, New York.
29. D. S. LESLIE AND E. J. COLLINS (2003), Convergent multiple-timescales reinforcement learning algorithms in normal form games, Ann. Appl. Probab., 13, pp. 1231-1251.
30. M. L. LITTMAN, M. KEARNS, AND S. SINGH (2001), An efficient, exact algorithm for solving tree-structured graphical games, in Advances in Neural Information Processing Systems, Vol. 14, T. G. Dietterich, S. Becker, and Z. Ghahramani, eds., MIT Press, Cambridge, MA.
31. J. MAYNARD SMITH (1982), Evolution and the Theory of Games, Cambridge University Press, Cambridge, UK.
32. N. MEGIDDO (1980), On repeated games with incomplete information played by non-Bayesian players, Internat. J. Game Theory, 9, pp. 157-167.
33. J. NASH (1950), Equilibrium points in n-person games, Proc. Natl. Acad. Sci. USA, 36, pp. 48-49.
34. R. PEMANTLE (1990), Nonconvergence to unstable points in urn models and stochastic approximations, Ann. Probab., 18, pp. 698-712.
35. A. E. ROTH AND I. EREV (1995), Learning in extensive-form games: Experimental data and simple dynamic models in the intermediate term, Games Econom. Behav., 8, pp. 164-212.
36. L. S. SHAPLEY (1964), Some topics in two person games, in Advances in Game Theory, M. Drescher, L. S. Shapley, and A. W. Tucker, eds., Princeton University Press, Princeton, NJ, pp. 1-28.
37. S. SINGH AND D. BERTSEKAS (1997), Reinforcement learning for dynamic channel allocation in cellular telephone systems, in Advances in Neural Information Processing Systems, Vol. 9, M. C. Mozer, M. I. Jordan, and T. Petsche, eds., MIT Press, Cambridge, MA, p. 974.
38. R. S. SUTTON AND A. G. BARTO (1998), Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA.