Nash Q-learning for general-sum stochastic games

Journal of Machine Learning Research

Volumn 4, Issue 6, 2004, Pages 1039-1069

  Hu, Junling ^a   Wellman, Michael P ^b  

^a Talkai Research   (United States)

^b UNIVERSITY OF MICHIGAN   (United States)

Author keywords

Multiagent Learning; Q learning; Reinforcement Learning

Indexed keywords

FUNCTIONS; GAME THEORY; MARKOV PROCESSES; MULTI AGENT SYSTEMS; PROBABILITY; ROBOTICS;

GRID GAMES; MULTIAGENT LEARNING; Q-LEARNING; REINFORCEMENT LEARNING;

LEARNING SYSTEMS;

EID: 4644369748     PISSN: 15324435     EISSN: None     Source Type: Journal    
DOI: 10.1162/1532443041827880     Document Type: Article

References (54)

1. Tucker Balch. Learning roles: Behavioral diversity in robot teams. In Sandip Sen, editor, Collected Papers from the AAAI-97 Workshop on Multiagent Learning. AAAI Press, 1997.
2. Cesar Bandera, Francisco J. Vico, Jose M. Bravo, Mance E. Harmon, and Leemon C. Baird. Residual Q-learning applied to visual attention. In Thirteenth International Conference on Machine Learning, pages 20-27, Bari, Italy, 1996.
3. Bikramjit Banerjee, Sandip Sen, and Jing Peng. Fast concurrent reinforcement learners. In Seventeenth International Joint Conference on Artificial Intelligence, pages 825-830, Seattle, 2001.
4. Craig Boutilier. Sequential optimality and coordination in multiagent systems. In Sixteenth International Joint Conference on Artificial Intelligence, pages 478-485, Stockholm, 1999.
5. Michael Bowling. Convergence problems of general-sum multiagent reinforcement learning. In Seventeenth International Conference on Machine Learning, pages 89-94, Stanford, 2000.
6. Michael Bowling and Manuela Veloso. Multiagent learning using a variable learning rate. Artificial Intelligence, 136:215-250, 2002.
7. Ronen I. Brafman and Moshe Tennenholtz. A near-optimal polynomial time algorithm for learning in certain classes of stochastic games. Artificial Intelligence, 121(1-2):31-47, 2000.
8. Ronen I. Brafman and Moshe Tennenholtz. A general polynomial time algorithm for near-optimal reinforcement learning. In Seventeenth International Joint Conference on Artificial Intelligence, pages 953-958, Seattle, 2001.
9. Georgios Chalkiadakis and Craig Boutilier. Coordination in multiagent reinforcement learning: a bayesian approach. In The Proceedings of the Second International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2003), pages 709-716, 2003.
10. Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. In Fifteenth National Conference on Artificial Intelligence, pages 746-752, Madison, WI, 1998.
11. Richard W. Cottle, J.-S. Pang, and R. E. Stone. The Linear Complementarity Problem. Academic Press, New York, 1992.
12. Edwin De Jong. Non-random exploration bonuses for online reinforcement learning. In Sandip Sen, editor, Collected Papers from the AAAI-97 Workshop on Multiagent Learning. AAAI Press, 1997.
13. Jerzy Filar and Koos Vrieze. Competitive Markov Decision Processes. Springer-Verlag, 1997.
14. A. M. Fink. Equilibrium in a stochastic n-person game. Journal of Science in Hiroshima University, Series A-I, 28:89-93, 1964.
15. Drew Fudenberg and David K. Levine. The Theory of Learning in Games. The MIT Press, 1998.
16. Amy Greenwald and Keith Hall. Correlated Q-learning. In Proceedings of the Twentieth International Conference on Machine Learning (ICML-2003), pages 242-249, Washington DC, 2003.
17. Stephan H. G. ten Hagen. Continuous State Space Q-Learning for Control of Nonlinear Systems. PhD thesis, University of Amsterdam, February 2001. http://carol.wins.uva.nl/~stephanh/phdthesis/.
18. John C. Harsanyi and Reinhard Selten. A General Theory of Equilibrium Selection in Games. The MIT Press, 1988.
19. Junling Hu. Learning in Dynamic Noncooperative Multiagent Systems. PhD thesis, University of Michigan, Ann Arbor, Michigan, July 1999.
20. Junling Hu and Michael P. Wellman. Multiagent reinforcement learning: Theoretical framework and an algorithm. In Fifteenth International Conference on Machine Learning, pages 242-250, Madison, WI, 1998.
21. Junling Hu and Michael P. Wellman. Experimental results on Q-learning for general-sum stochastic games. In Seventeenth International Conference on Machine Learning, pages 407-414, Stanford, 2000.
22. Rufus Isaacs. Differential Games: A Mathematical Theory with Applications to Warfare and Pursuit, Control and Optimization. R. E. Krieger Pub. Co., 1975.
23. Leslie Kaelbling, Michael L. Littman, and Andrew W. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237-285, 1996.
24. Michael Kearns, Michael L. Littman, and Satinder Singh. Graphical models for game theory. In Seventeenth Conference on Uncertainty in Artificial Intelligence, pages 253-260, Seattle, 2001.
25. Jeffrey O. Kephart and Gerald J. Tesauro. Pseudo-convergent Q-learning by competitive pricebots. In Seventeenth International Conference on Machine Learning, pages 463-470, Stanford, 2000.
26. Daphne Koller and B. Milch. Multi-agent influence diagrams for representing and solving games. In Seventeenth International Joint Conference on Artificial Intelligence, pages 1027-1034, Seattle, Washington, 2001.
27. Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Eleventh International Conference on Machine Learning, pages 157-163. New Brunswick, 1994.
28. Michael L. Littman. Friend-or-foe Q-learning in general-sum games. In Eighteenth International Conference on Machine Learning, pages 322-328, Williams College, MA, 2001a.
29. Michael L. Littman. Value-function reinforcement learning in Markov games. Cognitive Systems Research, 2:55-66, 2001b.
30. Michael L. Littman and Csaba Szepesvári. A generalized reinforcement-learning model: Convergence and applications. In Thirteenth International Conference on Machine Learning, pages 310-318, 1996.
31. Richard D. McKelvey and Andrew McLennan. Computation of equilibria in unite games. In Handbook of Computational Economics, volume 1. Elsevier, 1996.
32. Nicolas Meuleau and Paul Bourgine. Exploration of multi-state environments: Local measures and back-propagation of uncertainty. Machine Learning, 35(2): 117-154, 1999.
33. Tom Mitchell. Machine Learning, pages 367-390. McGraw-Hill, 1997.
34. John F. Nash. Non-cooperative games. Annals of Mathematics, 54:286-295, 1951.
35. Norihiko Ono and Kenji Fukumoto. Mulit-agent reinforcement learning: A modular approach. In Second International Conference on Multiagent Systems, pages 252-258, Kyoto, 1996.
36. Martin J. Osborne and Ariel Rubinstein. A Course in Game Theory. MIT Press, 1994.
37. Jing Peng and Ronald Williams. Incremental multi-step Q-learning. Machine Learning, 22:283-290, 1996.
38. Leon A. Petrosjan and Nikolay A. Zenkevich. Game Theory. World Scientific, Singapore, 1996.
39. Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, New York, 1994.
40. Yoav Shoham, Rob Powers, and Trond Grenager. Multi-agent reinforcement learning: A critical survey. Technical report, Stanford University, 2003.
41. Satinder Singh and Dimitri Bertsekas. Reinforcement learning for dynamic channel allocation in cellular telephone systems. In Advances in Neural Information Processing Systems, volume 9, pages 974-980. MIT Press, 1996.
42. Satinder Singh, Tommi Jaakkola, Michael L. Littman, and Csaba Szepesvári. Convergence results for single-step on-policy reinforcement learning algorithms. Machine Learning, 38(3):287-308, 2000.
43. Manu Sridharan and Gerald Tesauro. Multi-agent Q-learning and regression trees for automated pricing decisions. In Seventeenth International Conference on Machine Learning, pages 927-934, Stanford, 2000.
44. Peter Stone and Richard S. Sutton. Scaling reinforcement learning toward RoboCup soccer. In Proc. 18th International Conf. on Machine Learning, pages 537-544. Morgan Kaufmann, San Francisco, CA, 2001.
45. Nobuo Suematsu and Akira Hayashi. A multiagent reinforcement learning algorithm using extended optimal response. In First International Joint Conference on Autonomous Agents and Multiagent Systems, pages 370-377, Bologna, 2002.
46. Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
47. Csaba Szepesvári and Michael L. Littman. A unified analysis of value-function-based reinforcement-learning algorithms. Neural Computation, 11:8:2017-2059, 1999.
48. Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Tenth International Conference on Machine Learning, pages 330-337, Amherst, MA, 1993.
49. Frank Thusijsman. Optimality and Equilibria in Stochastic Games. Centrum voor Wiskunde en Informatica, Amsterdam, 1992.
50. Marilyn A. Walker. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. Journal of Artificial Intelligence Research, 12:387-416, 2000.
51. Christopher J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Cambridge, England, 1989.
52. Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 3:279-292, 1992.
53. Michael P. Wellman and Junling Hu. Conjectural equilibrium in multiagent learning. Machine Learning, 33:179-200, 1998.
54. Marco Wiering and Jurgen Schmidhuber. Fast online Q(λ). Machine Learning, 33:105-115, 1998.