A reinforcement learning algorithm for obtaining the Nash equilibrium of multi-player matrix games

IIE Transactions (Institute of Industrial Engineers)

Volumn 41, Issue 2, 2009, Pages 158-167

  Nanduri, Vishnu ^a   Das, Tapas K ^a  

^a University of South Florida   (United States)

Author keywords

Matrix games; Reinforcement Learning; Restructured electricity markets; Vash equilibrium

Indexed keywords

ALGORITHMS; APPROXIMATION ALGORITHMS; COMMERCE; COMPETITION; ELECTRICITY; ELECTRONIC COMMERCE; GAME THEORY; LEARNING SYSTEMS; REINFORCEMENT; REINFORCEMENT LEARNING; SOLUTIONS; STOCHASTIC MODELS; TELECOMMUNICATION NETWORKS;

ACTION SPACES; AVERAGE REWARDS; BROADER IMPACTS; COMPARATIVE ANALYSIS; COMPUTATIONAL ALGORITHMS; E COMMERCES; ELECTRIC POWER MARKETS; MARKET SCENARIOS; MARKOV DECISION PROCESSES; MARKOV DECISIONS; MATRIX GAMES; NASH EQUILIBRIUMS; REINFORCEMENT LEARNING ALGORITHMS; RESTRUCTURED ELECTRICITY MARKETS; SOLUTION APPROACHES; STOCHASTIC GAMES; STRATEGIC BIDDINGS; VASH EQUILIBRIUM;

LEARNING ALGORITHMS;

EID: 56749179073     PISSN: 0740817X     EISSN: 15458830     Source Type: Journal    
DOI: 10.1080/07408170802369417     Document Type: Article

References (37)

1. Berry, C. A., Hobbs, B. F., Meroney, W. A., O'Neill, R. P. and Stewart Jr., W. R. (1999) Understanding how market power can arise in network competition: A game theoretic approach. Utilities Policy, 8, pp. 139-158.
2. Bowling, M. and Veloso, M. M. (2000) An analysis of stochastic game theory for multiagent reinforcement learning, Computer Science Department, Carnegie Mellon University, Pittsburgh, PA
3. Borkar, V. S. (2002) Reinforcement learning in Markovian evolutionary games. Advances in Complex Systems, 5:1, pp. 55-72.
4. Cardell, J. B., Hitt, C. C. and Hogan, W. W. (1997) Market power and strategic interaction in electricity networks. Resource and Energy Economics, 19, pp. 109-137.
5. Cunningham, L. B., Baldick, R. and Baughman, M. L. (2002) An empirical study of applied game theory: Transmission constrained Cournot behavior. IEEE Transactions on Power Systems, 17:1, pp. 166-172.
6. Darken, C., Chang, J. and Moody, J. (1992) Learning rate schedules for faster stochastic gradient search. Neural Networks for Signal Processing 2-Proceedings of the 1992 IEEE Workshop IEEE Press, Piscataway, NJ
7. Das, T. K., Gosavi, A., Mahadevan, S. and Marchalleck, N. (1999) Solving semi-Markov decision problems using average reward reinforcement learning. Management Science, 45:4, pp. 560-574.
8. Ferrero, R. W., Shahidehpour, M. and Ramesh, V. C. (1999) Transaction analysis in deregulated power systems using game theory. IEEE Transactions on Power Systems, 12:3, pp. 1340-1347.
9. Filar, J. and Vrieze, K. (1997) Competitive Markov Decision Processes, Springer Verlag, New York, NY
10. Gosavi, A. (2004) Reinforcement learning for long-run average cost. European Journal of Operational Research, 155, pp. 654-674.
11. Gosavi, A., Bandla, N. and Das, T. K. (2002) A reinforcement learning approach to airline seat allocation for multiple fare classes with overbooking. IIE Transactions, 34:9, pp. 729-742.
12. Harker, P. T. and Pang, J. S. (1990) Finite dimensional variational inequality and nonlinear complementarity problems: A survey of theory, algorithms and applications. Mathematical Programming, 48, pp. 161-220.
13. Hobbs, B. F. (2001) Linear complementarity models of Nash-Cournot competition in bilateral and POOLCO power markets. IEEE Transactions on Power Systems, 16:2, pp. 194-202.
14. Hobbs, B. F., Carolyn, B. M. and Pang, J. S. (2006) Strategic gaming analysis for electric power systems: An MPEC approach. IEEE Transactions on Power Systems, 15:2, pp. 638-645.
15. Hu, J. and Wellman, M. P. (1998) Multiagent reinforcement learning: theoretical framework and an algorithm. Presented at the 15th International Conference on Machine Learning Madison, WI
16. Hu, J. and Wellman, M. P. (2003) Nash Q-learning for general-sum stochastic games. Journal of Machine Learning Research, 4, pp. 1039-1069.
17. Kaelbling, L. P., Littman, M. L. and Moore, A. W. (1996) Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, pp. 237-285.
18. Lee, K. H. and Baldick, R. (2003) Solving three-player games by the matrix approach with an application to an electric power market. IEEE Transactions on Power Systems, 18:4, pp. 166-172.
19. Lemke, C. E. and Howson Jr., J. T. (1964) Equilibrium points of bimatrix games. SIAM Journal of Applied Mathematics, 12, pp. 413-423.
20. Li, J., Ramachandran, K. and Das, T. K. (2007) A reinforcement learning (Nash-r) algorithm for average reward irreducible stochastic games. Journal of Machine Learning Research
21. Littman, M. L. (1994) Markov games as a framework for multi-agent reinforcement learning. Proceedings of the 11th International Conference on Machine Learning Morgan Kaufman, pp. 157-163.
22. Luo, Z., Pang, J. S. and Ralph, D. (1996) Mathematical Programs with Equilibrium Constraints, Cambridge University Press, Boston, MA
23. Mathiesen, L. (1985) Computational experience in solving equilibrium models by a sequence of linear complementarity problems. Operations Research, 33:6, pp. 1225-1250.
24. McKelvey, R. and McLennan, A. (1996) Computation of equilibria in finite games. Handbook of Computational Economics, pp. 87-142. Elsevier, The Netherlands
25. McKelvey, R. D., McLennan, A. M. and Turocy, T. L. (2007) Gambit: software tools for game theory, version 0.2007.01.30
26. Nash, J. (1951) Non-cooperative games. The Annals of Mathematics, 54:2, pp. 286-295.
27. Price, T. C. (1997) Using co-evolutionary programming to simulate strategic behavior in markets. Journal of Evolutionary Economics, 7, pp. 219-254.
28. Puterman, M. L. (1994) Markov Decision Processes, Wiley, New York, NY
29. Ralph, D. and Smeers, Y. (2006) Epecs as models for electricity markets. Presented at the Power Systems Conference and Exposition Atlanta, GA
30. Rosenmuller, J. (1971) On the generalization of the Lemke-Howson algorithm to noncooperative n-person games. SIAM Journal on Applied Mathematics, 21:1, pp. 73-79.
31. Shapley, L. S. (1953) Stochastic games. Classics in Game Theory, Princeton University Press, Princeton, NJ
32. Stoft, S. (1999) Using game theory to study market power in simple networks. IEEE Tutorial on Game Theory in Electric Power Markets, pp. 33-40. IEEE, Piscataway, NJ
33. Stoft, S. (2002) Power System Economics, IEEE Press, Piscataway, NJ
34. Su, C. L. (2005) Equilibrium problems with equilibrium constraints: stationarities, algorithms, and applications, Ph.D. Dissertation. Department of Management Sciences, Stanford University, Stanford, CA
35. Sutton, R. S. and Barto, A. G. (1998) Reinforcement Learning, The MIT Press, Cambridge, MA
36. Wilson, R. (1971) Computing equilibria of n-person games. SIAM Journal of Applied Mathematics, 21, pp. 80-87.
37. Yao, J., Oren, S. and Adler, I. (2004) Computing Cournot equilibria in two settlement electricity markets with transmission constraints. Proceedings of the 37th Hawaii International Conference on System Sciences, vol. 2 IEEE Computer Society, p. 20051b. Los Alamitos, CA