Contraction conditions for average and α-discount optimality in countable state Markov games with unbounded rewards

Mathematics of Operations Research

Volumn 22, Issue 3, 1997, Pages 588-618

  Altman, E ^a   Hordijk, A ^b   Spieksma, F M ^b  

^a INRIA   (France)

^b LEIDEN UNIVERSITY   (Netherlands)

Author keywords

Birth death control; Equilibrium policies; Noncooperative Markov games; Value iteration; geometric recurrence

Indexed keywords

ALGORITHMS; COMPUTATIONAL COMPLEXITY; CONVERGENCE OF NUMERICAL METHODS; ITERATIVE METHODS; MARKOV PROCESSES; MATHEMATICAL MODELS; OPTIMIZATION; PROBLEM SOLVING; QUEUEING THEORY; STATE SPACE METHODS;

COMPACT ACTION SPACES; STATIONARY EQUILIBRIUM POLICY; ZERO SUM MARKOV GAMES;

GAME THEORY;

EID: 0031210649     PISSN: 0364765X     EISSN: None     Source Type: Journal    
DOI: 10.1287/moor.22.3.588     Document Type: Article

References (45)

1. Altman, E. (1994a). Monotonicity of optimal policies in a zero sum game: A flow control model. Advances of Dynamic Games and App. 1 269-286.
2. _ (1994b). Flow control using the theory of zero-sum Markov games. IEEE Trans. Automat. Control, 39 814-818.
3. _ (1995). Constrained Markov decision processes. INRIA report No. 2574.
4. _, A. Hordijk (1995). Zero-sum Markov games and worst case optimal control of queueing systems. Queueing Systems 21 415-447.
5. _, E. A. Feinberg, A, Shwartz (1996). Weighted discounted zero-sum stochastic games with perfect information, Proceedings of the Seventh International Symposium on Dynamic Games and Applications, December 16-18, 1996 Shonan Village Center, Kanagawa, Japan, (to appear).
6. Bewley, T., E. Kohlberg (1976). The asymptotic theory of stochastic games. Math. Oper. Res. 1 197-208.
7. Borkar, V. S., M. K. Ghosh (1993). Denumerable state stochastic games with limiting average payoff. J. Optim. Theory Appl. 539-560.
8. Cavazos-Cadena, R. (1991). Recent results on conditions for the existence of average optimal stationary policies. Ann. Oper. Res. 28 3-28.
9. Chung, K. L. (1967). Markov Chains with Stationary Transition Probabilities, 2nd. ed., Springer Verlag, Berlin.
10. Dekker, R., A. Hordijk (1988). Average, sensitive and Blackwell optimality in denumcrablc state Markov decision chains with unbounded rewards. Math. Oper. Res. 13 395-421.
11. _, _ (1992). Recurrence conditions for average and Blackwell optimality in denumerable state Markov decision chains. Math Oper. Res. 17 271-290.
12. _, _, F. M. Spieksma (1994). On the relation between recurrence and ergodicity properties in denumerable Markov decision chains. Technical report, Leiden University, Technical report no. TW-91-02, Dept. of Math, and Comp. Sci., Univ. of Leiden (extended version), Math. Oper. Res. 19 539-559 (shortened version).
13. Derman, C., R. Strauch (1966). A note on memoryless rules for controlling sequential control processes. Ann. Math. Statist. 37 276-278.
14. van der Duyn Schouten, F. A. (1979). Markov decision drift Processes with continuous time parameter, Ph.D. Thesis, Leiden University.
15. Federgruen, A. (1978). On N-person stochastic games with denumerable state space. Adv. Appl. Probab. 10 452-471.
16. _ (1980a). On the functional equations in undiscounted and sensitive discounted stochastic games. Z. Oper. Res. 24 243-262.
17. _ (1980b). Successive approximation methods in undiscounted stochastic games. Oper. Res. 28 794-810.
18. Glicksberg, L. L. (1952). A further generalization of the Kakutani fixed point theorem, with application to Nash equilibrium points. Proc. Amer. Math. Soc. 3 170-174.
19. Hordijk, A. (1974). Dynamic Programming and Markov Potential Theory. Mathematical Centre Tract 51, C.W.I., Amsterdam.
20. _, F. A. van der Duyn Schouten (1983). Average optimal policies in Markov decision processes with applications to a queueing and a replacement model. Adv. Appl. Probab. 15 274-303.
21. _, P. J. Schweitzer, H. C. Tijms (1975). The asymptotic behaviour of the minimal total expected cost for the denumerable Markov decision model. J. Appl. Probab. 12 298-305.
22. _, F. M. Spieksma (1992). On ergodicity and recurrence properties of a Markov chain with an application to an open Jackson network. Adv. Appl. Probab. 24 343-376.
23. _, O. J. Vrieze, G. L. Wanrooij (1983). Semi-Markov strategies in stochastic games. Internat. J. Game Theory 12 81-89.
24. Lippman, S. A, (1975). Applying a new device in the optimization of exponential queueing systems. Oper. Res. 23 687-710.
25. Maitra, A., W. Sudderth (1993). Borel stochastic games with limsup payoff. Ann. Probab. 21 861-885.
26. Majumdar, M., R. Sundaran (1991). Symmetric stochastic games of resource extraction: The existence of nonrandomized stationary equilibrium. Stochastic Games and Related Topics, Kluwer Academic Publishers, Dordrecht, The Netherlands, 175-190.
27. Mertens, J.-F., A. Neyman (1981). Stochastic Games. Internat. J. Game Theory 10 53-66.
28. _, T. Parthasarathy (1987). Equilibria for discounted stochastic games. CORE Discussion Paper No. 8750, Université Catholique de Louvain.
29. Nowak, A. S. (1994). Stationary equilibria for nonzero-sum average payoff ergodic stochastic games with general state space. Basar and Haurie, Eds., Advances in dynamic games and applications, Annals of the Intern. Soc. of Dynamic Games, Vol. 1, Birkhauser, 231-246.
30. _, T. E. S. Raghavan (1992). Existence of stationary correlated equilibria with symmetric information for discounted stochastic games. Math. Oper. Res. 17 519-526.
31. T. Parthasarathy, S. Sinha (1989). Existence of stationary equilibrium strategies in nonzero-sum discounted games with uncountable state space and state independent transitions, Internat. J. Game Theory 18 189-194.
32. _, M. Stern (1977). Markov games - A survey. Roxin, Liu and Sternberg, Eds., Differential Games and Control Theory II, 1-46.
33. T. E. S. Raghavan, J. A. Filar (1991). Algorithms for stochastic games-A survey. Z. Oper. Res. 35 437-472.
34. Rieder, U. (1979). Equilibrium plans for nonzero-sum Markov games. Moeschlin and Pallaschke, Eds., Game Theory and Related Topics, North-Holland, Amsterdam, 91-102.
35. Royden, H. L. (1988). Real Analysis, 3rd ed., Macmillan Publishing Company, New York.
36. Schal, M. (1992). On the second optimality equation for semi-Markov decision models. Math. Oper. Res. 17 470-486.
37. Schweitzer, P. J. (1971). Iterative solution to the functional equations of undiscounted Markov renewal programming. J. Math. Anal. Appl. 34 495-501.
38. Sennott, L. I. (1994). Zero-sum stochastic games with unbounded costs: Discounted and average cost cases. Z. Oper. Res. 40 145-162.
39. Serfozo, R. F. (1978). An equivalence between continuous and discrete time Markov decision processes. Oper. Res. 27 616-620.
40. Shapley, L. S. (1953). Stochastic games. Proc. Nat. Acad. Sci. USA 39 1095-1100.
41. Spieksma, F. M. (1990). Geometrically ergodic Markov chains and the optimal control of queues, Ph.D. Thesis, Leiden University (available on request from the author).
42. _, R. L. Tweedie (1994). Strengthening ergodicity to geometric ergodicity of Markov chains. Stoch. Mod. 10 45-75.
43. van der Wal (1980a). Stochastic Dynamic Programming: successive approximations and nearly optimal strategies for Markov decision processes and Markov games. Ph.D. Thesis, Eindhoven.
44. _ (1980b). Successive approximations for average reward Markov games. Internat. J. Game Theory 9 13-24.
45. Wessels, J. (1977). Markov games with unbounded rewards. Memorandum COSOR 76-05; M. Schäl, Ed., Dynamische Optimierung, Bonner Math. Schriften 98, Bonn.