On the empirical state-action frequencies in Markov decision processes under general policies

Mathematics of Operations Research

Volumn 30, Issue 3, 2005, Pages 545-561

  Mannor, Shie ^a   Tsitsiklis, John N ^b  

^a MCGILL UNIVERSITY   (Canada)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Empirical measure; Large deviations; Markov decision processes; State action frequencies

Indexed keywords

EMPIRICAL MEASURE; LARGE DEVIATIONS; MARKOV DECISION PROCESSES; STATE-ACTION FREQUENCIES;

COMMUNICATION; DECISION MAKING; PROBABILITY; PUBLIC POLICY;

MARKOV PROCESSES;

EID: 31144437617     PISSN: 0364765X     EISSN: 15265471     Source Type: Journal    
DOI: 10.1287/moor.1050.0148     Document Type: Article

References (17)

1. Altman, E., A. Shwartz. 1991. Markov decision problems and state-action frequencies. SIAM J. Control Optim. 29(4) 786-809.
2. Altman, E., O. Zeitouni. 1994. Rate of convergence of empirical measures and costs in controlled Markov chains and transient optimality. Math. Oper. Res. 19(4) 955-974.
3. Balaji, S., S. P. Meyn. 2000. Multiplicative ergodicity and large deviations for an irreducible Markov chain. Stochastic Processes Their Appl. 90 123-144.
4. Bertsekas, D. P., J. N. Tsitsiklis. 1996. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA.
5. Bertsimas, D., J. N. Tsitsiklis. 1999. Introduction to Linear Optimization. Athena Scientific, Belmont, MA.
6. Dembo, A., O. Zeitouni. 1998. Large Deviations Techniques and Applications, 2nd ed. Springer, New York.
7. Derman, C. 1970. Finite State Markovian Decision Processes. Academic Press, Orlando, FL.
8. Gallager, R. G. 1996. Discrete Stochastic Processes. Kluwer Academics, Norwell, MA.
9. Glynn, P. W., D. Ormoneit. 2002. Hoeffding's inequality for uniformly ergodic Markov chains. Statist. Probab. Lett. 56 143-146.
10. Hajek, B. 1982. Hitting and occupation time bounds implied by drift analysis and applications. Adv. Appl. Probab. 14 502-525.
11. Hirsch, W. M. 1965. A strong law for the maximum cumulative sum of independent random variables. Comm. Pure Appl. Math. 18 109-127.
12. Kallenberg, L. C. M. 1983. Linear programming and finite Markovian control problems. Technical Report 148, Mathematics Centrum Tract, Amsterdam, The Netherlands.
13. Kumar, P. R., P. Varaiya. 1986. Stochastic Systems: Estimation, Identification and Adaptive Control. Prentice Hall, Englewood Cliffs, NJ.
14. Miller, H. D. 1961. A convexity property in the theory of random variables defined by a finite Markov chain. Ann. Math. Statist. 32(4) 1260-1270.
15. Puterman, M. 1994. Markov Decision Processes. Wiley-Interscience, New York.
16. Shimkin, N. 1993. Extremal large deviations in controlled I.I.D. processes with applications to hypothesis testing. Adv. Appl. Probab. 25 875-894.
17. Sutton, R. S., A. G. Barto. 1998. Reinforcement Learning. MIT Press, Cambridge, MA.