Convergence of simulation-based policy iteration

Probability in the Engineering and Informational Sciences

Volumn 17, Issue 2, 2003, Pages 213-234

  Cooper, William L ^a   Henderson, Shane G ^b   Lewis, Mark E ^c  

^a 4 174 EE CSCI Building   (United States)

^b School of Civil and Environmental Engineering   (United States)

^c UNIVERSITY OF MICHIGAN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

MARKOV PROCESSES;

ALMOST SURE CONVERGENCE; AVERAGE REWARD; MARKOV DECISION PROCESSES; OPTIMAL DECISION-RULE; OPTIMAL POLICIES; POLICY ITERATION; POLICY ITERATION ALGORITHMS; SIMULATION RUN LENGTH;

ITERATIVE METHODS;

EID: 0038380746     PISSN: 02699648     EISSN: None     Source Type: Journal    
DOI: 10.1017/S0269964803172051     Document Type: Article

References (22)

1. Barto, A.G., Sutton, R.S., & Anderson, C.W. (1983). Neuron-like elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man and Cybernetics 13: 835-846.
2. Bertsekas, D.P. (1995). Dynamic programming and optimal control, Vol. II. Belmont, MA: Athena Scientific.
3. Bertsekas, D.P. & Tsitsiklis, J.N. (1996). Neuro-dynamic programming, Belmont, MA: Athena Scientific.
4. Brémaud, P. (1999). Markov chains: Gibbs fields, Monte Carlo simulation, and queues. New York: Springer-Verlag.
5. Cao, X.-R. (1998). The relations among potentials, perturbation analysis, and Markov decision processes. Discrete-Event Dynamic Systems: Theory and Applications 8: 71-87.
6. Cao, X.-R. (1999). Single sample path-based optimization of Markov chains. Journal of Optimization Theory and Applications 100: 527-548.
7. Cao, X.-R. (2000). A unified approach to Markov decision problems and performance sensitivity analysis. Automatica 36: 771-774.
8. Cao, X.-R. & Chen, H.-F. (1997). Perturbation realization, potentials, and sensitivity analysis of Markov processes. IEEE Transactions on Automatic Control 42(10): 1382-1393.
9. Fristedt, B. & Gray, L. (1997). A modern approach to probability theory. Boston: Birkhauser.
10. Howard, R.A. (1960). Dynamic programming and Markov processes. New York: Wiley.
11. Konda, V.R. & Borkar, V.S. (1999). Actor-critic-type learning algorithms for Markov decision processes. SIAM Journal on Control and Optimization 38: 94-123.
12. Lewis, M.E. & Puterman, M.L. (2001). A probabilistic analysis of bias optimality in unichain Markov decision processes. IEEE Transactions on Automatic Control 46(1): 96-100.
13. Lewis, M.E. & Puterman, M.L. (2002). Bias optimality. In E. Feinberg & A. Shwartz (eds.), Handbook of Markov decision processes: Methods and applications. Boston: Kluwer Academic.
14. Propp, J.G. & Wilson, D.B. (1996). Exact sampling with coupled Markov chains and applications to statistical mechanics. Random Structures and Algorithms 9: 223-252.
15. Propp, J.G. & Wilson, D.B. (1998). How to get a perfectly random sample from a generic Markov chain and generate a random spanning tree of a directed graph. Journal of Algorithms 27:170-217.
16. Puterman, M.L. (1994). Markov decision processes: Discrete stochastic dynamic programming. New York: Wiley.
17. Resnick, S. (1992). Adventures in stochastic processes. Boston: Birkhauser.
18. Ross, S.M. (1996). Stochastic processes, 2nd ed. New York: Wiley.
19. Thorisson, H. (2000). Coupling, stationarity, and regeneration. New York: Springer-Verlag.
20. Tsitsiklis, I. (1994). Asynchronous stochastic approximation and Q-learning. Machine Learning 16: 195-202.
21. Watkins, C.J.C.H. & Dayan, P. (1992). Q-Learning. Machine Learning 8: 279-292.
22. Wilson, D.B. (2000). Layered multishift coupling for use in perfect sampling algorithms (with a primer on CFTP). Fields Institute Communications, Vol. 26. Providence, RI: American Mathematical Society.