Semi-Markov decision problems and performance sensitivity analysis

IEEE Transactions on Automatic Control

Volumn 48, Issue 5, 2003, Pages 758-769

  Cao, Xi Ren ^a  

^a HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

Discounted Poisson equations; Discrete event dynamic systems (DEDS); Lyapunov equations; Markov decision processes (MDPs); Perturbation analysis (PA); Perturbation realization; Poisson equations; Policy iteration; Potentials; Reinforcement learning (RL)

Indexed keywords

ALGORITHMS; DECISION MAKING; ITERATIVE METHODS; LYAPUNOV METHODS; MATHEMATICAL MODELS; PERTURBATION TECHNIQUES; POISSON EQUATION; SENSITIVITY ANALYSIS;

REINFORCEMENT LEARNING (RL);

MARKOV PROCESSES;

EID: 0038631988     PISSN: 00189286     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAC.2003.811252     Document Type: Article

References (23)

1. D. P. Bertsekas, Dynamic Programming and Optimal Control. Belmont, MA: Athena Scientific, 1995, vol. II.
2. A. Herman and R. J. Plemmons, "Nonnegative matrices in the mathematical sciences," SIAM J. Numer. Anal. vol. 11, pp. 145-154, 1974.
3. D. P. Bertsekas and T. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
4. X.-R. Cao, Realization Probabilities: The Dynamics of Queueing Systems. New York: Springer-Verlag, 1994.
5. _, "The Maclaurin series for performance functions of Markov chains," Adv. Appl. Probab., vol. 30, pp. 676-692, 1998.
6. _, "The relation among potentials, perturbation analysis, Markov decision processes, and other topics," J. Discrete Event Dyna. Syst., vol. 8, pp. 71-87, 1998.
7. _, "A unified approach to Markov decision problems and performance sensitivity analysis," Automatica, vol. 36, pp. 771-774, 2000.
8. _, "From perturbation analysis to Markov decision processes and reinforcement learning," J. Discrete Event Dyna. Syst., vol. 13, pp. 9-39, 2003.
9. X.-R. Cao and H. F. Chen, "Potentials, perturbation realization, and sensitivity analysis of Markov processes," IEEE Trans. Automat. Contr., vol. 42, pp. 1382-1393, Oct. 1997.
10. X.-R. Cao, Z. Y. Ren, S. Bhatnagar, M. Fu, and S. Marcus, "A time aggregation approach to Markov decision processes," Automatica, vol. 38, pp. 929-943, 2002.
11. C. Cassandras and S. Lafortune, Introduction to Discrete Event Dynamic Systems. Norwell, MA: Kluwer, 1999.
12. E. Çinlar, Introduction to Stochastic Processes. Upper Saddle River, NJ: Prentice Hall, 1975.
13. H.-T. Fang and X.-R. Cao, "Single sample path based recursive algorithms for Markov decision processes," IEEE Trans. Automat. Contr., 2003, to be published.
14. P.W. Glynn and S. P. Meyn, "A Lyapunov bound for solutions of Poisson's equation," Ann. Probab., vol. 24, pp. 916-931, 1996.
15. Y. C. Ho and X.-R. Cao, Perturbation Analysis of Discrete-Event Dynamic Systems. Nonvell, MA: Kluwer, 1991.
16. J. G. Kemeny and J. L. Snell, Finite Markov Chains. New York: Van Nostrand, 1960.
17. L. Kleinrock, Queueing Systems, Volume 1: Theory. New York: Wiley, 1975.
18. P. Marbach and T. N. Tsitsiklis, "Simulation-based optimization of Markov reward processes," IEEE Trans. Automat. Contr., vol. 46, pp. 191-209, Feb. 2001.
19. S. P. Meyn and R. L. Tweedie, Markov Chains and Stochastic Stability. London, U.K.: Springer-Verlag, 1993.
20. M. L. Puterman, Markov Decision Processes; Discrete Stochastic Dynamic Programming. New York: Wiley, 1994.
21. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.
22. R. S. Sutton, D. Precup, and S. Singh, "Between MDPs and Semi-MDPs: a framework for temporal abstraction in reinforcement learning," Artif. Intell., vol. 112, pp. 181-211, 1999.
23. H. C. Tijms, Stochastic Models - An Algorithmic Approach: Wiley, 1994.