Continuous-time Markov decision processes with nth-bias optimality criteria

Automatica

Volumn 45, Issue 7, 2009, Pages 1628-1638

  Zhang, Junyu ^a   Cao, Xi Ren ^b  

^a SUN YAT SEN UNIVERSITY   (China)

^b HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

Continuous time systems; Markov decision processes; Multichain model; nth bias optimality criteria; Performance analysis; Policy iteration algorithms; Sensitivity analysis

Indexed keywords

MARKOV DECISION PROCESSES; MULTICHAIN MODEL; NTH-BIAS OPTIMALITY CRITERIA; PERFORMANCE ANALYSIS; POLICY ITERATION ALGORITHMS;

ALGORITHMS; ITERATIVE METHODS; MARKOV PROCESSES; NONLINEAR CONTROL SYSTEMS; OPTIMIZATION; SENSITIVITY ANALYSIS;

CONTINUOUS TIME SYSTEMS;

EID: 67349166673     PISSN: 00051098     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.automatica.2009.03.009     Document Type: Article

References (26)

1. Anderson W.J. Continuous-time Markov chains (1991), Springer-Verlag, New York
2. Arapostathis A., Borkar V.S., Fernandez-Gaucherand E., Ghosh M.K., and Markus S.I. Discrete-time controlled Markov processes with average cost criterion: A survey. SIAM Journal on Control and Optimization 31 2 (1993) 282-344
3. Cao X.-R. From perturbation analysis to Markov decision processes and reinforcement learning. Discrete Event Dynamic Systems: Theory and Applications 13 1-2 (2003) 9-39
4. Cao X.-R. Stochastic learning and optimization - a sensitivity-based approach (2007), Springer, New York
5. Cao X.-R., and Guo X.P. A unified approach to Markov decision problems and performance sensitivity analysis with discounted and average criteria: Multichain cases. Automatica 40 9 (2004) 1749-1759
6. Cao X.-R., and Zhang J.Y. The nth-order bias optimality for multichain Markov decision processes. IEEE Transactions on Automatic Control 53 2 (2008) 496-508
7. Feller W. On the integro-differential equations of purely discontinuous Markoff processes. Transactions of the American Mathematical Society 48 (1940) 488-515
8. Guo X.P., and Cao X.-R. Optimal control of ergodic continuous-time Markov chains with average sample-path rewards. SIAM Journal on Control and Optimization 44 1 (2005) 29-48
9. Guo X.P., and Hernández-Lerma O. Drift and monotonicity conditions for continuous-time controlled Markov chains an average criterion. IEEE Transactions on Automatic Control 48 2 (2003) 236-245
10. Guo X.P., Hernández-Lerma O., and Prieto-Rumeau T. A survey of recent results on continuous-time Markov decision processes. Top 14 2 (2006) 177-261
11. Guo X.P., and Liu K. A note on optimality conditions for continuous-time Markov decision processes with average cost criterion. IEEE Transactions on Automatic Control 46 12 (2001) 1984-1989
12. Guo X.P., and Rieder U. Average optimality for continuous-time Markov decision processes in Polish spaces. The Annals of Applied Probability 16 2 (2006) 730-756
13. Guo, X. P., Song, X. Y., & Zhang, J. Y. (2009). Bias optimality for multichain continuous-time Markov decision processes. Preprint
14. Haviv M., and Puterman M.L. Bias optimality in controlled queueing systems. Journal of Applied Probability 35 1 (1998) 136-150
15. Howard R.A. Dynamic programming and Markov processes (1960), Wiley, New York
16. Kakumanu P. Nondiscounted continuous-time Markov decision processes with countable state and action spaces. SIAM Journal on Control 10 (1972) 210-220
17. Kitaev M.Y., and Rykov V.V. Controlled queueing systems (1995), CRC Press
18. Lewis M.E., and Puterman M.L. Bias optimality. In: Feinberg E.A., and Shwartz A. (Eds). Handbook of Markov decision processes (2002), Kluwer, Boston 89-111
19. Miller B.L. Finite state continuous time Markov decision processes with an infinite planning horizon. Journal Of Mathematical Analysis and Applications 22 (1968) 552-569
20. Oksendal B., and Sulem A. Applied stochastic control of jump diffusions (2007), Springer
21. Prieto-Rumeau T., and Hernández-Lerma O. The Laurent series, sensitive discount and Blackwell optimality for continuous-time controlled Markov chains. Mathematical Methods of Operations Research 61 1 (2005) 123-145
22. Prieto-Rumeau T., and Hernández-Lerma O. Bias optimality for continuous-time controlled Markov chains. SIAM Journal on Control and Optimization 45 1 (2006) 51-73
23. Puterman M.L. Markov decision processes: Discrete stochastic dynamic programming (1994), Wiley, New York
24. Sennott L.I. Stochastic dynamic programming and the control of queueing systems (1999), Wiley, New York
25. Taylor H.M. A Laurent series for the resolvent of a strongly continuous stochastic semi-group. Mathematical Programming Studies 6 (1976) 258-263
26. Veinott A.F. Discrete dynamic programming with sensitive discount optimality criteria. The Annals of Mathematical Statistics 40 5 (1969) 1635-1660