From perturbation analysis to Markov decision processes and reinforcement learning

Discrete Event Dynamic Systems: Theory and Applications

Volumn 13, Issue 1-2, 2003, Pages 9-39

  Cao, Xi Ren ^a  

^a HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

Gradient based policy iteration; Perturbation realization; Poisson equations; Potentials; Q learning; TD( )

Indexed keywords

CORRELATION METHODS; DECISION MAKING; DYNAMIC PROGRAMMING; GRADIENT METHODS; LEARNING SYSTEMS; MARKOV PROCESSES; OPTIMIZATION; PARAMETER ESTIMATION; PERTURBATION TECHNIQUES; POISSON EQUATION; SENSITIVITY ANALYSIS; STATE SPACE METHODS;

GRADIENT BASED POLICY ITERATION; MARKOV DECISION PROCESSES; NEURODYNAMIC PROGRAMMING; PERTURBATION ANALYSIS; PERTURBATION REALIZATION; Q-LEARNING; REINFORCEMENT LEARNING;

DECISION SUPPORT SYSTEMS;

EID: 0037289322     PISSN: 09246703     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1022188803039     Document Type: Article

References (43)

1. Abounadi, J., Bertsekas, D., and Borkar, V. Learning algorithms for Markov decision processes with average cost. Report LIDS-P-2434, Lab. for Info. and Decision Systems, October 1998; to appear in SIAM J. on Control and Optimization.
2. Altman, E. 1999. Constrained Markov Decision Processes, Chapman Hall/CRC.
3. Barto, A., Sutton, R., and Anderson, C. 1983. Neuron-like elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics 13: 835-846.
4. Baxter, J., and Bartlett, P. L. 2001. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research 15: 319-350.
5. Baxter, J., Bartlett, P. L., and Weaver, L. 2001. Experiments with infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research 15: 351-381.
6. Bertsekas, D. P. 1995. Dynamic Programming and Optimal Control. Vols. I. II. Athena Scientific, Belmont, Massachusetts.
7. Berman, A., and Plemmons, R. J. 1994. Nonnegative matrices in the mathematical sciences. SIAM, Philadelphia.
8. Bertsekas, D. P., and Tsitsiklis, T. N. 1996. Neuro-Dynamic Programming. Athena Scientific, Belmont, Massachusetts.
9. Cao, X. R. 1994. Realization Probabilities: The Dynamics of Queueing Systems. Springer-Verlag, New York.
10. Cao, X. R. 1998. The relation among potentials, perturbation analysis, Markov decision processes, and other topics. Journal of Discrete Event Dynamic Systems 8: 71-87.
11. Cao, X. R. 1999. Single sample path-based optimization of markov chains. Journal of Optimization: Theory and Application 100: 527-548.
12. Cao, X. R. 2000. A unified approach to Markov decision problems and performance sensitivity analysis. Automatica 36: 771-774.
13. Cao, X. R., and Chen, H. F. 1997. Potentials, perturbation realization, and sensitivity analysis of Markov processes. IEEE Transactions on AC 42: 1382-1393.
14. Cao, X. R., Ren, Z. Y., Bhatnagar, S., Fu, M., and Marcus, S. 2002. A time aggregation approach to Markov decision processes. Automatica 38: 929-943.
15. Cao, X. R., and Fang, H. T. Gradient-based policy iteration: an example. To appear in 2002 IEEE Conference on Decision and Control.
16. Cao, X. R., and Wan, Y. W. 1998. Algorithms for sensitivity analysis of Markov systems through potentials and perturbation realization. IEEE Transactions on Control Systems Technology 6: 482-494.
17. Cassandras, C. G., and Lafortune, S. 1999. Introduction to Discrete Event Systems. Kluwer Academic Publishers.
18. Chong, E. K. P., and Ramadge, P. J. 1992. Convergence of recursive optimization algorithms using infinitesimal perturbation analysis estimates. Journal of Discrete Event Dynamic Systems 1: 339-372.
19. Çinlar, E. 1975. Introduction to Stochastic Processes. Prentice Hall, Englewood cliffs, NJ.
20. Fang, H. T., and Cao, X. R. Single sample path-based recursive algorithms for Markov decision processes. IEEE Trans. on Automatic Control, submitted.
21. Feinberg, E. A., and Adam Shwartz (ed.) 2002. Handbook of Markov Decision Processes. Kluwer, 2002.
22. Fu, M. C. 1990. Convergence of a stochastic approximation algorithm for the GI/G/1 queue using infinitesimal perturbation analysis. Journal of Optimization Theory and Applications 65: 149-160.
23. Fu, M. C. and Hu, J. Q. 1997. Conditional Monte Carlo: Gradient Estimation and Optimization Applications. Kluwer Academic Publishers, Boston.
24. Glasserman, P. 1991. Gradient Estimation Via Perturbation Analysis. Kluwer Academic Publishers, Boston.
25. Glynn, P. W., and Meyn, S. P. 1996. A Lyapunov bound for solutions of Poisson's equation. Ann. Probab. 24: 916-931.
26. Gong, W. B., and Ho, Y. C. 1987. Smoothed perturbation analysis of discrete event systems. IEEE Transactions on Control Systems Technology 32: 858-866.
27. Ho, Y. C., and Cao, X. R. 1991. Perturbation Analysis of Discrete-Event Dynamic Systems. Kluwer Academic Publisher, Boston.
28. Ho, Y. C., and Cao, X. R. 1983. Perturbation analysis and optimization of queueing networks. Journal of Optimization Theory and Applications 40(4): 559-582.
29. Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. 1998. Planning and acting in partially observable stochastic domains. Artificial Intelligence 101.
30. Kemeny, J. G., and Snell, J. L. 1960. Finite Markov Chains. Van Nostrand, New York.
31. Konda, V. R., and Borkar, V. S. 1990. Actor-critic like learning algorithms for Markov decision processes. SIAM Journal on Control and Optimization 38: 94-123.
32. Konda, V. R., and Tsitsiklis, J. N. 2001. Actor-critic Algorithms. Submitted to SIAM Journal on Control and Optimisation, February.
33. Marbach, P., and Tsitsiklis, T. N. 2001. Simulation-based optimization of Markov reward processes. IEEE Transactions on Automatic Control 46: 191-209.
34. Meyn, S. P. 1997. The policy improvement algorithm for Markov decision processes with general state space. IEEE Transactions on Automatic Control 42: 1663-1680.
35. Meyn, S. P., and Tweedie, R. L. 1993. Markov Chains and Stochastic Stability. Springer-Verlag, London.
36. Puterman, M. L. 1994. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York.
37. Singh, S. P. 1994. Reinforcement learning algorithms for average-payoff Markovain decision processes. Proceedings of the Twelfth National Conference on Artificial Intelligence 202-207.
38. Smart, W. D., and Kaelbling, L. P. 2000. Practical reinforcement learning in continuous spaces. Proceedings of the Seventeenth International Conference on Machine Learning.
39. Sutton, R. S. 1988. Learning to predict by the methods of temporal differences. Machine Learning 3: 835-846.
40. Sutton, R. S., and Barto, A. G. 1998. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA.
41. Tsitsiklis, J. N., and Van Roy, B. 1999. Average cost temporal-difference learning. Automatica 35: 1799-1808.
42. Vazquez-Abad, F. J., Cassandras, C. G., and Julka, V. 1998. Centralized and decentralized asynchronous optimization of stochastic discrete event systems. IEEE Transactions on Automatic Control 43: 631-655.
43. Zhang, B., and Ho, Y. C. 1991. Performance gradient estimation for very large finite Markov chains. IEEE Transactions on Automatic Control 36: 1218-1227.