Potential-based online policy iteration algorithms for Markov decision processes

IEEE Transactions on Automatic Control

Volumn 49, Issue 4, 2004, Pages 493-505

  Fang, Hai Tao ^a   Cao, Xi Ren ^b  

^a CHINESE ACADEMY OF SCIENCES   (China)

^b HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

Markov decision process; Potential; Recursive optimization

Indexed keywords

DECISION SUPPORT SYSTEMS; ITERATIVE METHODS; MARKOV PROCESSES; OPTIMAL SYSTEMS;

MARKOV DECISION PROCESS; ONLINE POLICY ITERATION ALGORITHMS;

ALGORITHMS;

EID: 2442614974     PISSN: 00189286     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAC.2004.825647     Document Type: Article

References (38)

1. A. Arapostathis, V. S. Borkar, E. Fernandez-Gaucherand, M. K. Ghosh, and S. I. Marcus, "Discrete-time controlled Markov processes with average cost criterion: A survey," SIAM J. Control Optim., vol. 31, pp. 282-344, 1993.
2. D. P. Bertsekas, Dynamic Programming and Optimal Control. Belmont, MA: Athena Scientific, 1995, vol. I and II.
3. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
4. P. Brémaud, Markov Chains: Gibbs Fields, Monte Carlo Simulation, and Queues. New York: Springer-Verlag, 1998.
5. X.-R. Cao, "The relation among potentials, perturbation analysis, Markov decision processes, and other topics," J. Discrete Event Dyna. Syst., vol. 8, pp. 71-87, 1998.
6. ____, "Single sample path based optimization of Markov chains," J. Optim.: Theory Applicat. vol. 100, no. 3, pp. 527-548, 1999.
7. X.-R. Cao and H. F. Chen, "Perturbation realization, potentials and sensitivity analysis of Markov processes," IEEE Trans. Automat. Contr., vol. 42, pp. 1382-1393, Sept. 1997.
8. X.-R. Cao and H.-T. Fang, "Gradient-based policy iteration: An example," presented at the 2002 IEEE Conf. Decision Control.
9. X.-R. Cao, Z. Y. Ren, S. Bhatnagar, F. Fu, and S. Marcus, "A time aggregation approach to Markov decision processes," Automatica, vol. 38, pp. 929-943, 2002.
10. X.-R. Cao and Y. W. Wan, "Algorithms for sensitivity analysis of Markov systems through potentials and perturbation realization," IEEE Trans. Contr. Syst. Tech, vol. 6, pp. 482-494, July 1998.
11. C. G. Cassandras and S. Lafortune, Introduction to Discrete Event Systems. Norwell, MA: Kluwer, 1999.
12. W. L. Cooper, S. H. Henderson, and M. E. Lewis, "Convergence of simulation-based policy iteration," Probab. Eng. Inform. Sci., vol. 17, pp. 213-234, 2003.
13. E. K. P. Chong and P. J. Ramadge, "Stochastic optimization of regenerative systems using infinitesimal perturbation analysis," IEEE Trans. Automat. Contr., vol. 39, pp. 1400-1410, Oct. 1994.
14. M. C. Fu and J. Hu, "Smoothed perturbation analysis derivative estimation for Markov chains," Oper. Res. Lett., vol. 15, pp. 241-251, 1994.
15. P. W. Glynn, "Likelihood ratio gradient estimation: An overview," in Proc. 1987 Winter Simulation Conf., 1987, pp. 366-375.
16. ____, "Optimization of stochastic systems via simulation," in Proc. 1987 Winter Simulation Conf., 1989, pp. 90-105.
17. Y. C. Ho and X.-R. Cao, "Perturbation analysis and optimization of queueing networks," J. Optim. Theory Applicat., vol. 40, no. 4, pp. 559-582, 1983.
18. Y. C. Ho and X. R. Cao, Perturbation Analysis of Discrete-Event Dynamic Systems. Norwell, MA: Kluwer, 1991.
19. L. H. Lee, E. T. K. Lau, and Y. C. Ho, "Explanation of goal softening in ordinal optimization," IEEE Trans. Automat. Contr., vol. 44, pp. 94-99, Jan. 1999.
20. T. Jaakkola, S. P. Singh, and M. I. Jordan, "Reinforcement learning algorithm for partially observable Markov decision problems," in Advances in Neural Information Processing Systems. San Francisco, CA: Morgan Kaufman, 1995, vol. 7, pp. 345-352.
21. V. R. Konda and V. S. Borkar, "Actor-ctritic-type learning algorithms for Markov decision processes," SIAM J. Control Optim., vol. 38, pp. 94-123, 1999.
22. P. R. Kumar, "Re-entrant lines," Queueing Syst.: Theory Applicat. vol. 1, pp. 87-110, 1993.
23. P. Marbach and T. N. Tsitsiklis, "Simulation-based optimization of Markov reward processes," IEEE Trans. Automat. Contr., vol. 46, pp. 191-209, Feb. 2001.
24. S. P. Meyn and R. L. Tweedie, Markov Chains and Stochastic Stability. London, U.K.: Springer-Verlag, 1993.
25. S. P. Meyn, "The policy improvement algorithm for Markov decision processes with general state space," IEEE Trans. Automat. Contr., vol. 42, pp. 1663-1680, Oct. 1997.
26. E. L. Plambeck, B. R. Fu, S. M. Robinson, and R. Suri, "Sample-path optimization of convex stochastic performance functions," Math. Program. B, vol. 75, pp. 137-176, 1996.
27. M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming. New York: Wiley, 1994.
28. M. I. Reiman and A. Weiss, "Sensitivity analysis via likelihood ratio," Oper. Res., vol. 37, pp. 830-844, 1989.
29. R. V. Rubinstein, Monte Carlo Optimization, Simulation, and Sensitivity Analysis of Queueing Networks. New York: Wiley, 1986.
30. R. Suri and Y. T. Leung, "Single run optimization of discrete event simulations-An empirical study using M/M/I queue," IIE Trans., vol. 21, pp. 35-49, 1989.
31. R. S. Sutton, "Learning to predict by the methods of temporal differences," Machine Learn., vol. 3, pp. 835-846, 1988.
32. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.
33. J. N. Tsitsiklis and V. R. Konda "Actor-critic algorithms," Cambridge, MA, Tech. Rep., Lab. Inform. Decision Systems, Mass. Inst. Technol., 2001.
34. J. N. Tsitsiklis and B. Van Roy, "Feature-based methods for large-scale dynamic programming," Machine Learn., vol. 22, pp. 59-4, 1994.
35. ____, "An analysis of temporal-difference learning with function approximation," IEEE Trans. Automat. Contr., vol. 42, pp. 674-690, May 1997.
36. ____, "Average cost temporal-difference learning," Automatica, vol. 35, pp. 1799-1808, 1999.
37. C. Watkins, "Learning from delayed rewards," Ph.D. dissertaton, Cambridge Univ., Cambridge, U.K., 1989.
38. C. Watkins and P. Dayan, "Q-learning," Machine Learn., vol. 8, pp. 279-292, 1992.