The policy iteration algorithm for average reward Markov decision processes with general state space

IEEE Transactions on Automatic Control

Volumn 42, Issue 12, 1997, Pages 1663-1680

  Meyn, Sean P ^a,b  

^a IEEE   (United States)

^b University of Illinois at Urbana Champaign   (United States)

Author keywords

Howard's algorithm; Markov decision processes; Multiclass queueing networks; Poisson equation; Policy iteration algorithm

Indexed keywords

ALGORITHMS; COMPUTATIONAL METHODS; COMPUTER NETWORKS; CONVERGENCE OF NUMERICAL METHODS; ITERATIVE METHODS; MARKOV PROCESSES; PROBLEM SOLVING; STATE SPACE METHODS; SYSTEM STABILITY;

HOWARD'S ALGORITHM; LINEAR QUADRATIC GAUSSIAN PROBLEM; MARKOV DECISION PROCESSES; MULTICLASS QUEUEING NETWORKS; POISSON EQUATION; POLICY ITERATION ALGORITHM;

OPTIMAL CONTROL SYSTEMS;

EID: 0031344030     PISSN: 00189286     EISSN: None     Source Type: Journal    
DOI: 10.1109/9.650016     Document Type: Article

References (49)

1. A. Arapostathis, V. S. Borkar, E. Fernandez-Gaucherand, M. K. Ghosh, and S. I. Marcus, "Discrete-time controlled Markov processes with average cost criterion: A survey," SIAM J. Contr. Optim., vol. 31, pp. 282-344, 1993.
2. V. S. Borkar, "Topics in controlled Markov chains," in Pitman Res. Notes in Math. Series # 240. UK: Longman Scientific & Technical, 1991.
3. R. Cavazos-Cadena, "Value iteration in a class of communicating Markov decision chains with the average cost criterion," Univ. Autónoma Agraria Anonio Narro, Tech. Rep., 1996.
4. R. Cavazos-Cadena and E. Fernandez-Gaucherand, "Value iteration in a class of average controlled Markov chains with unbounded costs: Necessary and sufficient conditions for pointwise convergence," J. Appl. Probability, vol. 33, pp. 986-1002, 1996.
5. F. Charlot and A. Nafidi, "Irréducibilité, petits ensembles, et stabilité des réseaux de Jackson généralisés," Univ. de Rouen UFR des Sciences, Tech. Rep., 1996.
6. R.-R. Chen and S. P. Meyn, "Value iteration and optimization of multiclass queueing networks," to be published.
7. J. Humphrey, D. Eng, and S. P. Meyn, "Fluid network models Linear programs for control and performance bounds," in Proc. 13th IFAC World Congr., J. Cruz, J. Gertler, and M. Peshkin, Eds, San Francisco, CA, 1996, vol. B, pp. 19-24.
8. J. Dai and G. Weiss, "Stability and instability of fluid models for certain re-entrant lines," Math. Ops. Res., vol. 21, no. 1, pp. 115-134, Feb. 1996.
9. J. G. Dai, "On the positive Harris recurrence for multiclass queueing networks A unified approach via fluid limit models," Ann. Appl. Probab., vol. 5, pp. 49-77, 1995.
10. J. G. Dai and S. P. Meyn, "Stability and convergence of moments for multiclass queueing networks via fluid limit models," IEEE Trans. Automat. Contr., vol. 40, pp. 1889-1904, Nov. 1995.
11. R. Dekker, "Counterexamples for compact action Markov decision chains with average reward criteria," Comm. Statist.-Stoch. Models, vol. 3, pp. 357-368, 1987.
12. C. Derman, "Dunumerable state MDPs," Ann. Amth. Statist., vol. 37, pp. 1545-1554, 1966.
13. D. Down, S. P. Meyn, and R. L. Tweedie, "Geometric and uniform ergodicity of Markov processes," Ann. Probab., vol. 23, no. 4, pp. 1671-1691, 1996.
14. M. Duflo, Méthodes Récursives Aléatoires. Masson, 1990.
15. E. B. Dynkin and A. A. Yushkevich, "Controlled Markov processes," in volume Grundlehren der mathematischen Wissenschaften 235 of A Series of Comprehensive Studies in Mathematics. New York: Springer-Verlag, 1979.
16. D. Bertsimas, F. Avram, and M. Ricard, "Fluid models of sequencing problems in open queueing networks: An optimal control approach," Massachusetts Inst. Technol., Tech. Rep., 1995.
17. S. Foss, "Ergodicity of queueing networks," Siberian Math. J., vol. 32, pp. 183-202, 1991.
18. P. W. Glynn and S. P. Meyn, "A Lyapunov bound for solutions of Poisson's equation," Ann. Probab., vol. 24, Apr. 1996.
19. O. Hernández-Lerma and J. B. Lasserre, "Policy iteration for average cost Markov control processes on Borel spaces," IPN, Departamento de Matematicas, Mexico, and LAAS-CNRS, France, Tech. Rep., 1995; Acta Applicandae Mathematicae, to be published.
20. _, Discrete Time Markov Control Processes I. New York: Springer-Verlag, 1996.
21. O. Hernández-Lerma, R. Montes-de-Oca, and R. Cavazos-Cadena, "Recurrence conditions for Markov decision processes with Borel state space: A survey," Ann. Operations Res., vol. 28, pp. 29-46, 1991.
22. A. Hordijk, Dynamic Programming and Markov Potential Theory, 1977.
23. A. Hordijk and M. L. Puterman, "On the convergence of policy iteration," Math. Ops. Res., vol. 12, pp. 163-176, 1987.
24. A. Hordijk, F. M. Spieksma, and R. L. Tweedie, "Uniform stability conditions for general space Markov decision processes," Leiden Univ. and Colorado State Univ., Tech. Rep., 1995.
25. R. A. Howard, Dynamic Programming and Markov Processes. New York: Wiley, 1960.
26. V. R. Konda and V. S. Borkar, "Learning algorithms for Markov decision processes," Indian Inst. Sci., Bangalore, Tech. Rep., 1996.
27. P. R. Kumar and S. P. Meyn, "Duality and linear programs for stability and performance analysis queueing networks and scheduling policies," IEEE Trans. Automat. Contr., vol. 41, pp. 4-17, Jan. 1996.
28. S. Kumar and P. R. Kumar, "Fluctuation smoothing policies are stable for stochastic re-entrant lines," in Proc. 33rd IEEE Conf. Decision Contr., Dec. 1994.
29. H. Kwakernaak and R. Sivan, Linear Optimal Control Systems. New York: Wiley-Intersci., 1972.
30. S. P. Meyn, "Transience of multiclass queueing networks via fluid limit models," Ann. Appl. Probab., vol. 5, pp. 946-957, 1995.
31. S. P. Meyn and D. Down, "Stability of generalized Jackson networks," Ann. Appl. Probab., vol. 4, pp. 24-148, 1994.
32. S. P. Meyn and R. L. Tweedie, "Generalized resolvents and Harris recurrence of Markov processes," Contemporary Math., vol. 149, pp. 227-250, 1993.
33. _, Markov Chains and Stochastic Stability. London: Springer-Verlag, 1993.
34. _, "Stability of Markovian processes III: Foster-Lyapunov criteria for continuous time processes," Adv. Appl. Probab., vol. 25, pp. 518-548, 1993.
35. S. P. Meyn, "Stability and optimization of multiclass queueing networks and their fluid models," in Proc. Summer Seminar on "The Math. Stochastic Manufacturing Syst.," Amer. Math. Soc., 1997.
36. E. Nummelin, General Irreducible Markov Chains and Non-Negative Operators. Cambridge: Cambridge Univ. Press, 1984.
37. _, "On the Poisson equation in the potential theory of a single kernel," Math. Scand., vol. 68, pp. 59-82, 1991.
38. J. Perkins, "Control of push and pull manufacturing systems," Ph.D. thesis, Univ. Illinois, Urbana, IL, Sept. 1993; Tech. Rep. UILU-ENG-93-2237 (DC-155).
39. M. L. Puterman, Markov Decision Processes. New York: Wiley, 1994.
40. R. K. Ritt and L. I. Sennott, "Optimal stationary policies in general state space Markov decision chains with finite action set," Math. Ops. Res., vol. 17, no. 4, pp. 901-909, Nov. 1993.
41. S. M. Ross, "Applied probability models with optimization applications," in Dover Books on Advanced Mathematics, 1992; republication of the work first published by Holden-Day, 1970.
42. L. I. Sennott, "A new condition for the existence of optimal stationary policies in average cost Markov decision processes," Ops. Res. Lett., vol. 5, pp. 17-23, 1986.
43. _, "Average cost optimal stationary policies in infini te state Markov decision processes with unbounded cost," Ops. Res., vol. 37, pp. 626-633, 1989.
44. _, "The convergence of value iteration in average cost Markov decision chains," Ops. Res. Lett., vol. 19, pp. 11-16, 1996.
45. J. N. Tsitsiklis and B. Van Roy, "An analysis of temporal-difference learning with function approximation," Massachusetts Inst. Technol., Cambridge, MA, Tech. Rep. LIDS-P-2322, Mar. 1996: also IEEE Trans. Automat. Contr..
46. P. Tuominen and R. L. Tweedie, "Subgeometric rates of convergence of f-ergodic Markov chains," Adv. Appl. Probab., vol. 26, pp. 775-798, 1994.
47. R. Weber and S. Stidham, "Optimal control of service rates in networks of queues," Adv. Appl. Probab., vol. 19, pp. 202-218, 1987.
48. G. Weiss, "On the optimal draining of re-entrant fluid lines," Georgia Inst. Technol. Technion, Tech. Rep., 1994.
49. P. Whittle, Risk-Sensitive Optimal Control. Chichester, NY: Wiley, 1990.