Value iteration in a class of communicating Markov decision chains with the average cost criterion

SIAM Journal on Control and Optimization

Volumn 34, Issue 6, 1996, Pages 1848-1873

  Cavazos Cadena, Rolando ^a  

^a UNIVERSIDAD AUTÓNOMA AGRARIA ANTONIO NARRO   (Mexico)

Author keywords

Almost monotone cost function; Average cost criterion; Markov decision chains; Pointwise convergence; Value iteration scheme

Indexed keywords

CONTROL SYSTEM ANALYSIS; CONTROL THEORY; FUNCTIONS; ITERATIVE METHODS; MARKOV PROCESSES; STATE SPACE METHODS;

AVERAGE COST CRITERION; POINTWISE CONVERGENCE;

CONTROL;

EID: 0030290475     PISSN: 03630129     EISSN: None     Source Type: Journal    
DOI: 10.1137/S1064827590192863     Document Type: Article

References (30)

1. V. S. BORKAR, Controlled Markov chains and stochastic networks, SIAM J. Control Optim., 21 (1983), pp. 652-666.
2. _, On minimum cost per unit of time control of Markov chains, SIAM J. Control Optim., 22 (1984), pp. 965-978.
3. R. CAVAZOS-CADENA, Weak conditions for the existence of average optimal stationary policies in average Markov decision chains with unbounded costs, Kybernetika, 25 (1989), pp. 145-156.
4. _, Solution to the optimality equation in a class of Markov decision chains with the average cost criterion, Kybernetika, 27 (1991), pp. 26-37.
5. _, Recent results on conditions for the existence of average optimal stationary policies, Ann. Oper. Res., 28 (1991), pp. 3-28.
6. R. CAVAZOS-CADENA AND L. I. SENNOTT, Comparing recent assumptions for the existence of optimal stationary policies, Oper. Res. Lett., 11 (1992), pp. 33-37.
7. R. CAVAZOS-CADENA, Undiscounted value iteration in stable Markov decision chains with bounded rewards, J. Math. Systems Estim. Control, 6 (1994), pp. 243-246.
8. _, Cesàro convergence of the undiscounted value iteration method in Markov decision processes under the Lyapunov stability condition. Bol. Soc. Mat. Mexicana (2), 38 (1993), pp. 33-46.
9. _, Value Iteration in Controlled Markov Chains with Penalized Costs: Cesàro Convergence Results, Report 03-91-DEC, Universidad Autónoma Agraria Antonio Narro, Saltillo Coah, México, 1991.
10. R. CAVAZOS-CADENA AND E. FERNÁNDEZ-GAUCHERAND, Value iteration in stable Markov decision chains with unbounded costs: Necessary and sufficient conditions for pointwise convergence, J. Appl. Probab., (1996), to appear.
11. J. DUGUNDJI, Topology, Allyn and Bacon, Boston, 1966.
12. A. FEDERGRUEN AND P. J. SCHWEITZER, A survey of asymptotic value iteration for undiscounted Markovian decision processes, in Recent Developments in Markov Decision Processes, R. Hartley, L. C. Thomas, and D. J. White, eds., Academic Press, New York, 1980.
13. M. K. GOSH AND S. I. MARCUS, On strong average optimality of Markov decision processes with unbounded costs, Oper. Res. Lett., 11 (1992), pp. 99-104.
14. O. HERNÁNDEZ-LERMA, Adaptive Markov Control Processes, Springer-Verlag, New York, 1989.
15. K. HINDERER, Foundations of Non-Stationary Dynamic Programming with Discrete Time Parameter, Lecture Notes in Oper. Res. 33, Springer-Verlag, New York, 1970.
16. A. HORDIJK, Dynamic Programming and Markov Potential Theory, Mathematical Centre Tract 51, Mathematisch Centrum, Amsterdam, 1974.
17. A. HORDIJK, P. J. SCHWEITZER, AND H. C. TIJMS, The asymptotic behavior of the minimal total expected cost for the denumerable state Markov decision model, J. Appl. Probab., 12 (1975), pp. 298-305.
18. M. LOÈVE, Probability Theory I, Springer-Verlag, New York, 1978.
19. R. MONTES-DE-OCA AND O. HERNÁNDEZ-LERMA, Value iteration in average cost Markov control processes on Borel spaces, Acta Appl. Math., 42 (1996), pp. 203-221.
20. M. L. PUTERMAN, Markov Decision Processes, Wiley, New York, 1994.
21. H. L. ROYDEN, Real Analysis, 2nd ed., Macmillan, New York, 1968.
22. S. M. ROSS, Applied Probability Models with Optimization Applications, Holden-Day, San Francisco, 1970.
23. L. I. SENNOTT, Average cost optimal stationary policies in infinite state Markov decision processes with unbounded costs, Oper. Res., 37 (1989), pp. 626-633.
24. _, Value iteration in countable state Markov decision processes with unbounded costs, Ann. Oper. Res., 28 (1991), pp. 261-272.
25. _, The convergence of value iteration in average cost Markov decision chains, (1994), submitted.
26. P. J. SCHWEITZER AND A. FEDERGRUEN, The asymptotic behavior of undiscounted value iteration in Markov decision proceses, Math. Oper. Res., 2 (1977), pp. 360-381.
27. P. J. SCHWEITZER, Iterative solution of the functional equations for undiscounted Markov renewal programming, J. Math. Anal. Appl., 34 (1971), pp. 495-501.
28. L. C. THOMAS, Connectedness conditions for denumerable state Markov decision processes, in Recent Developments in Markov Decision Processes, R. Hartley, L. C. Thomas, and D. J. White, eds., Academic Press, New York, 1980.
29. R. R. WEBER AND S. STIDHAM, Optimal control of service rates in networks of queues, Advances in Applied Probability, 19 (1987), pp. 202-218.
30. D. J. WHITE, Dynamic programming, Markov chains, and the method of succesive approximations, J. Math. Anal. Appl., (1963), pp. 373-376.