Value iteration in a class of average controlled markov chains with unbounded costs: Necessary and sufficient conditions for pointwise convergence

Journal of Applied Probability

Volumn 33, Issue 4, 1996, Pages 986-1002

  Cavazos Cadena, Rolando ^a   Fernández Gaucherand, Emmanuel ^b  

^a UNIVERSIDAD AUTÓNOMA AGRARIA ANTONIO NARRO   (Mexico)

^b University of Arizona   (United States)

Author keywords

Average cost criterion; Controlled Markov chains; Lyapunov function condition; Necessary and sufficient conditions; Pointwise convergence; Value iteration scheme

Indexed keywords

EID: 0040898816     PISSN: 00219002     EISSN: None     Source Type: Journal    
DOI: 10.1017/S0021900200100427     Document Type: Article

References (22)

1. ARAPOSTATHIS, A., BORKAR, V. S., FERNÁNDEZ-GAUCHERAND, E., GOSH, M. K. AND MARCUS, S. I. (1993) Discrete-time controlled Markov processes with an average cost criterion: a survey. SIAM J. Control Optim. 31, 282-344.
2. BORKAR, V. S. (1991) Topics in Controlled Markov Chains (Pitman Research Notes in Mathematics Series 240). Longman, London.
3. CAVAZOS-CADENA, R. (1991) Recent results on conditions for the existence of average optimal stationary policies. Ann. Operat. Res. 28, 3-28.
4. CAVAZOS-CADENA, R. (1994) Cesàro convergence of the undiscounted value iteration method in Markov decision processes under the Lyapunov stability condition. Boletin de la Sociedad Mathemática Mexicana. To appear.
5. CAVAZOS-CADENA, R. (1995) Undiscounted value iteration in stable Markov decision processes with bounded rewards. J. Math. Systems, Estimation and Control. To appear.
6. CAVAZOS-CADENA, R. AND FERNÁNDEZ-GAUCHERAND, E. (1995) Denumerable controlled Markov chains with average reward criterion: sample path optimality. ZOR-Math. Meth. Operat. Res. 41, 89-108.
7. CAVAZOS-CADENA, R. AND HERNÁNDEZ-LERMA, O. (1992) Equivalence of Lyapunov stability criteria in a class of Markov decision processes. Appl. Math. Optim. 26, 113-137.
8. DUGUNDJI, J. (1966) Topology. Allyn and Bacon, Boston.
9. FEDERGRUEN, A. AND SCHWEITZER, P. J. (1978) Discounted and undiscounted value iteration in Markov decision problems: A survey. In Dynamic Programming and its Applications. ed. M. L. Puterman. Academic Press, New York. pp. 23-52.
10. FEDERGRUEN, A. AND SCHWEITZER, P. J. (1980) A survey of asymptotic value-iteration for undiscounted Markovian decision processes. In Recent Developments in Markov Decision Processes. ed. R. Hartley, L. C. Thomas and D. J. White. Academic Press, New York. pp. 73-109.
11. HERNÁNDEZ-LERMA, O. (1989) Adaptive Markov Control Processes. Springer, New York.
12. HERNÁNDEZ-LERMA, O. AND LASSERRE, J. B. (1993) Value iteration and rolling plans for Markov control processes with unbounded rewards. J. Math. Anal Appl. 177, 38-55.
13. HINDERER, K. (1970) Foundations of Non-Stationary Dynamic Programming with Discrete Time Parameter. Springer, New York.
14. HORDUK, A. (1974) Dynamic Programming and Markov Potential Theory. (Mathematical Centre Tract 51.) Mathematisch Centrum, Amsterdam.
15. HORDIJK, A, SCHWEITZER, P. J. AND TUMS, H. C. (1975) The asymptotic behavior of the minimal total expected cost for the denumerable state Markov decision model. J. Appl. Prob. 12, 298-305.
16. MONTES-DE-OCA, R. AND HERNÁNDEZ-LERMA, O. (1993) Value iteration in average cost Markov control processes on Borel spaces. Submitted.
17. PUTERMAN, M. (1994) Markov Decision Processes. Wiley, New York.
18. ROYDEN, H. L. (1968) Real Analysis. 2nd edn. Macmillan, New York.
19. Ross, S. M. (1970) Applied Probability Models with Optimization Applications Holden-Day, San Francisco.
20. SCHWEITZER, P. J. (1971) Iterative solution of the functional equations for undiscounted Markov renewal programming. J. Math. Anal. Appl. 34, 495-501.
21. SENNOTT, L. I. (1991) Value iteration in countable state average cost Markov decision processes with unbounded costs. Ann. Operat. Res. 28, 261-272.
22. THOMAS, L. C. (1965) Connectedness conditions for denumerable state Markov decision processes. In Recent Developments in Markov Decision Processes, ed. R. Hartley, L. C. Thomas and D. J. White. Academic Press, New York. pp. 181-204.