Value iteration in average cost Markov control processes on Borel spaces

Acta Applicandae Mathematicae

Volumn 42, Issue 2, 1996, Pages 203-222

  Montes de Oca, Raúl ^a   Hernández Lerma, Onésimo ^b  

^a DEPARTAMENTO DE QUÍMICA   (Mexico)

^b DEPARTAMENTO DE FÍSICA   (Mexico)

Author keywords

Average cost; Markov control (or decision) processes; Value iteration (or successive approximations)

Indexed keywords

EID: 0346144338     PISSN: 01678019     EISSN: None     Source Type: Journal    
DOI: 10.1007/BF00047169     Document Type: Article

References (42)

1. Arapostathis, A., Borkar, V., Fernández-Gaucherand, E., Ghosh, M. K., and Marcus, S. I.: Controlled Markov processes with an average cost criterion: a survey, SIAM J. Control Optim. 31 (1993), 282-344.
2. Ash, R. B.: Real Analysis and Probability, Academic Press, New York, 1972.
3. Bertsekas, D. P.: Dynamic Programming: Deterministic and Stochastic Models, Prentice-Hall, Englewood Cliffs, N.J., 1987.
4. Diebolt, J. and Guégan, D.: Probabilistic properties of the general nonlinear Markovian process of order one and applications to time series modelling, Rapport Technique No. 125, Laboratoire de Statistique Théorique et Appliquée, CNRS-URA 1321, Université Paris VI, 1990.
5. Dugundji, J.: Topology, Allyn and Bacon, Boston, 1966.
6. Dynkin, E. B. and Yushkevich, A. A.: Controlled Markov Processes, Springer-Verlag, Berlin, 1979.
7. Fernández-Gaucherand, E., Marcus, S. I., and Arapostathis, A.: Convex stochastic control problems, Proc. 31st IEEE-CDC, Tucson, AZ, Dec. 1992.
8. Flynn, J.: On optimality criteria for dynamic programs with long finite horizons, J. Math. Anal. Appl. 76 (1980), 202-208.
9. Georgin, J.-P.: Contrôle de chaînes de Markov sur des espaces arbitraires, Ann. Inst. Henri Poincaré, Sect. B 23 (1978), 255-277.
10. Gordienko, E. and Hernández-Lerma, O.: Average cost Markov control processes with weighted norms: existence of canonical policies, Reporte Interno # 157, Departamento de Matemáticas, CINVESTAV-IPN, 1994, to appear in Applic. Math.
11. Gordienko, E. and Hernández-Lerma, O.: Average cost Markov control processes with weighted norms: value iteration; Reporte Interno # 158, Departamento de Matemáticas, CINVESTAV-IPN, to appear in Applic. Math. 23 (1995).
12. Hernández-Lerma, O.: Adaptive Markov Control Processes, Springer-Verlag, New York, 1989.
13. Hernández-Lerma, O.: Average optimality in dynamic programming on Borel spaces - unbounded costs and controls, Systems Control Lett. 17 (1991), 237-242.
14. Hernández-Lerma, O.: Existence of average optimal policies in Markov control processes with strictly unbounded costs, Kybernetika (Prague) 29 (1993), 1-17.
15. Hernández-Lerma, O. and Cavazos-Cadena, R.: Density estimation and adaptive control of Markov processes: average and discounted criteria, Acta Appl. Math. 20 (1990), 285-307.
16. Hernández-Lerma, O., Hennet, J. C., and Lasserre, J. B.: Average cost Markov decision processes: optimality conditions, J. Math. Anal. Appl. 158 (1991), 396-406.
17. Hernández-Lerma, O. and Lasserre, J. B.: Value iteration and rolling plans for Markov control processes with unbounded rewards, J. Math. Anal. Appl. 177 (1993), 38-55.
18. Hernández-Lerma, O. and Lasserre, J. B.: Linear programming and average optimality of Markov control processes on Borel spaces - unbounded costs, SIAM J. Control Optim. 32 (1994), 480-500.
19. Hernández-Lerma, O. and Lasserre, J. B.: Invariant probabilities for Feller-Markov chains, Reporte Interno No. 122, CINVESTAV-IPN, México, 1993, to appear in J. Appl. Math. Stoch. Anal.
20. Hernández-Lerma, O. and Marcus, S. I.: Discretization procedures for adaptive Markov control processes, J. Math. Anal. Appl. 137 (1989), 485-514.
21. Hernández-Lerma, O., Montes-de-Oca, R., and Cavazos-Cadena, R.: Recurrence conditions for Markov decision processes with Borel state space: a survey, Ann. Oper. Res. 28 (1991), 29-46.
22. Hernández-Lerma, O. and Muñoz de Ozak, M.: Discrete-time Markov control processes with discounted unbounded costs: optimality criteria, Kybernetika (Prague) 28 (1992), 191-212.
23. Hernández-Lerma, O. and Runggaldier, J. W.: Monotone approximations for convex stochastic control problems, J. Math. Systems, Estimation Control 4 (1994), 99-140.
24. Hinderer, K.: Foundations of Non-Stationary Dynamic Programming with Discrete Time Parameter, Lecture Notes Oper. Res. 33, Springer-Verlag, New York, 1970.
25. Mokkadem, A.: Sur un modèle autorégressif nonlinéare: ergodicité et ergodicité géométrique, J. Time Series Anal. 8 (1987), 195-205.
26. Montes-de-Oca, R.: The average cost optimality equation for Markov control processes on Borel spaces, Systems Control Lett. 22 (1994), 351-357.
27. Montes-de-Oca, R. and Hernández-Lerma, O.: Conditions for average optimality in Markov control processes with unbounded costs and controls, J. Math. Systems, Estimation Control 4 (1994), 1-19.
28. Montes-de-Oca, R., Minjárez-Sosa, J. A., and Hernández-Lerma, O.: Conditions for average optimality in Markov control processes on Borel spaces, Boletín Soc. Mat. Mexicana 39 (1994), 39-50.
29. Orey, S.: Limit Theorems for Markov Chains Transition Probabilities, Van Nostrand-Reinhold, London, 1971.
30. Puterman, M. L.: Markov Decision Processes, Wiley, New York, 1994.
31. Rieder, U.: Measurable selection theorems for optimization problems, Manuscripta Math. 24 (1978), 507-518.
32. Rieder, U.: On non-discounted dynamic programming with arbitrary state space, Preprint, University of Ulm, 1979.
33. Royden, H. L.: Real Analysis, 3rd. edn, Macmillan, New York, 1968.
34. Schäl, M.: Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal, Z. Wahrsch. verw. Geb. 32 (1975), 179-196.
35. Schäl, M.: Average optimality in dynamic programming with general state space, Math. Oper. Res. 18 (1993), 163-172.
36. Sennott, L. I.: Value iteration in countable state average cost Markov decision processes with unbounded costs, Ann. Oper. Res. 29 (1991), 261-271.
37. Sennott, L. I.: The average cost optimality equation and critical number policies, Prob. Eng. Inform. Sci. 7 (1993), 47-67.
38. Tijms, H. C.: On dynamic programming with arbitrary state space, compact action space and the average reward as criterion, Report BW 55/75, Mathematisch Centrum, Amsterdam, 1975.
39. Tweedie, R. L.: Invariant measures for Markov chains with no irreducibility assumptions, J. Appl. Prob. 25A (1988), 275-285.
40. White, D. J.: Dynamic programming, Markov chains, and the method of successive approximations, J. Math. Anal. Appl. 6 (1963), 373-376.
41. Wijngaard, J.: Existence of average optimal strategies in markovian decision problems with strictly unbounded costs, in: M. L. Puterman (ed.), Dynamic Programming and Its Applications, Academic Press, New York, 1978, pp. 369-386.
42. Yushkevich, A. A.: On a class of strategies in general Markov decision models, Theory Probab. Appl. 18 (1973), 777-779.