Constrained Markov decision processes with total cost criteria: Lagrangian approach and dual linear program

Mathematical Methods of Operations Research

Volumn 48, Issue 3, 1998, Pages 387-417

  Altman, Eitan ^a  

^a INRIA   (France)

Author keywords

Countable state space; Extra constraints; Markov decision process; Total cost

Indexed keywords

EID: 1942424978     PISSN: 14322994     EISSN: None     Source Type: Journal    
DOI: 10.1007/s001860050035     Document Type: Article

References (77)

1. Altman E (1993) Asymptotic properties of constrained Markov decision processes. ZOR 37:151-170
2. Altman E (1994) Denumerable constrained Markov Decision Processes and finite approximations. Math. of Operations Research 19:169-191
3. Altman E (1996) Constrained Markov decision processes with total cost criteria: Occupation measures and primal LP. ZOR 43:45-72
4. Altman E (1995) Constrained Markov decision processes. Book in preparation
5. Altman E, Shwartz A (1991a) Markov decision problems and state-action frequencies. SIAM J. Control and Optimization 29:786-809
6. Altman E, Shwartz A (1993) Time-sharing policies for controlled Markov chains. Operations Research 41:1116-1124
7. Altman E, Shwartz A (1995) Constrained Markov games: Nash equilibria. Annals of dynamic games, submitted
8. Altman E, Spieksma F (1995) The linear program approach in Markov decision problems revisited. ZOR 42:169-188
9. Altman E, Zeitouni O (1994) Rate of convergence of empirical measures and costs in controlled Markov chains and transient optimality. Math, of Operations Research 19:955-974
10. Anderson J, Nash P (1987) Linear programming in infinite-dimentional spaces. Wiley, England
11. Arapostathis A, Borkar VS, Fernández-Gaucherand E, Ghosh MK, Marcus SI (1993) Discrete-time controlled Markov processes with average cost criterion: A survey. SIAM J. Control and Optimization 31:282-344
12. Aubin JP (1993) Optima and equilibria, An introduction to nonlinear analysis. Springer-Verlag
13. Bayal-Gursoy M, Ross KW (1992) Variability sensitive Markov decision processes. Math, of Operations Research 17:558-571
14. Billingsley P (1968) Convergence of probability measures. Wiley, New-York
15. Beutler FJ, Ross KW (1985) Optimal policies for controlled Markov chains with a constraint. J. Mathematical Analysis and Applications 112:236-252
16. Beutler FJ, Ross KW (1986) Time-average optimal constrained semi-Markov decision processes. Advances of Applied Probability 18:341-359
17. Borkar VS (1988) A convex analytic approach to Markov decision processes. Prob. Th. Rel. Fields 78:583-602
18. Borkar VS (1990) Topics in controlled Markov chains. Longman Scientific & Technical
19. Borkar VS (1994) Ergodic control of Markov Chains with constraints - the general case. SIAM J. Control and Optimization 32:176-186
20. Cavazos-Cadena R, Hernaández-Lerma O (1992) Equivalence of Lyapunov stability criteria in a class of Markov decision processes. Appl. Math. Optim. 26:113-137
21. De Ghellinck GT (1960) Les problèmes de décisions séquentielles. Cahiers du Centre de Recherche Opérationelle 2:161-179
22. Denardo EV (1970a) On linear programming in a Markov decision problem. Management Science 16:281-288
23. Denardo EV, Fox BL (1968) Multichain Markov renewal programs. SIAM J. of Applied Math. 16:468-487
24. D'Epenoux F (1960) Sur un problème de production et de stockage dans l'aléatoire. Revue Française de Recherche Opérationelle 14:3-16
25. D'Epenoux F (1963) A probabilistic production and inventory problem. Management Science 10:98-108
26. Derman C (1970) Finite state Markovian decision processes. Academic Press
27. Derman C, Klein M (1965) Some remarks on finite horizon Markovian decision models. Operations research 13:272-278
28. Derman C, Strauch RE (1966) On memoryless rules for controlling sequential control processes. Ann. Math. Stat 37:276-278
29. Derman C, Veinott AF Jr (1972) Constrained Markov decision chains. Management Science 19:389-390
30. Doob JL (1994). Measure theory, Springer-Verlag
31. Dynkin E, Yushkevich A (1979) Controlled Markov processes. Springer-Verlag, Berlin
32. Federgruen A (1979) Geometric convergence of value-iteration in multichain Markov decision problems. Adv. Appl. Prob. 11:188-217
33. Feinberg EA (1986) Sufficient classes of strategies in discrete dynamic programming I: Decomposition of randomized strategies and embedded models. SIAM Theory Probab. Appl., 31:658-668
34. Feinberg EA (1995) Constrained semi-Markov decision processes with average rewards. ZOR 39:257-288
35. Feinberg EA, Reiman MI (1994) Optimality of randomized trunk reservation. Probability in the Engineering and Informational Sciences 8:463-489
36. Feinberg EA, Sonin I (1983) Stationary and Markov policies in countable state dynamic programming. Lecture notes in Mathematics 1021:111-129
37. Feinberg EA, Sonin I (1993) The existence of an equivalent stationary strategy in the case of discount factor equal one. Unpublished Draft
38. Feinberg EA, Sonin I (1995) Notes on equivalent stationary policies in Markov decision processes with total rewards. ZOR 44:205-221
39. Feinberg EA, Shwartz A (1995) Constrained discounted dynamic programming. Math. of Operations Research 20:302-320
40. Filar JA, Kallenberg LCM, Lee HM (1989) Varinace-penalized Markov decision processes. Math. of Operations Research, 14:147-161
41. Filar JA, Lee HM (1985) Gain/variability tradeoffs in undiscounted Markov decision processes. Proceedings of 24th Conference on Decision and Control IEEE, pp. 1106-1112
42. Fisher L, Ross SM (1968) An example in denumerable decision processes. Ann. Math. Stat. 39:674-675
43. Heilmann WR (1977) Generalized linear programming in Markovian decision problems. Bonner Math. Schriften 98:33-39
44. Heilmann WR (1978) Solving stochastic dynamic programming problems by linear programming - an annotated bibliography. Z. Oper. Res. 22:43-53
45. Hernández-Lerma O, Hernández-Hernández D (1994) Discounted cost Markov decision processes on Borel spaces: The linear programming formulation. J. of Math. Anal. and Appl. 183:335-351
46. Hernández-Lerma O, Lasserre JB (1994) Linear programming and average optimality on Borel spaces-unbounded costs. SIAM J. Control and Optimization 32:480-500
47. Hinderer K (1970) Foundation of non-stationary dynamic programming with discrete time parameter. Vol. 33, Lecture Notes in Operations Research and Mathematical Systems, Springer-Verlag, Berlin
48. Hordijk A (1977) Dynamic programming and Markov potential theory. Second Edition, Mathematical Centre Tracts 51, Mathematisch Centrum, Amsterdam
49. Hordijk A, Kallenberg LCM (1979) Linear programming and Markov decision chains. Management Science 25:352-362
50. Hordijk A, Kallenberg LCM (1984) Constrained undiscounted stochastic dynamic programming. Mathematics of Operations Research 9:276-289
51. Hordijk A, Lasserre JB (1994) Linear programming formulation of MDPs in countable state space: The multichain case. ZOR Research 40:91-108
52. Huang Y, Kallenberg LCM (1994) On finding optimal policies for Markov decision chains: A unifying framework for mean-variance tradeoffs. Math. of Operations Research 19:434-448
53. Kallenberg LCM (1983) Linear programming and finite Markovian control problems. Mathematical Centre Tracts 148, Amsterdam
54. Kallenberg LCM (1994) Survey of linear programming for standard and nonstandard Markovian control problems, Part I: Theory. ZOR 40:1-42
55. Kawai H (1987) A variance minimization problem for a Markov decision process. European Journal of operations Research 31:140-145
56. Krylov N (1985) Once more about the connection between elliptic operators and Ito's stochastic equations. In: Krylov N et al. (eds.) Statistics and control of stochastic processes, Steklov Seminar 1984 Optimization Software, New York, pp. 69-101
57. Kushner H, Kleinman J (1971) Mathematical programming and the control of Markov chains. Internat. J. Control 13:801-820
58. Lasserre JB (1994) Average optimal stationary policies and Linear programming in countable state Markov decision processes. J. Math. Anal. Appl. 183:233-249
59. Manne AS (1960) Linear programming and sequential decisions. Management Science 6:259-267
60. Piunovskiy AB (1997) Optimal Control of Random Sequences in Problems with Constraints. Mathematics and its Applications, Kluwer Academic Publishers
61. Puterman M (1994), Markov decision processes. John Wiley & Sons, New York
62. Raghavan TES, Filar JA (1991) Algorithms for stochastic games - a survey. ZOR 35:437-472
63. Rockafellar RT (1989) Conjugate duality and optimization. Society for Industrial and Applied Mathematics, 2nd printing, Philadelphia
64. Ross KW (1989) Randomized and past-dependent policies for Markov decision processes with multiple constraints. Operations Research 37:474-477
65. Ross K, Varadarajan R (1989) Markov decision processes with sample path constraints: The communicating case. Operations Research 37:780-790
66. Ross K, Varadarajan R (1991) Multichain Markov decision processes with a sample path constraint: A decomposition approach. Math. of Operations Research 16:195-207
67. Royden HL (1988) Real analysis. 3rd Edition, Macmillan publishing Company, New York
68. Shapley LS (1953) Stochastic games. Proceedings Nat. Acad. of Science USA 39:1095-1100
69. Sennott LI (1991) Constrained discounted Markov decision chains. Probability in the Engineering and Informational Sciences 5:463-475
70. Sennott LI (1993) Constrained average cost Markov decision chains. Probability in the Engineering and Informational Sciences 7:69-83
71. Sobel MJ (1985) Maximal mean/standard deviation ratio in undiscounted MDP. OR Letters 4:157-159
72. Sobel MJ (1994) Mean-variance tradeoffs in an undiscounted MDP. Operations Research 42:175-188
73. Spieksma FM (1990) Geometrically ergodic Markov chains and the optimal control of queues. Ph.D. thesis, University of Leiden
74. Tidball M, Altmian A (1996) Continuity of optimal values and solutions of convex optimization, and constrained control of Markov chains. SIAM J. Control and Optimization, submitted
75. Van Der Wal J (1981) Stochastic dynamic programming. Mathematical Centre Tract 139, Mathematisch Centrum, Amsterdam
76. White DJ (1994) A mathematical programming approach to a problem in variance penalised Markov decision processes. OR Spectrum 15:225-230
77. Williams D (1992) Probability and martingales. Cambridge University Press