A cost-shaping linear program for average-cost approximate dynamic programming with performance guarantees

Mathematics of Operations Research

Volumn 31, Issue 3, 2006, Pages 597-620

  De Farias, Daniela Pucci ^a   Van Roy, Benjamin ^b  

^a Massachusetts Institute of Technology Cambridge   (United States)

^b STANFORD UNIVERSITY   (United States)

Author keywords

Approximate dynamic programming; Average cost; Linear programming

Indexed keywords

APPROXIMATE DYNAMIC PROGRAMMING; AVERAGE COSTS; DIFFERENTIAL COSTS; MARKOV DECISION PROCESSES (MDP);

ALGORITHMS; DYNAMIC PROGRAMMING; FUNCTIONS; LYAPUNOV METHODS; MARKOV PROCESSES; OPTIMIZATION; PROBLEM SOLVING; QUEUEING THEORY;

LINEAR PROGRAMMING;

EID: 33748414214     PISSN: 0364765X     EISSN: 15265471     Source Type: Journal    
DOI: 10.1287/moor.1060.0208     Document Type: Article

References (36)

1. Adelman, D. 2004. A price-directed approach to stochastic inventory/routing. Oper. Res. 52(4) 499-514.
2. Bertsekas, D. P. 2001. Dynamic Programming and Optimal Control, 2nd ed. Athena Scientific, Belmont, MA.
3. Bertsekas, D. P., J. N. Tsitsiklis. 1996. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA.
4. Borkar, V. 2001. Convex analytic methods in Markov decision processes. E. Feinberg, A. Shwartz, eds. Handbook of Markov Decision Processes: Methods and Applications. Kluwer, Boston, MA.
5. Boyan, J. A., A. W. Moore. 1995. Generalization in reinforcement learning: Safely approximating the value function. Advances in Neural Information Processing Systems, Vol. 7. MIT Press, Cambridge, MA.
6. Chen, R.-R., S. Meyn. 1999. Value iteration and optimization of multiclass queueing networks. Queueing Systems 32 65-97.
7. de Farias, D. P., B. Van Roy. 2004. On constraint sampling in the linear programming approach to approximate dynamic programming. Math. Oper. Res. 29(3) 462-478.
8. de Farias, D. P., B. Van Roy. 2003. Approximate linear programming for average-cost dynamic programming. Advances in Neural Information Processing Systems, Vol. 15. MIT Press, Cambridge, MA.
9. de Farias, D. P., B. Van Roy. 2003. The linear programming approach to approximate dynamic programming. Oper. Res. 51(6) 850-865.
10. D'Epenoux, P. 1963. A probabilistic production and inventory problem. Management Sci. 10(1) 98-108.
11. Parias, V. F., B. Van Roy. 2006. Terris: A study of randomized constraint sampling. G. Calafiore, P. Dabbene, eds. Probabilistic and Randomized Methods for Design Under Uncertainty. Springer-Verlag, London, UK.
12. Gordon, G. J. 1995. Stable function approximation in dynamic programming. Technical Report CMU-CS-95-103, Carnegie Mellon University, Pittsburgh, PA.
13. Gordon, G. J. 1995. Stable function approximation in dynamic programming. Machine Learning: Proc. Twelfth Internat. Conf. (ICML), San Francisco, CA.
14. Gordon, G. J. 1999. Approximate solutions to Markov decision processes. Ph.D. thesis, Carnegie Mellon University, Pittsburgh, PA.
15. Guestrin, C. 2003. Planning under uncertainty in complex structured environments. Ph.D. thesis, Stanford University, Stanford, CA.
16. Guestrin, C., M. Hauskrecht, B. Kveton. 2004. Solving factored MDPs with continuous and discrete variables. Twentieth Conf. Uncertainty in Artificial Intelligence, Banff, Alberta, Canada.
17. Guestrin, C., D. Koller, R. Parr. 2003. Efficient solution algorithms for factored MDPs. J. Artificial Intelligence Res. 19 399-468.
18. Harmon, M. E., L. C. Baird, A. H. Klopf. 1995. Advantage updating applied to a differential game. Advances in Neural Information Processing Systems, Vol. 7. MIT Press, Cambridge, MA.
19. Hauskrecht, M., B. Kveton. 2003. Linear program approximations to factored continuous-state Markov decision processes. Advances in Neural Information Processing Systems, Vol. 17. MIT Press, Cambridge, MA.
20. Henderson, S. G., S. P. Meyn, V. B. Tadić. 2003. Performance evaluation and policy selection in multiclass networks. Discrete Event Dynam. Systems: Theory Appl. 13(Special issue on learning, optimization and decision making (invited)) 149-189.
21. Hernández-Lerma, O., J. B. Lasserre. 2001. The linear programming approach. E. Feinberg, A. Shwartz, eds. Handbook of Markov Decision Processes: Methods and Applications. Kluwer, Boston, MA.
22. Kartashov, N. V. 1985. Criteria for uniform ergodicity and strong stability of Markov chains with a common phase space. Theory Probab. Appl. 30 71-89.
23. Kartashov, N. V. 1985. Inequalities in theorems of ergodicity and stability for Markov chains with a common phase space. Theory Probab. Appl. 30 247-259.
24. Kearns, M., S. Singh. 2002. Near-optimal reinforcement learning in polynomial time. Machine Learning 49(2) 209-232.
25. Manne, A. S. Linear programming and sequential decisions. Management Sci. 6(3) 259-267.
26. Meyn, S. P. 1997. The policy iteration algorithm for average reward Markov decision processes with general state space. Trans. Automatic Control 42(12) 1663-1680.
27. Meyn, S. P. 2005. Workload models for stochastic networks: Value functions and performance evaluation. IEEE Trans. Automatic Control 50(8) 1106-1122.
28. Meyn, S. P., R. Tweedie. 1993. Markov Chains and Stochastic Stability. Springer-Verlag.
29. Morrison, J. R., P. R. Kumar. 1999. New linear program performance bounds for queueing networks. J. Optim. Theory Appl. 100(3) 575-597.
30. Munos, R. 2003. Error bounds for approximate policy iteration. Machine Learning: Proc. Twentieth Internat. Conf. (ICML). AAAI Press, Menlo Park, CA.
31. Puterman, M. L. 1994. Markov Decision Processes. John Wiley & Sons, New York.
32. Schuurmans, D., R. Patrascu. 2001. Direct value-approximation for factored MDPs. Advances in Neural Information Processing Systems, Vol. 14. MIT Press, Cambridge, MA.
33. Schweitzer, P. J., A. Seidman. 1985. Generalized polynomial approximation in Markov decision processes. J. Math. Anal. Appl. 110 568-582.
34. Trick, M., S. Zin. 1997. Spline approximations to value functions: A linear programming approach. Macroeconomic Dynam. 1(1) 255-277.
35. Tsitsiklis, J. N., B. Van Roy. 1996. Feature-based methods for large-scale dynamic programming. Machine. Learning 22 59-94.
36. Veatch, M. H. 2005. Approximate dynamic programming for networks: Fluid models and constraint reduction. Submitted for publication.