Solving factored MDPs with hybrid state and action variables

Journal of Artificial Intelligence Research

Volumn 27, Issue , 2006, Pages 153-201

  Kveton, Branislav ^a   Hauskrecht, Milos ^b   Guestrin, Carlos ^c  

^a UNIVERSITY OF PITTSBURGH   (United States)

^b Learning Research and Development Ctr   (United States)

^c NONE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION THEORY; DECISION SUPPORT SYSTEMS; FUNCTIONS; LINEAR PROGRAMMING; OPTIMIZATION; PROBLEM SOLVING;

DISCRETE VARIABLES; HYBRID APPROXIMATE LINEAR PROGRAMMING (HALF); MARKOV DECISION PROCESS (MDP) MODELS; OPTIMIZATION PROBLEMS;

MARKOV PROCESSES;

EID: 33750586671     PISSN: 10769757     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.2085     Document Type: Article

References (68)

1. Andrieu, C., de Freitas, N., Doucet, A., &; Jordan, M. (2003). An introduction to MCMC for machine learning. Machine Learning, 50, 5-43.
2. Astrom, K. (1965). Optimal control of Markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10(1), 174-205.
3. Bellman, R. (1957). Dynamic Programming. Princeton University Press, Princeton, NJ.
4. Bellman, R., Kalaba, R., & Kotkin, B. (1963). Polynomial approximation - a new computational technique in dynamic programming: Allocation processes. Mathematics of Computation, 17(82), 155-161.
5. Bertsekas, D. (1995). A counterexample for temporal differences learning. Neural Computation, 7(2), 270-279.
6. Bertsekas, D., & Tsitsiklis, J. (1996). Neuro-Dynamic Programming. Athena Scientific, Belmont, MA.
7. Bertsimas, D., &: Tsitsiklis, J. (1997). Introduction to Linear Optimization. Athena Scientific, Belmont, MA.
8. Boutilier, C., Dearden, R., & Goldszmidt, M. (1995). Exploiting structure in policy construction. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pp. 1104-1111.
9. Bresina, J., Dearden, R., Meuleau, N., Ramakrishnan, S., Smith, D., &: Washington, R. (2002). Planning under continuous time and resource uncertainty: A challenge for AI. In Proceedings of the 18th Conference on Uncertainty in Artificial Intelligence, pp. 77-84.
10. Casella, G., & Robert, C. (1996). Rao-Blackwellisation of sampling schemes. Biometrika, 83(1), 81-94.
11. Chow, C.-S., &; Tsitsiklis, J. (1991). An optimal one-way multigrid algorithm for discrete-time stochastic control. IEEE Transactions on Automatic Control, 36(8), 898-914.
12. Cooper, G. (1988). A method for using belief networks as influence diagrams. In Proceedings of the Workshop on Uncertainty in Artificial Intelligence, pp. 55-63.
13. Crites, R., & Barto, A. (1996). Improving elevator performance using reinforcement learning. In Advances in Neural Information Processing Systems 8, pp. 1017-1023.
14. de Farias, D. P., & Van Roy, B. (2003). The linear programming approach to approximate dynamic programming. Operations Research, 51(6), 850-856.
15. de Farias, D. P., & Van Roy, B. (2004). On constraint sampling for the linear programming approach to approximate dynamic programming. Mathematics of Operations Research, 29(3), 462-478.
16. Dean, T., & Kanazawa, K. (1989). A model for reasoning about persistence and causation. Computational Intelligence, 5, 142-150.
17. Dechter, R. (1996). Bucket elimination: A unifying framework for probabilistic inference. In Proceedings of the 12th Conference, on Uncertainty in Artificial Intelligence, pp. 211-219.
18. Duane, S., Kennedy, A. D., Pendleton, B., & Roweth, D. (1987). Hybrid Monte Carlo. Physics Letters B, 195(2), 216-222.
19. Feng, Z., Dearden, R., Meuleau, N., & Washington, R. (2004). Dynamic programming for structured continuous Markov decision problems. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, pp. 154-161.
20. Ferns, N., Panangaden, P., & Precup, D. (2005). Metrics for Markov decision processes with infinite state spaces. In Proceedings of the 21st Conference on Uncertainty in Artificial Intelligence.
21. Geman, S., &; Geman, D. (1984). Stochastic relaxation, Gibbs distribution, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6(6), 721-741.
22. Gordon, G. (1999). Approximate Solutions to Markov Decision Processes. Ph.D. thesis, Carnegie Mellon University.
23. Guestrin, C. (2003). Planning Under Uncertainty in Complex Structured Environments. Ph.D. thesis, Stanford University.
24. Guestrin, C., Hauskrecht, M., &; Kveton, B. (2004). Solving factored MDPs with continuous and discrete variables. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, pp. 235-242.
25. Guestrin, C., Koller, D., Gearhart, C., &: Kanodia, N. (2003). Generalizing plans to new environments in relational MDPs. In Proceedings of the 18th International Joint Conference on Artificial Intelligence, pp. 1003-1010.
26. Guestrin, C., Koller, D., & Parr, R. (2001). Max-norm projections for factored MDPs. In Proceedings of the 17th International Joint Conference on Artificial Intelligence, pp. 673-682.
27. Guestrin, C., Koller, D., &: Parr, R. (2002). Multiagent planning with factored MDPs. In Advances in Neural Information Processing Systems 14, pp. 1523-1530.
28. Guestrin, C., Koller, D., Parr, R., &; Venkataraman, S. (2003). Efficient solution algorithms for factored MDPs. Journal of Artificial Intelligence Research, 19, 399-468.
29. Guestrin, C., Venkataraman, S., &; Koller, D. (2002). Context specific multiagent coordination and planning with factored MDPs. In Proceedings of the 18th National Conference on Artificial Intelligence, pp. 253-259.
30. Hastings, W. K. (1970). Monte Carlo sampling methods using Markov chains and their application. Biometrika, 57, 97-109.
31. Hauskrecht, M. (2000). Value-function approximations for partially observable Markov decision processes. Journal of Artificial Intelligence Research, 13, 33-94.
32. Hauskrecht, M., & Kveton, B. (2004). Linear program approximations for factored continuous-state Markov decision processes. In Advances in Neural Information Processing Systems 16, pp. 895-902.
33. Higdon, D. (1998). Auxiliary variable methods for Markov chain Monte Carlo with applications. Journal of the American Statistical Association, 55(442), 585-595.
34. Howard, R., & Matheson, J. (1984). Influence diagrams. In Readings on the Principles and Applications of Decision Analysis, Vol. 2, pp. 719-762. Strategic Decisions Group, Menlo Park, CA.
35. Jeffreys, H., &; Jeffreys, B. (1988). Methods of Mathematical Physics. Cambridge University Press, Cambridge, United Kingdom.
36. Jensen, F., Jensen, F., &; Dittmer, S. (1994). From influence diagrams to junction trees. In Proceedings of the 10th Conference on Uncertainty in Artificial Intelligence, pp. 367-373.
37. Khachiyan, L. (1979). A polynomial algorithm in linear programming. Doklady Akademii Nauk SSSR, 244, 1093-1096.
38. Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983). Optimization by simulated annealing. Science, 220(4598), 671-680.
39. Koller, D., &; Parr, R. (1999). Computing factored value functions for policies in structured MDPs. In Proceedings of the 16th International Joint Conference on Artificial Intelligence, pp. 1332-1339.
40. Kveton, B., & Hauskrecht, M. (2004). Heuristic refinements of approximate linear programming for factored continuous-state Markov decision processes. In Proceedings of the 14th International Conference on Automated Planning and Scheduling, pp. 306-314.
41. Kveton, B., & Hauskrecht, M. (2005). An MCMC approach to solving hybrid factored MDPs. In Proceedings of the 19th International Joint Conference on Artificial Intelligence, pp. 1346-1351.
42. Kveton, B., & Hauskrecht, M. (2006a). Learning basis functions in hybrid domains. In Proceedings of the 21st National Conference on Artificial Intelligence, pp. 1161-1166.
43. Kveton, B., & Hauskrecht, M. (2006b). Solving factored MDPs with exponential-family transition models. In Proceedings of the 16th International Conference on Automated Planning and Scheduling, pp. 114-120.
44. Mahadevan, S. (2005). Samuel meets Amarel: Automating value function approximation using global state space analysis. In Proceedings of the 20th National Conference on Artificial Intelligence, pp. 1000-1005.
45. Mahadevan, S., & Maggioni, M. (2006). Value function approximation with diffusion wavelets and Laplacian eigenfunctions. In Advances in Neural Information Processing Systems 18, pp. 843-850.
46. Mahadevan, S., Maggioni, M., Ferguson, K., &: Osentoski, S. (2006). Learning representation and control in continuous Markov decision processes. In Proceedings of the 21st National Conference on Artificial Intelligence.
47. Manne, A. (1960). Linear programming and sequential decisions. Management Science, 6(3), 259-267.
48. Metropolis, N., Rosenbluth, A., Rosenbluth, M., Teller, A., & Teller, E. (1953). Equation of state calculations by fast computing machines. Journal of Chemical Physics, 21, 1087-1092.
49. Munos, R., & Moore, A. (2002). Variable resolution discretization in optimal control. Machine Learning, 49, 291-323.
50. Ortiz, L. (2002). Selecting Approximately-Optimal Actions in Complex Structured Domains. Ph.D. thesis, Brown University.
51. Park, J., & Darwiche, A. (2001). Approximating MAP using local search. In Proceedings of the 17th Conference on Uncertainty in Artificial Intelligence, pp. 403-410.
52. Park, J., & Darwiche, A. (2003). Solving MAP exactly using systematic search. In Proceedings of the 19th Conference on Uncertainty in Artificial Intelligence, pp. 459-468.
53. Patrascu, R., Poupart, P., Schuurmans, D., Boutilier, C., & Guestrin, C. (2002). Greedy linear value-approximation for factored Markov decision processes. In Proceedings of the 18th National Conference on Artificial Intelligence, pp. 285-291.
54. Puterman, M. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, New York, NY.
55. Rust, J. (1997). Using randomization to break the curse of dimensionality. Econometrica, 65(3), 487-516.
56. Sanner, S., &; Boutilier, C. (2005). Approximate linear programming for first-order MDPs. In Proceedings of the 21st Conference on Uncertainty in Artificial Intelligence.
57. Schuurmans, D., & Patrascu, R. (2002). Direct value-approximation for factored MDPs. In Advances in Neural Information Processing Systems 14, pp. 1579-1586.
58. Schweitzer, P., &; Seidmann, A. (1985). Generalized polynomial approximations in Markovian decision processes. Journal of Mathematical Analysis and Applications, 110, 568-582.
59. Sondik, E. (1971). The Optimal Control of Partially Observable Markov Decision Processes. Ph.D. thesis, Stanford University.
60. Sutton, R., &: Barto, A. (1998). Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA.
61. Tesauro, G. (1992). Practical issues in temporal difference learning. Machine Learning, 8(3-4), 257-277.
62. Tesauro, G. (1994). TD-Gammon, a self-teaching backgammon program, achieves masterlevel play. Neural Computation, 6(2), 215-219.
63. Tesauro, G. (1995). Temporal difference learning and TD-Gammon. Communications of the ACM, 38(3), 58-68.
64. Trick, M., & Zin, S. (1993). A linear programming approach to solving stochastic dynamic programs. Tech. rep., Carnegie Mellon University.
65. Van Roy, B. (1998). Planning Under Uncertainty in Complex Structured Environments. Ph.D. thesis, Massachusetts Institute of Technology.
66. Yuan, C., Lu, T.-C., & Druzdzel, M. (2004). Annealed MAP. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, pp. 628-635.
67. Zhang, W., & Dietterich, T. (1995). A reinforcement learning approach to job-shop scheduling. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pp. 1114-1120.
68. Zhang, W., & Dietterich, T. (1996). High-performance job-shop scheduling with a time-delay TD(A) network. In Advances in Neural Information Processing Systems 8, pp. 1024-1030.