An introduction to fully and partially observable Markov decision processes

Decision Theory Models for Applications in Artificial Intelligence: Concepts and Solutions

Volumn , Issue , 2011, Pages 33-62

  Poupart, Pascal ^a  

^a UNIVERSITY OF WATERLOO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84889585200     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4018/978-1-60960-165-2.ch003     Document Type: Chapter

References (66)

1. Abbeel, P., & Ng, A. (2004). Apprenticeship learning via inverse reinforcement learning, International Conference on Machine Learning (ICML), Banff, Alberta, (pp. 1-8).
2. Altman, E. (1999). Constrained Markov Decision Processes. Boca Raton, FL: Chapman & Hall/CRC.
3. Amato, C., Bernstein, D. S., & Zilberstein, S. (2007). Solving POMDPs using quadratically constrained linear programs. International Joint Conferences on Artificial Intelligence (pp. 2418-2424). IJCAI.
4. Amato, C., Bernstein, D. S., & Zilberstein, S. (2010). Optimizing fixed-size stochastic controllers for POMDPs and decentralized POMDPs. Autonomous Agents and Multi-Agent Systems, 21(3), 293-320. doi:10.1007/s10458-009-9103-z
5. Bellman, R. (1957). A Markov decision process. Indiana University Mathematics Journal, 6(4), 679-684. doi:10.1512/iumj.1957.6.06038
6. Bengio, Y., & Frasconi, P. (1996). Input/output HMMs for sequence processing. IEEE Transactions on Neural Networks, 7(5), 1231-1249. doi:10.1109/72.536317
7. Bernstein, D. S., Amato, C., Hanse, E. A., & Zilberstein, S. (2009). Policy iteration for decentralized control of Markov decision processes. [JAIR]. Journal of Artificial Intelligence Research, 34, 89-132.
8. Braziunas, D., & Boutilier, C. (2004). Stochastic local search for POMDP controllers. National Conference on Artificial Intelligence (AAAI), San Jose, California, (pp. 690-696).
9. Choi, J., & Kim, K.-E. (2011). Inverse reinforcement learning in partially observable domains. [JMLR]. Journal of Machine Learning Research, 12, 691-730.
10. Delgado, K. V., Sanner, S., & Nunes de Barros, L. (2011). Efficient solutions to factored MDPs with imprecise transition probabilities. [AIJ]. Artificial Intelligence, 175(9-10), 1498-1527. doi:10.1016/j.artint.2011.01.001
11. Dempster, A., Laird, L., & Rubin, D. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B. Methodological, 39(1), 1-38.
12. Dietterich, T. (1998). The MAXQ method for hierarchical reinforcement learning. International Conference on Machine Learning, (pp. 118-126).
13. Elizalde, F., Sucar, L. E., Luque, M., Diez, F. J., & Reyes, A. (2008). Policy explanation in factored Markov decision processes. European Workshop on Probabilistic Graphical Models, (pp. 97-104).
14. Fard, M. M., & Pineau, J. (2011). Non-deterministic policies in Markovian decision processes. [JAIR]. Journal of Artificial Intelligence Research, 40, 1-24.
15. Feng, Z., Dearden, R., Meuleau, N., & Washington, R. (2004). Dynamic programming for structured continuous Markov decision problems. International Conference on Uncertainty in Artificial Intelligence (UAI), (pp. 154-161).
16. Gmytrasiewicz, P., & Doshi, P. (2005). A framework for sequential planning in multiagent settings [JAIR]. Journal of Artificial Intelligence Research, 24, 49-79.
17. Guestrin, C., Koller, D., Parr, R., & Venkataraman, S. (2003). Efficient solution algorithms for factored MDPs. [JAIR]. Journal of Artificial Intelligence Research, 19, 399-468.
18. Guo, X., & Hernandez-Lerma, O. (2009). Continuous time Markov decision processes. Applications of Mathematics, 62, 9-18.
19. Hansen, E. A. (1997). An improved policy iteration algorithm for partially observable MDPs. Advances in Neural Information Processing systems (pp. 1015-1021). Denver, Colorado: NIPS.
20. Hansen, E. A. (1998). Solving POMDPs by searching in the policy space. International Conference on Uncertainty in Artificial Intelligence (UAI), Madison, Wisconsin, (pp. 211-219).
21. Hansen, E. A., & Zhou, R. (2003). Synthesis of hierarchical finite-state controllers for POMDPs. International Conference on Automated Planning and Scheduling (ICAPS), (pp. 113-122).
22. Hoey, J., & Poupart, P. (2005). Solving POMDPs with continuous or large discrete observation spaces. International Joint Conferences on Artificial Intelligence (pp. 1332-1338). IJCAI.
23. Hoey, J., St-Aubin, R., Hu, J. A., & Boutilier, C. (1999). SPUDD: stochastic planning using decision diagrams. International Conference on Uncertainty in Artificial Intelligence (UAI), (pp. 279-288).
24. Howard, R. A. (1971). Dynamic Probabilistic Systems (Vol. II). New York, NY: John Wiley & Sons.
25. Kaelbling, L. P., Littman, M., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101, 99-134. doi:10.1016/S0004-3702(98)00023-X
26. Kearns, M., Mansour, Y., & Ng, A. Y. (1999). A sparse sampling algorithm for near-optimal planning in large Markov decision processes. International Joint Conferences on Artificial Intelligence (pp. 1324-1331). Stockholm, Sweden: IJCAI.
27. Khan, O. Z., Poupart, P., & Black, J. P. (2009). Minimal sufficient explanations for factored Markov decision processes. International Conference on Automated Plannning and Scheduling (ICAPS).
28. Kim, D., Lee, J., Kim, K.-E., & Poupart, P. (2011). Point-Based Value Iteration for Constrained POMDPs. International Joint conferences on Artificial Intelligence (pp. 1968-1974). IJCAI.
29. Kurniawati, H., Hsu, D., & Lee, W. (2008). SAR-SOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. International Conference on Robotics: Science and Systems (RSS).
30. Kveton, B., Hauskrecht, M., & Guestrin, C. (2006). Solving factored MDPs with hybrid state and action variables. [JAIR]. Journal of Artificial Intelligence Research, 27, 153-201.
31. Lusena, C., Goldsmith, J., & Mundhenk, M. (2001). Nonapproximability results for partially observable Markov decision processes. [JAIR]. Journal of Artificial Intelligence Research, 14, 83-103.
32. Meuleau, N., Benazera, E., Brafman, R.I., & Hansen, E.A. & Mausam. (2009). A Heuristic Search Approach to Planning with Continuous Resources in Stochastic Domains. [JAIR]. Journal of Artificial Intelligence Research, 34, 27-59.
33. Meuleau, N., Kim, K.-E., Kaelbling, L. P., & Cassandra, A. R. (1999), Solving POMDPS by searching the space of finite policies, International Conference on Uncertainty in Artificial Intelligence (UAI), Stockholm, Sweden, (pp. 417-426).
34. Neu, G., & Szepesvari, C. (2007). Apprenticeship learning using inverse reinforcement learning and gradient methods. International Conference on Uncertainty in Artificial Intelligence (UAI), Vancouver, Canada, (pp. 295-302).
35. Ng, A., & Russell, S. (2000). Algorithms for inverse reinforcement learning. International Conference on Machine Learning (ICML), Stanford, California, (pp. 663-670).
36. Oh, E., & Kim, K.-E. (2011). A geometric traversal algorithm for reward-uncertain MDPs. International Conference on Uncertainty in Artificial Intelligence (UAI), Barcelona, Spain.
37. Papadimitriou, C. H., & Tsitsilis, J. N. (1987). The complexity of Markov decision processes. Mathematics of Operations Research, 12(3), 441-450. doi:10.1287/moor.12.3.441
38. Parr, R., & Russell, S. J. (1997). Reinforcement learning with hierarchies of machines. Advances in Neural Information Processing Systems. NIPS.
39. Pineau, J., Gordon, G., & Thrun, S. (2006). Anytime point-based approximations for large POMDPs. [JAIR]. Journal of Artificial Intelligence Research, 27, 335-380.
40. Pineau, J., Gordon, G. J., & Thrun, S. (2003). Policy-contingent abstraction for robust robot control. International Conference on Uncertainty in Artificial Intelligence (UAI), (pp. 477-484).
41. Piunovskiy, A. B., & Mao, X. (2000). Constrained Markovian decision processes: the dynamic programming approach. Operations Research Letters, 27(3), 119-126. doi:10.1016/S0167-6377(00)00039-0
42. Porta, J. M., Vlassis, N. A., Spaan, M. T. J., & Poupart, P. (2006). Point-Based Value Iteration for Continuous POMDPs. [JMLR]. Journal of Machine Learning Research, 7, 2329-2367.
43. Poupart, P., & Boutilier, C. (2003). Bounded finite state controllers, Advances in Neural Information Processing Systems. Vancouver, Canada: NIPS.
44. Poupart, P., Kim, K. E., & Kim, D. (2011). Closing the Gap: Improved Bounds on Optimal POMDP Solutions, International Conference on Automated Planning and Scheduling (ICAPS), Freiburg, Germany.
45. Poupart, P., Lang, T., & Toussaint, M. (2011). Analyzing and Escaping Local Optima in Planning as Inference for Partially Observable Domains. European Conference on Machine Learning (ECML), Athens, Greece.
46. Puterman, M. (2005). Markov Decision Processes: Discrete Stochastic Dynamic Programming (2nd ed.). New York, NY: John Wiley & Sons.
47. Regan, K., & Boutilier, C. (2010). Robust policy computation in reward-uncertain MDPs using nondominated policies. National Conference on Artificial Intelligence (AAAI).
48. Regan, K., & Boutilier, C. (2011a). Eliciting additive reward functions for Markov decision processes. International Joint Conferences on Artificial Intelligence (pp. 2159-2164). Barcelona, Spain: IJCAI.
49. Regan, K., & Boutilier, C. (2011b). Robust Online Optimization of Reward-Uncertain MDPs. International Joint Conferences on Artificial Intelligence (pp. 2165-2171). Barcelona, Spain: IJCAI.
50. Ross, S., Pineau, J., Paquet, S., & Chaib-Draa, B. (2008). Online planning algorithms for POMDPs. [JAIR]. Journal of Artificial Intelligence Research, 32, 663-704.
51. Sanner, S., Delgado, K. V., & de Barros, L. N. (2011). Symbolic dynamic programming for discrete and continuous state MDPs. International Conference on Uncertainty in Artificial Intelligence (UAI), Barcelona, Spain.
52. Sanner, S., & McAllester, D. (2005). Affine algebraic decision diagrams (AADDs) and their application to structured probabilistic inference. International Joint Conference on Artificial Intelligence (IJCAI).
53. Satia, J. K., & Lave, R. E. Jr. (1970). Markovian decision processes with uncertain transition probabilities. Operations Research, 21, 728-740. doi:10.1287/opre.21.3.728
54. Schuurmans, D., & Patrascu, R. (2001). Direct value-approximation for factored MDPs. Advances in Neural Information Processing Systems (pp. 1579-1586). Vancouver, BC: NIPS.
55. Shani, G., Brafman, R. I., & Shimony, S. E. (2007). Forward search value iteration for POMDPs. International Joint Conferences on Artificial Intelligence (pp. 2619-2624). IJCAI.
56. Shani, G., Poupart, P., Brafman, R. I., & Shimony, S. E. (2008). Efficient ADD Operations for Point-Based Algorithms. International Conference on Automated Planning and Scheduling (ICAPS), pp. 330-337.
57. Silver, D., & Veness, J. (2010). Monte Carlo planning in large POMDPs, Advances in Neural Information Processing Systems. Vancouver, BC: NIPS.
58. Sim, H. S., Kim, K.-E., Kim, J. H., Chang, D.-S., & Koo, M.-Y. (2008). Symbolic heuristic search value iteration for factored POMDPs. National Conference on Artificial Intelligence (AAAI), pp. 1088-1093.
59. Smallwood, R. D., & Sondik, E. J. (1973). The optimal control of partially observable Markov processes over a finite horizon. Operations Research, 21, 1071-1088. doi:10.1287/opre.21.5.1071
60. Smith, T., & Simmons, R. G. (2005). Point-based POMDP algorithms: improved analysis and implementation, International Conference on Uncertainty in Artificial Intelligence (UAI), pp. 542-547.
61. Spaan, M. T. J., & Vlassis, N. A. (2005). Perseus: randomized point-based value iteration for POM-DPs. [JAIR]. Journal of Artificial Intelligence Research, 24, 195-220.
62. Sutton, R., Precup, D., & Singh, S. P. (1999). Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. [AIJ]. Artificial Intelligence, 112(1-2), 181-211. doi:10.1016/S0004-3702(99)00052-1
63. Thrun, S. (1999). Monte Carlo POMDPs. [NIPS]. Advances in Neural Information Processing Systems, 1064-1070.
64. Toussaint, M., Harmeling, S., & Storkey, A. (2006). Probabilistic inference for solving (PO) MDPs, Technical Report EDI-INF-RR-0934, School of Informatics, University of Edinburgh.
65. White, C. C. III, & El-Deib, H. K. (1994). Markov decision processes with imprecise transition probabilities. Operations Research, 42(4), 739-749. doi:10.1287/opre.42.4.739
66. Zhang, N. L., & Liu, W. (1996). Planning in stochastic domains: problem characteristics and approximation. Technical Report HKUST-CS96-31, Hong Kong University of Science and Technology.