Partially observable Markov decision processes: A geometric technique and analysis

Operations Research

Volumn 58, Issue 1, 2010, Pages 214-228

  Zhang, Hao ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

Analysis of algorithms; Artificial intelligence; Combinatorics; Computational complexity; Computers computer science; Dynamic programming; Markov; Mathematics

Indexed keywords

ANALYSIS OF ALGORITHMS; COMBINATORICS; CONVEX HULL; CONVEX POLYTOPES; DEGREE OF OBSERVABILITY; DISCOUNTED REWARD; FINITE STATE; GEOMETRIC TECHNIQUES; INFINITE HORIZONS; MINKOWSKI SUM; PARTIALLY OBSERVABLE MARKOV DECISION PROCESS;

ARTIFICIAL INTELLIGENCE; COMPUTATIONAL EFFICIENCY; COMPUTATIONAL GEOMETRY; DYNAMIC PROGRAMMING; MARKOV PROCESSES; TOPOLOGY;

COMPUTATIONAL COMPLEXITY;

EID: 77249107864     PISSN: 0030364X     EISSN: 15265463     Source Type: Journal    
DOI: 10.1287/opre.1090.0697     Document Type: Article

References (67)

1. Anily, S., A. Grosfeld-Nir. 2006. An optimal lot-sizing and offline inspection policy in the case of nonrigid demand. Oper. Res. 54(2) 311-323.
2. Aoki, M. 1965. Optimal control of partially observable Markovian systems. J. Franklin Inst. 280 367-386.
3. Astrom, K. J. 1965. Optimal control of Markov decision processes with incomplete state estimation. J. Math. Anal. Appl. 10 174-205.
4. Berg, M. de, M. van Kreveld, M. Overmars, O. Schwarzkopf. 2000. Computational Geometry: Algorithms and Applications. Springer-Verlag, Berlin.
5. Bertsekas, D. 1976. Dynamic Programming and Stochastic Control. Academic Press, New York.
6. Blackwell, D. 1953. Equivalent comparisons of experiments. Ann. Math. Statist. 24 265-272.
7. Boissonnat, J-D., M. Yvinec. 1998. Algorithmic Geometry. Cambridge University Press, Cambridge, UK.
8. Boutilier, C. 2002. A POMDP formulation of preference elicitation problems. Proc. Eighteenth National Conf. Artificial Intelligence (AAAI-02). AAAI Press, Menlo Park, CA, 239-246.
9. Cassandra, A. R., L. P. Kaelbling, J. A. Kurien. 1996. Acting under uncertainty: Discrete Bayesian models for mobile robot navigation. Proc. IEEE/RSJ Internat. Conf. Intelligent Robots and Systems. IEEE, Piscataway, NJ, 963-972.
10. Cassandra, A. R., L. P. Kaelbling, M. L. Littman. 1994. Acting optimally in partially observable stochastic domains. Proc. Twelfth National Conf. Artificial Intelligence (AAAI-94). AAAI Press, Menlo Park, CA, 1023-1028.
11. Cassandra, A. R., M. Littman, N. L. Zhang. 1997. Incremental pruning: A simple, fast, exact method for partially observable Markov decision processes. Proc. Thirteenth Conf. Uncertainty in Artificial Intelligence (UAI-97). Morgan Kaufmann, San Francisco, 54-61.
12. Cheng, H.-T. 1988. Algorithms for partially observable Markov decision processes. Ph.D. dissertation, University of British Columbia, Vancouver, British Columbia, Canada.
13. Drake, A. 1962. Observation of a Markov process through a noisy channel. Ph.D. dissertation, Massachusetts Institute of Technology, Cambridge, MA.
14. Eckles, J. E. 1966. Optimum replacement of stochastically failing systems. Ph.D. dissertation, Stanford University, Stanford, CA.
15. Eckles, J. E. 1968. Optimum maintenance with incomplete information. Oper. Res. 16 1058-1067.
16. Ehrenfeld, S. 1976. On a sequential Markovian decision procedure with incomplete information. Comput. Oper. Res. 3 39-48.
17. Feng, Z., S. Zilberstein. 2004. Region-based incremental pruning for POMDPs. Proc. 20th Conf . Uncertainty in Artificial Intelligence (UAI-04). Morgan Kaufmann, San Francisco, 146-153.
18. Fernández-Gaucherand, E., A. Arapostathis, S. I. Marcus. 1991. On the average cost optimality equation and the structure of optimal policies for partially observable Markov decision processes. Ann. Oper. Res. 29 439-470.
19. Fukuda, K. 2004. From the zonetope construction to the Minkowski addition of convex polytopes. J. Symbolic Comput. 38 1261-1272.
20. Gritzmann, P., B. Sturmfels. 1993. Minkowski addition of polytopes: Computational complexity and applications to Grobner bases. SIAM J. Discrete Math. 6(2) 246-269.
21. Grosfeld-Nir, A. 1996. A two-state partially observable Markov decision process with uniformly distributed observations. Oper. Res. 44(3) 458-463.
22. Grosfeld-Nir, A. 2007. Control limits for two-state partially observable Markov decision processes. Eur. J. Oper. Res. 182 300-304.
23. Hansen, E. A. 1998a. An improved policy iteration algorithm for partially observable MDPs. Advances in Neural Inform. Processing Systems 10 (NIPS-97). MIT Press, Cambridge, MA, 1015-1021.
24. Hansen, E. A. 1998b. Solving POMDPs by searching in policy space. Proc. Fourteenth Conf . Uncertainty in Artificial Intelligence (UAI-98). Morgan Kaufmann, San Francisco, 211-219.
25. Howard, R. 1960. Dynamic Programming and Markov Processes. MIT Press, Cambridge, MA.
26. Hsu, S.-P., D.-M. Chuang, A. Arapostathis. 2006. On the existence of stationary optimal policies for partially observed MDPs under the long-run average cost criterion. Systems Control Lett. 55 165-173.
27. Kaelbling, L. P., M. Littman, A. R. Cassandra. 1998. Planning and acting in partially observable stochastic domains. Artificial Intelligence 101 99-134.
28. Kakalik, J. S. 1965. Optimum policies for partially observable Markov systems. Technical Report 18, Operations Research Center, Massachusetts Institute of Technology, Cambridge, MA.
29. Lane, D. 1989. A partially observable model of decision making by fishermen. Oper. Res. 37(2) 240-254.
30. Littman, M. L., A. R. Cassandra, L. P. Kaelbling. 1995. Learning policies for partially observable environments: Scaling up. Proc. Twelfth Internat. Conf. Machine Learning (ICML-95). Morgan Kaufmann, San Francisco, 362-370.
31. Lovejoy, W. S. 1987. Some monotonicity results for partially observed Markov decision processes. Oper. Res. 35 (5) 736-743.
32. Lovejoy, W. S. 1991a. Computationally feasible bounds for partially observed Markov decision processes. Oper. Res. 39 162-175.
33. Lovejoy, W. S. 1991b. A survey of algorithmic methods for partially observed Markov decision processes. Ann. Oper. Res. 28 47-66.
34. Lusena, C., J. Goldsmith, M. Mundhenk. 2001. Nonapproximability results for partially observable Markov decision processes. J. Artificial Intelligence Res. 14 83-103.
35. Marbach, P., O. Mihatsch, J. N. Tsitsiklis. 2000. Call admission control and routing in integrated service networks using neuro-dynamic programming. IEEE J. Selected Areas Commun. 18(2) 197-208.
36. Meuleau, N., K.-E. Kim, L. P. Kaelbling, A. R. Cassandra. 1999. Solving POMDPs by searching the space of finite policies. Proc. Fifteenth Conf. Uncertainty in Artificial Intelligence (UAI-99). Morgan Kaufmann, San Francisco, 417-426.
37. Meuleau, N., M. Hauskrecht, K.-E. Kim, L. Peshkin, L. P. Kaelbling, T. Dean, C. Boutilier. 1998. Solving very large weakly coupled Markov decision processes. Proc. Fifteenth National Conf. Artificial Intelligence (AAAI-98). AAAI Press, Menlo Park, CA, 165-172.
38. Monahan, G. E. 1982. A survey of partially observable Markov decision processes: Theory, models and algorithms. Management Sci. 28 1-16.
39. Montemerlo, M., J. Pineau, N. Roy, S. Thrun, V. Verma. 2002. Experiences with a mobile robotic guide for the elderly. Proc. Eighteenth National Conf. Artificial Intelligence (AAAI-02). AAAI Press, Menlo Park, CA, 587-592.
40. Paek, T., E. Horvitz. 2000. Conversation as action under uncertainty. Proc. Sixteenth Conf. Uncertainty in Artificial Intelligence (UAI-2000). Morgan Kaufmann, San Francisco, 455-464.
41. Papadimitriou, C. H., J. N. Tsitsiklis. 1987. The complexity of Markov decision processes. Math. Oper. Res. 12(3) 441-450.
42. Paz, A. 1971. Introduction to Probabilistic Automata. Academic Press, New York.
43. Pineau, J., G. Gordon, S. Thrun. 2003. Point-based value iteration: An anytime algorithm for POMDPs. Proc. Eighteenth Internat. Joint Conf. Artificial Intelligence (IJCAI-03). Morgan Kaufmann, San Francisco, 1025-1030.
44. Platzman, L. K. 1977. Finite memory estimation and control of finite probabilistic systems. Ph.D. dissertation, Massachusetts Institute of Technology, Cambridge, MA.
45. Platzman, L. K. 1980. Optimal infinite-horizon undiscounted control of finite probabilistic systems. SIAM J. Control Optim. 18(4) 362-380.
46. Poupart, P. 2005. Exploiting structure to efficiently solve large scale partially observable Markov decision processes. Ph.D. dissertation, University of Toronto, Toronto, Ontario, Canada.
47. Poupart, P., C. Boutilier. 2004. Bounded finite state controllers. Advances in Neural Inform. Processing Systems 16 (NIPS-03). MIT Press, Cambridge, MA, 823-830
48. Puterman, M. L. 1994. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley and Sons, New York.
49. Rockafellar, R. T. 1970. Convex Analysis. Princeton University Press, Princeton, NJ.
50. Ross, S. 1971. Quality control under Markovian deterioration. Management Sci. 17 587-596.
51. Satia, J. K., R. E. Lave. 1973. Markovian decision processes with probabilistic observation of states. Management Sci. 20 1-13.
52. Smallwood, R. D., E. J. Sondik. 1973. The optimal control of partially observable Markov processes over a finite horizon. Oper. Res. 21 1071-1088.
53. Sondik, E. J. 1971. The optimal control of partially observable Markov processes. Ph.D. dissertation, Stanford University, Stanford, CA.
54. Sondik, E. J. 1978. The optimal control of partially observable Markov processes over the infinite horizon: Discounted costs. Oper. Res. 26 282-304.
55. Sulganik, E. 1995. On the structure of Blackwell's equivalence classes of information systems. Math. Soc. Sci. 29 213-223.
56. Thrun, S. 2000. Monte Carlo POMDPs. Advances in Neural Inform. Processing Systems 12 (NIPS-99). MIT Press, Cambridge, MA, 1064-1070.
57. Treharne, J. T., C. R. Sox. 2002. Adaptive inventory control for nonstationary demand and partial information. Management Sci. 48(5) 607-624.
58. Wang, R. 1977. Optimal replacement policy under unobservable states. J. Appl. Probab. 14 340-348.
59. White, C. C. 1977. A Markov quality control process subject to partial observation. Management Sci. 23 843-852.
60. White, C. C. 1979. Optimal control-limit strategies for a partially observed replacement problem. Internat. J. Systems Sci. 10 321-331.
61. White, C. C. 1980. Monotone control laws for noisy, countable-state Markov chains. Eur. J. Oper. Res. 5 124-132.
62. White, C. C. 1991. A survey of solution techniques for the partially observed decision process. Ann. Oper. Res. 32 215-230.
63. White, C. C., W. T. Scherer. 1989. Solution procedures for partially observed Markov decision processes. Oper. Res. 37 791-797.
64. White, C. C., W. T. Scherer. 1994. Finite-memory suboptimal design for partially observed Markov decision processes. Oper. Res. 42 439-455.
65. Yu, H., D. Bertsekas. 2008. On near optimality of the set of finite-state controllers for average cost POMDP. Math. Oper. Res. 33(1) 1-11.
66. Zhang, N. L., W. Liu. 1996. Planning in stochastic domains: Problem characteristics and approximation. Technical Report HKUST-CS96-31, Hong Kong University of Science and Technology, Hong Kong.
67. Zhang, B., Q. Cai, J. Mao, B. Guo. 2001. Planning and acting under uncertainty: A new model for spoken dialogue systems. Proc. Seventeenth Conf. Uncertainty in Artificial Intelligence (UAI-01). Morgan Kaufmann, San Francisco, 572-579.