Complexity of finite-horizon Markov decision process problems

Journal of the ACM

Volumn 47, Issue 4, 2000, Pages 681-720

  Mundhenk, Martin ^a   Goldsmith, Judy ^b,d   Lusena, Christopher ^b,d   Allender, Eric ^c,e  

^a UNIVERSITY OF TRIER   (Germany)

^b UNIVERSITY OF KENTUCKY   (United States)

^c RUTGERS UNIVERSITY   (United States)

^d UNIVERSITY OF KENTUCKY   (United States)

^e RUTGERS UNIVERSITY   (United States)

Author keywords

Computational complexity; Markov decision processes; NP; NPPP; Partially observable Markov decision processes; PL; PSPACE; Succinct representations

Indexed keywords

CONTROLLED STOCHASTIC SYSTEMS; MARKOV DECISION PROCESSES; PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES (POMPD);

ALGORITHMS; COMPUTATIONAL COMPLEXITY; CONTROL EQUIPMENT; DECISION MAKING; MARKOV PROCESSES; MATHEMATICAL MODELS; POLYNOMIALS;

STOCHASTIC PROGRAMMING;

EID: 0001205548     PISSN: 00045411     EISSN: None     Source Type: Journal    
DOI: 10.1145/347476.347480     Document Type: Article

References (92)

1. ALLENDER, E., AND OGIHARA, M. 1996. Relationships among PL, #L, and the determinant. ^ KAIRO Theoret. Inf. Appl. 30, 1, 1-21.
2. ALVAREZ, C., AND JENNER, B. 1993. A very hard log-space counting class. Theoret. Compiit. Sei. 107, 3-30.
3. AOKI, M. 1965. Optimal control of partially observed Markovian systems. J. Franklin Inst. 280, 367-368.
4. ARROYO-FIGUEROA, G., AND SUGAR, L. E. 1999. A temporal Bayesian network for diagnosis and prediction. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. MorganKaufman, San Francisco, Calif.
5. ASTROM, K. 1965. Optimal control of Markov processes with incomplete state information. J. Math. Andy. Appl 10, 174-205.
6. BÄCKSTROM, C. 1995. Expressive equivalence of planning formalisms. Artif. Int. 76, 17-34.
7. BALCâZAR, J., LOZANO, A., AND TORâN, J. 1992. The complexity of algorithmic problems on succinct instances. In Computer Science, R. Baeza-Yates and U. Manber, Eds. Plenum Press, New York, pp. 351-377.
8. BARRY, P., AND LASKEY, K. B. 1999. An application of uncertain reasoning to requirements engineering. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. MorganKaufman, San Francisco, Calif.
9. BEAUQUIER, D., BURAGO, D., AND SLISSENKO, A. 1995. On the complexity of finite memory policies for Markov decision processes. In Mathematical Foundations of Computer Science. Lecture Notes in Computer Science, vol. 969. Springer-Verlag, New York, pp. 191-200.
10. BEIGEL, R., REINGOLD, N., AND SPIELMAN, D. 1995. PP is closed under intersection. J. Comput. Syst. Sei. SO, 191-202.
11. BELLMAN, R. 1957. Dynamic Programming. Princeton University Press, Princeton, N.J.
12. BLONDEL, V., AND TSITSIKLIS, J. 1998. A survey of computational complexity results in systems and control. Available from http://web.mit.edu/jnt/www/publ.html or http://web.mit.cdu/~jnt/survey.ps. (postscript, 785K), (to appear in Automatica).
13. BLYTHE, J. 1999. Decision-theoretic planning. AI Magazine 20, 2, 37-54.
14. BOMAN, DAVIDSSON, AND YOUNES. 1999. Artificial decision making in intelligent buildings. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. Morgan-Kaufman, San Francisco, Calif.
15. BOUTILIER, C., DEAN, T., AND HANKS, S. 1995a. Planning under uncertainty: Structural assumptions and computational leverage. In Proceedings of the 2nd European Workshop on Planning.
16. BOUTILIER, C., DEAN, T., AND HANKS, S. 1999a. Decision-theoretic planning: Structural assumptions and computational leverage. J. AI Res. 11, 1-94.
17. BOUTILIER, C., DEARDEN, R., AND GOLDSZMIDT, M. 1995b. Exploiting structure in policy construction. In Proceedings of the 14th International Conference on AI.
18. BOUTILIER, C., GOLDSZMIDT, M., AND SABATA, B. 1999b. Continuous value function approximation for sequential bidding policies. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. Morgan-Kaufman, San Francisco, Calif.
19. BRYANT, R. E. 1991. On the complexity of VLSI implementations and graph representations of boolean functions with application to integer multiplication. IEEE Trans. Comput. C-40, 2, 205-213.
20. BURAGO, D., DE ROUGEMONT, M., AND SLISSENKO, A. 1996. On the complexity of partially observed Markov decision processes. Theoret. Comput. Sei. 157, 2, 161-183.
21. BYLANDER, T. 1994. The computational complexity of propositional STRIPS planning. Artif. Int. 69, 165-204.
22. CASSANDRA, A. 1998. Exact and Approximate Algorithms for Partially Observable Markov Decision Processes. Ph.D. dissertation. Brown Univ., Providence, R.I.
23. CASSANDRA, A., KAELBLING, L., AND LITTMAN, M. 1994. Acting optimally in partially observable stochastic domains. In Proceedings ofAAAl-94.
24. CASSANDRA, A., KAELBLING, L., AND LITTMAN, M. 1995. Efficient dynamic-programming updates in partially observable Markov decision processes. Tech. Rep. CS-95-19. Brown Univ., Providence, R.I.
25. CASSANDRA, A., LITTMAN, M., AND ZHANG, N. 1997. Incremental pruning: A simple, fast, exact method for partially observable Markov decision processes. In Proceedings of the 13th Conference on Uncertainty in Artificial Intelligence (UAI-97). D. Geiger and P. P. Shenoy, Eds. MorganKaufmann, San Francisco, Calif., pp. 54-61.
26. CHAPMAN, D. 1987. Planning for conjunctive goals. Artif. Int. 32, 333-379.
27. DYNKIN, E. 1965. Controlled random sequences. Theory Prob. Appl. X, 10, 1-14.
28. EROL, K., HENDLER, J., AND NAU, D. 1996. Complexity results for hierarchical task-network planning. Ann. Math. Artif. Int. 18, 69-93.
29. EROL, K., NAU, D., AND SUBRAHMANIAN, V. 1995. Complexity, decidability and undecidability results for domain-independent planning. Artif. Int. 76, 75-88.
30. FENNER, S., FORTNOW, L., AND KURTZ, S. 1994. Gap-definable counting classes. J. Comput. Syst. Sei. 48, l, 116-148.
31. GALPERIN, H., AND WIGDERSON, A. 1983. Succinct representation of graphs. Inf. Cont. 56, 183-198.
32. GOLABI, K., KuLKARNi, R., AND WAY, G. 1982. A statewide pavement management system. Interfaces 12, 5-21.
33. GOLDSMITH, J., LITTMAN, M., AND MUNDHENK, M. 1997. The complexity of plan existence and evaluation in probabilistic domains. In Proceedings of the 13th Conference on Uncertainty in AI. Morgan-Kaufmann, San Francisco, Calif.
34. GOLDSMITH, J., AND MUNDHENK, M. 1998. Complexity issues in Markov decision processes. In Proceedings of IEEE Conference on Computational Complexity. IEEE Computer Society Press, Los Alamitos, Calif.
35. HANSEN, E. 1998a. Finite-Memory Control of Partially Observable Systems. Ph.D. dissertation, Dept. of Computer Science, University of Massachusetts at Amherst, Amherst, Massachusetts.
36. HANSEN, E. 1998b. Solving POMDPs by searching in policy space. In Proceedings of AAAI Conference Uncertainty in AI.
37. HAUSKRECHT, M. 1997. Planning and Control in Stochastic Domains with Imperfect Information. Ph.D. dissertation. Massachusetts Institute of Technology, Cambridge, Mass.
38. HOCHBAUM, D. 1997. Approximation Algorithms for NP-Hard Problems. PWS Publishing Company.
39. HOEY, J., ST.-AUBIN, R., Hu, A., AND BOUTILIER, C. 1999. SPUDD: Stochastic planning using decision diagrams. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. Morgan-Kaufman, San Francisco, Calif., pp. 279-288.
40. HORVITZ, E., JACOBS, A., AND HOVEL, D. 1999. Attention-sensitive alerting in computing systems. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. Morgan-Kaufman, San Francisco, Calif.
41. HOWARD, R. 1960. Dynamic Programming and Markov Processes. MIT Press, Cambridge, Mass.
42. JUNG, H. 1984. On probabilistic tape complexity and fast circuits for matrix inversion problems. In Proceedings of the llth ICALP. Lecture Notes in Computer Science, vol. 172. Springer-Verlag, New York, pp. 281-291.
43. JUNO, H. 1985. On probabilistic time and space. In Proceedings of the 12th ICALP. Lecture Notes in Computer Science, vol. 194. Springer-Verlag, New York, pp. 281-291.
44. KORB, K. B., NICHOLSON, A. E., AND JITNAH, N. 1999. Bayesian poker. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. Morgan-Kaufman, San Francisco, Calif.
45. KUSHMERICK, N., HANKS, S., AND WELD, D. 1995. An algorithm for probabilistic planning. Artif. Int. 76, 239-286.
46. LADNER, R. 1989. Polynomial space counting problems. SIAM J. Comput. 18, 1087-1097.
47. LITTMAN, M. 1996a. Algorithms for Sequential Decision Making. Ph.D. dissertation. Brown Univ., Providence, R.I.
48. LITTMAN, M. 1996b. Probabilistic STRIPS planning is EXPTIME-complete. Tech. Rep. CS1996-18 (November). Dept. Computer Science, Duke Univ.
49. LITTMAN, M. 1997a. Probabilistic prepositional planning: Representations and complexity. In Proceedings of the 14th National Conference on Artificial Intelligence. AAAI Press/The MIT Press, Cambridge, Mass.
50. LITTMAN, M. 1997b. Probabilistic propositional planning:. Representations and complexity. In Proceedings of the 14th National Conference on Artificial Intelligence. AAAI Press/MIT Press, Cambridge, Mass.
51. LITTMAN, M. L. 1999a. Initial experiments in probabilistic satisfiability. In Proceedings ofAAAI-99. In preparation for conference submission.
52. LITTMAN, M. L. 1999b. Initial experiments in stochastic satisfiability. In Proceedings of the 16th National Conference on Artificial Intelligence. AAAI Press/MIT Press, Cambridge, Mass.
53. LITTMAN, M., DEAN, T., AND KAELBLING, L. 1995. On the complexity of solving Markov decision problems. In Proceedings of the llth Annual Conference on Uncertainty in Artificial Intelligence, pp. 394-402.
54. LITTMAN, M., GOLDSMITH, J., AND MUNDHENK, M. 1998. The computational complexity of probabilistic plan existence and evaluation. J. AI Res. 9, 1-36.
55. LOVEJOY, W. 1991. A survey of algorithmic methods for partially observed Markov decision processes. Ann. Oper. Res. 28, 47-66.
56. LUSENA, C, Li, T., SITTINGER, S., WELLS, C., AND GOLDSMITH, J. 1999. My brain is full: When more memory helps. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. pp. 374-381.
57. MADANI, O., HANKS, S., AND CONDON, A. 1999. On the undecidability of probabilistic planning and infinite-horizon partially observable Markov decision problems. In Proceedings of the 16th National Conference on Artificial Intelligence.
58. MAJERCIK, M., AND LITTMAN, M. 1998a. Maxplan: A new approach to probabilistic planning. In Artificial Intelligence and Planning Systems, pp. 86-93.
59. MAJERCIK, S. M., AND LITTMAN, M. L. 1998b. Using caching to solve larger probabilistic planning problems. In Proceedings of 15th National Conference on Artificial Intelligence, pp. 954-959.
60. MEULEAU, N., KIM, K.-E., KAELBLING, L. P., AND CASSANDRA, A. R. 1999. Solving POMDPs by searching the space of finite policies. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence, pp. 417-426.
61. MEULEAU, N., PESHKIN, L., KIM, K.-E., AND KAELBLING, L. P. 1999. Learning finite-state controllers for partially observable environments. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence, pp. 427-436.
62. MISLEVY, R. J., ALMOND, R. G., YAN, D., AND STEINBERG, L. S. 1999. Bayes nets in educational assessment: Where the numbers come from. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. Morgan-Kaufman, San Francisco, Calif.
63. MONAIIAN, G. 1982. A survey of partially observable Markov decision processes: Theory, models, and algorithms. Manag. Sei. 28, 1-16.
64. MUNDHENK, M. 2000. The complexity of optimal small policies. Math. Oper. Res. 25, 1, 118-129.
65. MUNDHENK, M., GOLDSMITH, J., AND ALLENDER, E. 1997. The complexity of the policy existence problem for partially-observable finite-horizon Markov decision processes. In Proceedings of the 25th Mathematical Foundations of Computer Sciences. Lecture Notes in Computer Science, vol. 1295: Springer-Verlag, New York, pp. 129-138.
66. NGUYEN, H., AND HADDAWY, P. 1999. The decision-theoretic interactive video advisor. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. Morgan-Kaufman, San Francisco, Calif.
67. PAPADIMITRIOU, C. 1994. Computational Complexity. Addison-Wesley, Reading, Mass.
68. PAPADIMITRIOU, C., AND TSITSIKLIS, J. 1986. Intractable problems in control theory. SLAM J. Cont. Optim. 24, 4, 639-654.
69. PAPADIMITRIOU, C., AND TSITSIKLIS, J. 1987. The complexity of Markov decision processes. Math. Oper. Res. 12, 3, 441-450.
70. PAPADIMITRIOU, C., AND YANNAKAKIS, M. 1986. A note on succinct representations of graphs. Inf. Cont. 71, 181-185.
71. PARR, R., AND RUSSELL, S. 1995. Approximating optimal policies for partially observable stochastic domains. In Proceedings of IJCAI-95.
72. PESHKIN, L., MEULEAU, N., AND KAELBLING, L. P. 1999. Learning policies with external memory. In Proceedings of the 16th International Conference on Machine Learning.
73. PLATZMAN, L. 1977. Finite-memory estimation and control of finite probabilistic systems. Ph.D. dissertation. Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, Mass.
74. PORTINALE, L., AND BoBBio, A. 1999. Bayesian networks for dependability analysis: an application to digital control reliability. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. Morgan-Kaufman, San Francisco, Calif.
75. PUTERMAN, M. 1994. Markov Decision Processes. Wiley, New York.
76. PYEATT, L. 1999. Integration of Partially Observable Markov Decision Processes and Reinforcement Learning for Simulated Robot Navigation. Ph.D. dissertation. Colorado State Univ.
77. SIIATKAY, H. 1999. Learning hidden Markov models with geometrical constraints. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. Morgan-Kaufman, San Francisco, Calif.
78. SIMMONS, R., AND KOENIG, S. 1995. Probabilistic robot navigation in partially observable environments. In Proceedings of IJCAI-95.
79. SMALLWOOD, R., AND SONDIK, E. 1973. The optimal control of partially observed Markov processes over the finite horizon. Oper. Res. 21, 1071-1088.
80. SONDIK, E. 1971. The optimal control of partially observable Markov processes. Ph.D. dissertation. Stanford Univ., Stanford, Calif.
81. STREIBEL, C. 1965. Sufficient statistics in the optimal control of stochastic systems. J. Math. Anal. Appl. 12, 576-592.
82. SUTTON, R. S., AND BARTO, A. G. 1998. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, Mass.
83. TODA, S. 1991. PP is as hard as the polynomial-time hierarchy. SIAM J. Comput. 20, 865-877.
84. TORâN, J. 1991. Complexity classes defined by counting quantifiers. J. ACM 38, 3, 753-774.
85. TSENG, P. 1990. Solving /j-horizon, stationary Markov decision problems in time proportional to log h. Oper. Res. Lett. 9, 5 (Sept.), 287-297.
86. VAN DER GAAG, L., RENOOIJ, S., WITTEMAN, C, ALEMAN, B., AND TALL, B. 1999. How to elicit many probabilities. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. Morgan-Kaufman, San Francisco, Calif., pp. 647-654.
87. VINAY, V. 1991. Counting auxiliary pushdown automata and semi-unbounded arithmetic circuits. In Proceedings of the 6th Structure in Complexity Theory Conference. IEEE Computer Society Press, Los Alamitos, Calif., pp. 270-284.
88. WAGNER, K. 1986. The complexity of combinatorial problems with succinct input representation. Acta. Inf. 23, 325-356.
89. WELCH, R. L., AND SMOTH, C. 1999. A process control algorithm for concentrating mixed-waste based on Bayesian cg-networks. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. Morgan-Kaufman, San Francisco, Calif.
90. WHITE, C., III. 1991. Partially observed Markov decision processes: A survey. Ann. Oper. Res. 32, 215-230.
91. ZHANG, N., AND Liu, W. 1997. A model approximation scheme for planning in partially observable stochastic domains. J. Artif. Int. Res. 7, 199-230.
92. ZHANG, N. L., LEE, S. S., AND ZHANG, W. 1999. A method for speeding up value iteration in partially observable Markov decision processes. In Proceedings of the 15th Conference on Uncertainty in Artificial Intelligence. Morgan-Kaufman, San Francisco, Calif., pp. 696-703.