Policy iteration for decentralized control of markov decision processes

Journal of Artificial Intelligence Research

Volumn 34, Issue , 2009, Pages 89-132

  Bernstein, Daniel S ^a   Amato, Christopher ^a   Hansen, Eric A ^b   Zilberstein, Shlomo ^a  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

^b MISSISSIPPI STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BEHAVIORAL RESEARCH; CONTROLLERS; DYNAMIC PROGRAMMING; ITERATIVE METHODS; LEARNING ALGORITHMS; MARKOV PROCESSES; MULTIPURPOSE ROBOTS; PROBABILITY DISTRIBUTIONS; SOFTWARE AGENTS; STOCHASTIC SYSTEMS;

DISTRIBUTED AGENTS; FINITE-STATE CONTROLLERS; MARKOV DECISION PROCESSES; MULTI-ROBOT SYSTEMS; OPTIMAL ALGORITHM; PARTIALLY OBSERVABLE MARKOV DECISION PROCESS; POLICY ITERATION; POLICY ITERATION ALGORITHMS;

PROCESS CONTROL;

EID: 65349083220     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.2667     Document Type: Article

References (41)

1. Amato, C, Bernstein, D. S., & Zilberstein, S. (2007). Optimizing memory-bounded controllers for decentralized POMDPs. In Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence.
2. Astrom, K. J. (1965). Optimal control of Markov decision processes with incomplete state estimation. Journal of Mathematical Analysis and Applications, 10, 174-205.
3. Aumann, R. J. (1974). Subjectivity and correlation in randomized strategies. Journal of Mathematical Economics, 1, 67-96.
4. Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002). The complexity of decentralized control of Markov decision processes. Mathematics of Operations Research, 27(A), 819-840.
5. Bernstein, D. S., Hansen, E. A., & Zilberstein, S. (2005). Bounded policy iteration for decentralized POMDPs. In Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, pp. 1287-1292.
6. Bonet, B., & Geffner, H. (2000). Planning with incomplete information as heuristic search in belief space. In Proceedings of the Fifth International Conference on AI Planning and Scheduling, pp. 52-61.
7. Cassandra, A., Littman, M. L., & Zhang, N. L. (1997). Incremental pruning: A simple, fast, exact method for partially observable Markov decision processes. In Proceedings of the Thirteenth Annual Conference on Uncertainty in Artificial Intelligence, pp. 54-61.
8. Cheng, H.-T. (1988). Algorithms for Partially Observable Markov Decision Processes. Ph.D. thesis, University of British Columbia.
9. Emery-Montemerlo, R., Gordon, G., Schnieder, J., & Thrun, S. (2004). Approximate solutions for partially observable stochastic games with common payoffs. In Proceedings of the Third International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 136-143.
10. Feng, Z., & Zilberstein, S. (2004). Region-based incremental pruning for POMDPs. In Proceedings of the Twentieth Conference on Uncertainty in Artificial Intelligence, pp. 146-153.
11. Feng, Z., & Zilberstein, S. (2005). Efficient maximization in solving POMDPs. In Proceedings of the Twentieth National Conference on Artificial Intelligence, pp. 975-980.
12. Hansen, E. (1998a). Solving POMDPs by searching in policy space. In Proceedings of the Fourteenth Annual Conference on Uncertainty in Artificial Intelligence, pp. 211-219.
13. Hansen, E. A. (1998b). Finite-Memory Control of Partially Observable Systems. Ph.D. thesis, University of Massachusetts Amherst, Amherst, Massachusetts.
14. Hansen, E. A., Bernstein, D. S., & Zilberstein, S. (2004). Dynamic programming for partially observable stochastic games. In Proceedings of the Nineteenth National Conference on Artificial Intelligence, pp. 709-715.
15. Ji, S., Parr, R., Li, H., Liao, X., & Carin, L. (2007). Point-based policy iteration. In Proceedings of the Twenty-Second National Conference on Artificial Intelligence, pp. 1243-1249.
16. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2), 99-134.
17. Koller, D., & Milch, B. (2003). Multi-agent influence diagrams for representing and solving games. Games and Economic Behavior, 45(l), 181-221.
18. La Mura, P. (2000). Game networks. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence, pp. 335-342.
19. Lark III, J. W. (1990). Applications of Best-First Heuristic Search to Finite-Horizon Partially Observed Markov Decision Processes. Ph.D. thesis, University of Virginia.
20. Littman, M. L., Cassandra, A. R., & Kaelbling, L. P. (1995). Learning policies for partially observable environments: Scaling up. In Proceedings of the Twelfth International Conference on Machine Learning, pp. 362-370.
21. Nair, R., Pynadath, D., Yokoo, M., Tambe, M., & Marsella, S. (2003). Taming decentralized POMDPs: Towards efficient policy computation for multiagent settings. In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, pp. 705- 711.
22. Nair, R., Varakantham, P., Tambe, M., & Yokoo, M. (2005). Networked distributed POMDPs: A synthesis of distributed constraint optimization and POMDPs. In Proceedings of the Twentieth National Conference on Artificial Intelligence, pp. 133-139.
23. Parker, D. A. (2002). Implementation of Symbolic Model Checking for Probabilistic Systems. Ph.D. thesis, University of Birmingham, Birmingham, England.
24. Peshkin, L., Kim, K.-E., Meuleau, N., & Kaelbling, L. P. (2000). Learning to cooperate via policy search. In Proceedings of the Sixteenth International Conference on Uncertainty in Artificial Intelligence, pp. 489-496.
25. Pineau, J., Gordon, G., & Thrun, S. (2003). Point-based value iteration: An anytime algorithm for POMDPs. In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, pp. 1025-1031.
26. Platzman, L. K. (1980). A feasible computational approach to infinite-horizon partially- observed Markov decision processes. Tech. rep., Georgia Institute of Technology. Reprinted in Working Notes of the 1998 AAA I Fall Symposium on Planning Using Partially Observable Markov Decision Processes.
27. Poupart, P., & Boutilier, C. (2003). Bounded finite state controllers. In Proceedings of Advances in Neural Information Processing Systems 16.
28. Seuken, S., & Zilberstein, S. (2007). Improved memory-bounded dynamic programming for decentralized POMDPs. In Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence.
29. Simmons, R., & Koenig, S. (1995). Probabilistic navigation in partially observable environments. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1080-1087.
30. Singh, S. (1994). Learning to Solve Markovian Decision Processes. Ph.D. thesis, University of Massachusetts, Amherst, Massachusetts.
31. Singh, S. P., Jaakkola, T., & Jordan, M. I. (1994). Learning without state-estimation in partially observable markovian decision processes. In Proceedings of the Eleventh International Conference on Machine Learning, pp. 284-292.
32. Small wood, R. D., & Sondik, E. J. (1973). The optimal control of partially observable Markov processes over a finite horizon. Operations Research, 21(h), 1071-1088.
33. Smith, T., & Simmons, R. (2005). Point-based POMDP algorithms: Improved analysis and implementation. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence, pp. 542-547.
34. Smith, T., Thompson, D. R., & Wettergreen, D. S. (2007). Generating exponentially smaller POMDP models using conditionally irrelevant variable abstraction. In Proceedings of the Seventeenth International Conference on Applied Planning and Scheduling.
35. Sondik, E. J. (1978). The optimal control of partially observable Markov processes over the infinite horizon: Discounted costs. Operations Research, 26, 282-304.
36. Szer, D., & Charpillet, F. (2005). An optimal best-first search algorithm for solving infinite horizon DEC-POMDPs. In Proceedings of the Sixteenth European Conference on Machine Learning, pp. 389-399.
37. Szer, D., & Charpillet, F. (2006). Point-based dynamic programming for DEC-POMDPs. In Proceedings of the Twenty-First National Conference on Artificial Intelligence, pp. 1233-1238.
38. Szer, D., Charpillet, F., & Zilberstein, S. (2005). MAA*: A heuristic search algorithm for solving decentralized POMDPs. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence, pp. 576-590.
39. Witsenhausen, H. S. (1971). Separation of estimation and control for discrete time systems. Proceedings of the IEEE, 55(11), 1557-1566.
40. Zhang, N. L., & Lee, S. S. (1998). Planning with partially observable Markov decision processes: Advances in exact solution methods. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pp. 523-530.
41. Zhang, N. L., & Zhang, W. (2001). Speeding up the convergence of value iteration in partially observable Markov decision processes. Journal of Artificial Intelligence Research, 14, 29-51.