Incremental clustering and expansion for faster optimal planning in decentralized POMDPs

Journal of Artificial Intelligence Research

Volumn 46, Issue , 2013, Pages 449-509

  Oliehoek, Frans A ^a   Spaan, Matthijs T J ^b   Amato, Christopher ^c   Whiteson, Shimon ^d  

^a MAASTRICHT UNIVERSITY   (Netherlands)

^b DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

^c MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^d UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

GENERAL MODEL; HYBRID HEURISTICS; INCREMENTAL CLUSTERING; MULTI-AGENT PLANNING; OPTIMAL PLANNING; OPTIMAL SOLUTIONS; PARTIALLY OBSERVABLE MARKOV DECISION PROCESS; THEORETICAL GUARANTEES;

GAME THEORY; HEURISTIC ALGORITHMS; MULTI AGENT SYSTEMS; TREES (MATHEMATICS);

CLUSTERING ALGORITHMS;

EID: 84878301624     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.3804     Document Type: Review

References (107)

1. Allen, M., & Zilberstein, S. (2007). Agent influence as a predictor of difficulty for decentralized problem-solving. In Proceedings of the Twenty-Second AAAI Conference on Artificial Intelligence.
2. Amato, C., Bernstein, D. S., & Zilberstein, S. (2006). Optimal fixed-size controllers for decentralized POMDPs. In Proc. of the AAMAS Workshop on Multi-Agent Sequential Decision Making in Uncertain Domains.
3. Amato, C., Bernstein, D. S., & Zilberstein, S. (2007). Optimizing memory-bounded controllers for decentralized POMDPs. In Proc. of Uncertainty in Artificial Intelligence.
4. Amato, C., Bernstein, D. S., & Zilberstein, S. (2010). Optimizing fixed-size stochastic controllers for POMDPs and decentralized POMDPs. Autonomous Agents and Multi-Agent Systems, 21 (3), 293-320.
5. Amato, C., Bonet, B., & Zilberstein, S. (2010). Finite-state controllers based on Mealy machines for centralized and decentralized POMDPs. In Proceedings of the Twenty- Fourth AAAI Conference on Artificial Intelligence.
6. Amato, C., Carlin, A., & Zilberstein, S. (2007). Bounded dynamic programming for decentralized POMDPs. In Proc. of the AAMAS Workshop on Multi-Agent Sequential Decision Making in Uncertain Domains.
7. Amato, C., Dibangoye, J. S., & Zilberstein, S. (2009). Incremental policy generation for finite-horizon DEC-POMDPs. In Proc. of the International Conference on Automated Planning and Scheduling.
8. Aras, R., & Dutech, A. (2010). An investigation into mathematical programming for finite horizon decentralized POMDPs. Journal of Artificial Intelligence Research, 37, 329- 396.
9. Becker, R., Carlin, A., Lesser, V., & Zilberstein, S. (2009). Analyzing myopic approaches for multi-agent communication. Computational Intelligence, 25 (1), 31-50.
10. Becker, R., Zilberstein, S., & Lesser, V. (2004). Decentralized Markov decision processes with event-driven interactions. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
11. Becker, R., Zilberstein, S., Lesser, V., & Goldman, C. V. (2003). Transition-independent decentralized Markov decision processes. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
12. Bernstein, D. S., Amato, C., Hansen, E. A., & Zilberstein, S. (2009). Policy iteration for decentralized control of Markov decision processes. Journal of Artificial Intelligence Research, 34, 89-132.
13. Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002). The complexity of decentralized control of Markov decision processes. Mathematics of Operations Research, 27 (4), 819-840.
14. Bertsekas, D. P. (2005). Dynamic Programming and Optimal Control (3rd ed., Vol. I). Athena Scientific.
15. Beynier, A., & Mouaddib, A.-I. (2011). Solving efficiently decentralized MDPs with temporal and resource constraints. Autonomous Agents and Multi-Agent Systems, 23 (3), 486- 539.
16. Boularias, A., & Chaib-draa, B. (2008). Exact dynamic programming for decentralized POMDPs with lossless policy compression. In Proc. of the International Conference on Automated Planning and Scheduling.
17. Bu̧soniu, L., Babuška, R., & De Schutter, B. (2008). A comprehensive survey of multi-agent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 38 (2), 156-172.
18. Carlin, A., & Zilberstein, S. (2008). Value-based observation compression for DEC-POMDPs. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
19. Carrillo, H., & Lipman, D. (1988). The multiple sequence alignment problem in biology. SIAM Journal on Applied Mathematics, 48 (5), 1073-1082.
20. Cassandra, A., Littman, M. L., & Zhang, N. L. (1997). Incremental pruning: A simple, fast, exact method for partially observable Markov decision processes. In Proc. of Uncertainty in Artificial Intelligence.
21. Cassandra, A. R. (1998). Exact and Approximate Algorithms for Partially Observable Markov Decision Processes. Unpublished doctoral dissertation, Brown University.
22. Dechter, R., Flerova, N., & Marinescu, R. (2012). Search algorithms for m best solutions for graphical models. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence.
23. Dekel, E., Fudenberg, D., & Morris, S. (2006). Topologies on types. Theoretical Economics, 1 (3), 275-309.
24. Dibangoye, J. S., Amato, C., Doniec, A., & Charpillet, F. (2013). Producing efficient errorbounded solutions for transition independent decentralized MDPs. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems. (Submitted for publication)
25. Dibangoye, J. S., Mouaddib, A.-I., & Chai-draa, B. (2009). Point-based incremental pruning heuristic for solving finite-horizon DEC-POMDPs. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
26. Doshi, P., & Gmytrasiewicz, P. (2009). Monte Carlo sampling methods for approximating interactive POMDPs. Journal of Artificial Intelligence Research, 34, 297-337.
27. Doshi, P., Zeng, Y., & Chen, Q. (2008). Graphical models for interactive POMDPs: representations and solutions. Autonomous Agents and Multi-Agent Systems, 18 (3), 376-416.
28. Edelkamp, S., & Schr?odl, S. (2012). Heuristic search: theory and applications. Morgan Kaufmann.
29. Eker, B., & Akin, H. L. (2010). Using evolution strategies to solve DEC-POMDP problems. Soft Computing - A Fusion of Foundations, Methodologies and Applications, 14 (1), 35-47.
30. Eker, B., & Akin, H. L. (2013). Solving decentralized POMDP problems using genetic algorithms. Autonomous Agents and Multi-Agent Systems, 27 (1), 161-196.
31. Emery-Montemerlo, R. (2005). Game-Theoretic Control for Robot Teams. Unpublished doctoral dissertation, Carnegie Mellon University.
32. Emery-Montemerlo, R., Gordon, G., Schneider, J., & Thrun, S. (2004). Approximate solutions for partially observable stochastic games with common payoffs. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
33. Emery-Montemerlo, R., Gordon, G., Schneider, J., & Thrun, S. (2005). Game theoretic control for robot teams. In Proc. of the IEEE International Conference on Robotics and Automation.
34. Givan, R., Dean, T., & Greig, M. (2003). Equivalence notions and model minimization in Markov decision processes. Artificial Intelligence, 14 (1-2), 163-223.
35. Gmytrasiewicz, P. J., & Doshi, P. (2005). A framework for sequential planning in multi-agent settings. Journal of Artificial Intelligence Research, 24, 49-79.
36. Goldman, C. V., Allen, M., & Zilberstein, S. (2007). Learning to communicate in a decentralized environment. Autonomous Agents and Multi-Agent Systems, 15 (1), 47-90.
37. Goldman, C. V., & Zilberstein, S. (2003). Optimizing information exchange in cooperative multi-agent systems. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
38. Goldman, C. V., & Zilberstein, S. (2004). Decentralized control of cooperative systems: Categorization and complexity analysis. Journal of Artificial Intelligence Research, 22, 143-174.
39. Goldman, C. V., & Zilberstein, S. (2008). Communication-based decomposition mechanisms for decentralized MDPs. Journal of Artificial Intelligence Research, 32, 169-202.
40. Hansen, E. A., Bernstein, D. S., & Zilberstein, S. (2004). Dynamic programming for partially observable stochastic games. In Proc. of the National Conference on Artificial Intelligence.
41. Hauskrecht, M. (2000). Value-function approximations for partially observable Markov decision processes. Journal of Artificial Intelligence Research, 13, 33-94.
42. Hsu, K., & Marcus, S. (1982). Decentralized control of finite state Markov processes. IEEE Transactions on Automatic Control, 27 (2), 426-431.
43. Huhns, M. N. (Ed.). (1987). Distributed Artificial Intelligence. Pitman Publishing Ltd.
44. Ikeda, T., & Imai, H. (1999). Enhanced A&z.ast; algorithms for multiple alignments: optimal alignments for several sequences and k-opt approximate alignments for large cases. Theoretical Computer Science, 210 (2), 341-374.
45. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101 (1-2), 99-134.
46. Kumar, A., & Zilberstein, S. (2009). Constraint-based dynamic programming for decentralized POMDPs with structured interactions. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
47. Kumar, A., & Zilberstein, S. (2010a). Anytime planning for decentralized POMDPs using expectation maximization. In Proc. of Uncertainty in Artificial Intelligence.
48. Kumar, A., & Zilberstein, S. (2010b). Point-based backup for decentralized POMDPs: Complexity and new algorithms. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
49. Littman, M., Cassandra, A., & Kaelbling, L. (1995). Learning policies for partially observable environments: Scaling up. In Proc. of the International Conference on Machine Learning.
50. Marecki, J., Gupta, T., Varakantham, P., Tambe, M., & Yokoo, M. (2008). Not all agents are equal: scaling up distributed POMDPs for agent networks. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
51. Marecki, J., & Tambe, M. (2007). On opportunistic techniques for solving decentralized Markov decision processes with temporal constraints. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
52. Melo, F. S., & Veloso, M. (2011). Decentralized MDPs with sparse interactions. Artificial Intelligence, 175 (11), 1757-1789.
53. Mostafa, H., & Lesser, V. (2011). A compact mathematical formulation for problems with structured agent interactions. In Proc. of the AAMAS Workshop on Multi-Agent Sequential Decision Making in Uncertain Domains.
54. Nair, R., Roth, M., & Yohoo, M. (2004). Communication for improving policy computation in distributed POMDPs. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
55. Nair, R., Tambe, M., Yokoo, M., Pynadath, D. V., & Marsella, S. (2003). Taming decentralized POMDPs: Towards efficient policy computation for multiagent settings. In Proc. of the International Joint Conference on Artificial Intelligence.
56. Nair, R., Varakantham, P., Tambe, M., & Yokoo, M. (2005). Networked distributed POMDPs: A synthesis of distributed constraint optimization and POMDPs. In Proc. of the National Conference on Artificial Intelligence.
57. Oliehoek, F. A. (2010). Value-Based Planning for Teams of Agents in Stochastic Partially Observable Environments. Amsterdam University Press. (Doctoral dissertation, University of Amsterdam)
58. Oliehoek, F. A. (2012). Decentralized POMDPs. In M. Wiering & M. van Otterlo (Eds.), Reinforcement learning: State of the art (Vol. 12). Springer Berlin Heidelberg.
59. Oliehoek, F. A., Kooi, J. F., & Vlassis, N. (2008). The cross-entropy method for policy search in decentralized POMDPs. Informatica, 32, 341-357.
60. Oliehoek, F. A., & Spaan, M. T. J. (2012). Tree-based solution methods for multiagent POMDPs with delayed communication. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence.
61. Oliehoek, F. A., Spaan, M. T. J., Dibangoye, J., & Amato, C. (2010). Heuristic search for identical payoff Bayesian games. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
62. Oliehoek, F. A., Spaan, M. T. J., & Vlassis, N. (2007). Dec-POMDPs with delayed communication. In Proc. of the AAMAS Workshop on Multi-Agent Sequential Decision Making in Uncertain Domains.
63. Oliehoek, F. A., Spaan, M. T. J., & Vlassis, N. (2008). Optimal and approximate Q-value functions for decentralized POMDPs. Journal of Artificial Intelligence Research, 32, 289-353.
64. Oliehoek, F. A., Spaan, M. T. J., Whiteson, S., & Vlassis, N. (2008). Exploiting locality of interaction in factored Dec-POMDPs. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
65. Oliehoek, F. A., & Vlassis, N. (2007). Q-value functions for decentralized POMDPs. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
66. Oliehoek, F. A., Whiteson, S., & Spaan, M. T. J. (2009). Lossless clustering of histories in decentralized POMDPs. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
67. Oliehoek, F. A., Whiteson, S., & Spaan, M. T. J. (2013). Approximate solutions for factored Dec-POMDPs with many agents. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems. (Submitted for publication)
68. Oliehoek, F. A., Witwicki, S., & Kaelbling, L. P. (2012). Influence-based abstraction for multiagent systems. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence.
69. Ooi, J. M., & Wornell, G. W. (1996). Decentralized control of a multiple access broadcast channel: Performance bounds. In Proc. of the 35th conference on decision and control.
70. Osborne, M. J., & Rubinstein, A. (1994). A course in game theory. The MIT Press.
71. Pajarinen, J., & Peltonen, J. (2011). Efficient planning for factored infinite-horizon DECPOMDPs. In Proc. of the International Joint Conference on Artificial Intelligence.
72. Panait, L., & Luke, S. (2005). Cooperative multi-agent learning: The state of the art. Autonomous Agents and Multi-Agent Systems, 11 (3), 387-434.
73. Puterman, M. L. (1994). Markov Decision Processes - Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc.
74. Pynadath, D. V., & Marsella, S. C. (2007). Minimal mental models. In Proceedings of the Twenty-Second AAAI Conference on Artificial Intelligence.
75. Pynadath, D. V., & Tambe, M. (2002). The communicative multiagent team decision problem: Analyzing teamwork theories and models. Journal of Artificial Intelligence Research, 16, 389-423.
76. Rabinovich, Z., Goldman, C. V., & Rosenschein, J. S. (2003). The complexity of multiagent systems: the price of silence. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
77. Rabinovich, Z., & Rosenschein, J. S. (2005). Multiagent coordination by extended Markov tracking. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
78. Ross, S., Pineau, J., Paquet, S., & Chaib-draa, B. (2008). Online planning algorithms for POMDPs. Journal of Artificial Intelligence Research, 32, 664-704.
79. Roth, M., Simmons, R., & Veloso, M. (2005). Reasoning about joint beliefs for executiontime communication decisions. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
80. Roth, M., Simmons, R., & Veloso, M. (2007). Exploiting factored representations for decentralized execution in multi-agent teams. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
81. Seuken, S., & Zilberstein, S. (2007a). Improved memory-bounded dynamic programming for decentralized POMDPs. In Proc. of Uncertainty in Artificial Intelligence.
82. Seuken, S., & Zilberstein, S. (2007b). Memory-bounded dynamic programming for DECPOMDPs. In Proc. of the International Joint Conference on Artificial Intelligence.
83. Seuken, S., & Zilberstein, S. (2008). Formal models and algorithms for decentralized decision making under uncertainty. Autonomous Agents and Multi-Agent Systems, 17 (2), 190- 250.
84. Spaan, M. T. J., Gordon, G. J., & Vlassis, N. (2006). Decentralized planning under uncertainty for teams of communicating agents. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
85. Spaan, M. T. J., & Melo, F. S. (2008). Interaction-driven Markov games for decentralized multiagent planning under uncertainty. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
86. Spaan, M. T. J., & Oliehoek, F. A. (2008). The MultiAgent Decision Process toolbox: software for decision-theoretic planning in multiagent systems. In Proc. of the AAMAS Workshop on Multi-Agent Sequential Decision Making in Uncertain Domains.
87. Spaan, M. T. J., Oliehoek, F. A., & Amato, C. (2011). Scaling up optimal heuristic search in Dec-POMDPs via incremental expansion. In Proc. of the International Joint Conference on Artificial Intelligence.
88. Spaan, M. T. J., Oliehoek, F. A., & Vlassis, N. (2008). Multiagent planning under uncertainty with stochastic communication delays. In Proc. of the International Conference on Automated Planning and Scheduling.
89. Sycara, K. P. (1998). Multiagent systems. AI Magazine, 19 (2), 79-92.
90. Szer, D., Charpillet, F., & Zilberstein, S. (2005). MAA&z.ast;: A heuristic search algorithm for solving decentralized POMDPs. In Proc. of Uncertainty in Artificial Intelligence.
91. Tsitsiklis, J., & Athans, M. (1985). On the complexity of decentralized decision making and detection problems. IEEE Transactions on Automatic Control, 30 (5), 440-446.
92. Varaiya, P., & Walrand, J. (1978). On delayed sharing patterns. IEEE Transactions on Automatic Control, 23 (3), 443-445.
93. Varakantham, P., Kwak, J. young, Taylor, M. E., Marecki, J., Scerri, P., & Tambe, M. (2009). Exploiting coordination locales in distributed POMDPs via social model shaping. In Proc. of the International Conference on Automated Planning and Scheduling.
94. Varakantham, P., Marecki, J., Yabu, Y., Tambe, M., & Yokoo, M. (2007). Letting loose a SPIDER on a network of POMDPs: Generating quality guaranteed policies. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
95. Varakantham, P., Nair, R., Tambe, M., & Yokoo, M. (2006). Winning back the cup for distributed POMDPs: planning over continuous belief spaces. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
96. Velagapudi, P., Varakantham, P., Scerri, P., & Sycara, K. (2011). Distributed model shaping for scaling to decentralized POMDPs with hundreds of agents. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
97. Vlassis, N. (2007). A Concise Introduction to Multiagent Systems and Distributed Artificial Intelligence. Morgan & Claypool Publishers.
98. Williamson, S. A., Gerding, E. H., & Jennings, N. R. (2009). Reward shaping for valuing communications during multi-agent coordination. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
99. Witwicki, S. J. (2011). Abstracting Influences for Efficient Multiagent Coordination Under Uncertainty. Unpublished doctoral dissertation, University of Michigan, Ann Arbor, Michigan, USA
100. Witwicki, S. J., & Durfee, E. H. (2010). Influence-based policy abstraction for weakly-coupled Dec-POMDPs. In Proc. of the International Conference on Automated Planning and Scheduling.
101. Wu, F., Zilberstein, S., & Chen, X. (2010a). Point-based policy generation for decentralized POMDPs. In Proc. of the International Conference on Autonomous Agents and Multi Agent Systems.
102. Wu, F., Zilberstein, S., & Chen, X. (2010b). Rollout sampling policy iteration for decentralized POMDPs. In Proc. of Uncertainty in Artificial Intelligence.
103. Wu, F., Zilberstein, S., & Chen, X. (2011). Online planning for multi-agent systems with bounded communication. Artificial Intelligence, 175 (2), 487-511.
104. Xuan, P., Lesser, V., & Zilberstein, S. (2001). Communication decisions in multi-agent cooperation: Model and experiments. In Proc. of the International Conference on Autonomous Agents.
105. Yoshizumi, T., Miura, T., & Ishida, T. (2000). A&z.ast; with partial expansion for large branching factor problems. In Proc. of the National Conference on Artificial Intelligence.
106. Zeng, Y., & Doshi, P. (2012). Exploiting model equivalences for solving interactive dynamic influence diagrams. Journal of Artificial Intelligence Research, 43, 211-255.
107. Zeng, Y., Doshi, P., Pan, Y., Mao, H., Chandrasekaran, M., & Luo, J. (2011). Utilizing partial policies for identifying equivalence of behavioral models. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence.