Optimal and approximate Q-value functions for decentralized POMDPs

Journal of Artificial Intelligence Research

Volumn 32, Issue , 2008, Pages 289-353

  Oliehoek, Frans A ^a   Spaan, Matthijs T J ^b   Vlassis, Nikos ^c  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b INSTITUTO SUPERIOR TÉCNICO   (Portugal)

^c TECHNICAL UNIVERSITY OF CRETE   (Greece)

Author keywords

[No Author keywords available]

Indexed keywords

DECISION MAKING; DYNAMIC PROGRAMMING;

BENCH-MARK PROBLEMS; DECISION-THEORETIC PLANNING; EFFICIENT COMPUTATION; EXPERIMENTAL EVALUATION; OPTIMAL POLICIES; SEQUENTIAL DECISION MAKING; SINGLE-AGENT; VALUE FUNCTIONS;

RATIONAL FUNCTIONS;

EID: 52249098423     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.2447     Document Type: Article

References (86)

1. Aicardi, M., Davoli, F., & Minciardi, R. (1987). Decentralized optimal control of Markov chains with a common past information set. IEEE Transactions on Automatic Control, 32(11), 1028-1031.
2. Altman, E. (2002). Applications of Markov decision processes in communication networks. In Feinberg, E. A., & Shwartz, A. (Eds.), Handbook of Markov Decision Processes: Methods and Applications. Kluwer Academic Publishers.
3. Amato, C., Bernstein, D. S., & Zilberstein, S. (2006). Optimal fixed-size controllers for decentralized POMDPs. In Proc. of the AAMAS Workshop on Multi-Agent Sequential Decision Making in Uncertain Domains (MSDM).
4. Amato, C., Bernstein, D. S., & Zilberstein, S. (2007a). Optimizing memory-bounded controllers for decentralized POMDPs. In Proc. of Uncertainty in Artificial Intelligence.
5. Amato, C., Carlin, A., & Zilberstein, S. (2007b). Bounded dynamic programming for decentralized POMDPs. In Proc. of the AAMAS Workshop on Multi-Agent Sequential Decision Making in Uncertain Domains (MSDM).
6. Arai, T., Pagello, E., & Parker, L. (2002). Editorial: Advances in multirobot systems. IEEE Transactions on Robotics and Automation, 18(5), 655-661.
7. Aras, R., Dutech, A., & Charpillet, F. (2007). Mixed integer linear programming for exact finite-horizon planning in decentralized POMDPs. In Proc. of the International Conference on Automated Planning and Scheduling.
8. Becker, R., Lesser, V., & Zilberstein, S. (2005). Analyzing myopic approaches for multiagent communication. In Proc. of the International Conference on Intelligent Agent Technology, pp. 550-557.
9. Becker, R., Zilberstein, S., & Lesser, V. (2004a). Decentralized Markov decision processes with event-driven interactions. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 302-309.
10. Becker, R., Zilberstein, S., Lesser, V., & Goldman, C. V. (2004b). Solving transition independent decentralized Markov decision processes. Journal of Artificial Intelligence Research, 22, 423-455.
11. Bernstein, D. S. (2005). Complexity Analysis and Optimal Algorithms for Decentralized Decision Making. Ph.D. thesis, University of Massachusets Amherst.
12. Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002). The complexity of decentralized control of Markov decision processes. Mathematics of Operations Research, 27(4), 819-840.
13. Bernstein, D. S., Hansen, E. A., & Zilberstein, S. (2005). Bounded policy iteration for decentralized POMDPs. In Proc. of the International Joint Conference on Artificial Intelligence, pp. 1287-1292.
14. Bertsekas, D. P. (2005). Dynamic Programming and Optimal Control (3rd edition)., Vol. I. Athena Scientific.
15. Beynier, A., & Mouaddib, A.-I. (2005). A polynomial algorithm for decentralized Markov decision processes with temporal constraints. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 963-969.
16. Beynier, A., & Mouaddib, A.-I. (2006). An iterative algorithm for solving constrained decentralized Markov decision processes. In Proc. of the National Conference on Artificial Intelligence.
17. Binmore, K. (1992). Fun and Games. D.C. Heath and Company.
18. de Boer, P.-T., Kroese, D. P., Mannor, S., & Rubinstein, R. Y. (2005). A tutorial on the cross-entropy method. Annals of Operations Research, 134(1), 19-67.
19. Boutilier, C. (1996). Planning, learning and coordination in multiagent decision processes. In Proc. of the 6th Conference on Theoretical Aspects of Rationality and Knowledge, pp. 195-210.
20. Boutilier, C., Dean, T., & Hanks, S. (1999). Decision-theoretic planning: Structural assumptions and computational leverage. Journal of Artificial Intelligence Research, 11, 1-94.
21. Chades, I., Scherrer, B., & Charpillet, F. (2002). A heuristic approach for solving decentralized-POMDP: assessment on the pursuit problem. In Proc. of the 2002 ACM Symposium on Applied Computing, pp. 57-62.
22. Cogill, R., Rotkowitz, M., Roy, B. V., & Lall, S. (2004). An approximate dynamic programming approach to decentralized control of stochastic systems. In Proc. of the 2004 Allerton Conference on Communication, Control, and Computing.
23. Emery-Montemerlo, R. (2005). Game-Theoretic Control for Robot Teams. Ph.D. thesis, Carnegie Mellon University.
24. Emery-Montemerlo, R., Gordon, G., Schneider, J., & Thrun, S. (2004). Approximate solutions for partially observable stochastic games with common payoffs. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 136-143.
25. Emery-Montemerlo, R., Gordon, G., Schneider, J., & Thrun, S. (2005). Game theoretic control for robot teams. In Proc. of the IEEE International Conference on Robotics and Automation, pp. 1175-1181.
26. Fernández, J. L., Sanz, R., Simmons, R. G., & Diéguez, A. R. (2006). Heuristic anytime approaches to stochastic decision processes. Journal of Heuristics, 12(3), 181-209.
27. Gmytrasiewicz, P. J., & Doshi, P. (2005). A framework for sequential planning in multiagent settings. Journal of Artificial Intelligence Research, 24, 49-79.
28. Goldman, C. V., Allen, M., & Zilberstein, S. (2007). Learning to communicate in a decentralized environment. Autonomous Agents and Multi-Agent Systems, 15(1), 47-90.
29. Goldman, C. V., & Zilberstein, S. (2003). Optimizing information exchange in cooperative multi-agent systems. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 137-144.
30. Goldman, C. V., & Zilberstein, S. (2004). Decentralized control of cooperative systems: Categorization and complexity analysis.. Journal of Artificial Intelligence Research, 22, 143-174.
31. Guestrin, C., Koller, D., Parr, R., & Venkataraman, S. (2003). Efficient solution algorithms for factored MDPs. Journal of Artificial Intelligence Research, 19, 399-468.
32. Hansen, E. A., Bernstein, D. S., & Zilberstein, S. (2004). Dynamic programming for partially observable stochastic games. In Proc. of the National Conference on Artificial Intelligence, pp. 709-715.
33. Hauskrecht, M. (2000). Value-function approximations for partially observable Markov decision processes.. Journal of Artificial Intelligence Research, 13, 33-94.
34. Hsu, K., & Marcus, S. (1982). Decentralized control of finite state Markov processes. IEEE Transactions on Automatic Control, 27(2), 426-431.
35. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2), 99-134.
36. Kim, Y., Nair, R., Varakantham, P., Tambe, M., & Yokoo, M. (2006). Exploiting locality of interaction in networked distributed POMDPs. In Proc. of the AAAI Spring Symposium on Distributed Plan and Schedule Management.
37. Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I., & Osawa, E. (1997). RoboCup: The robot world cup initiative. In Proc. of the International Conference on Autonomous Agents.
38. Kitano, H., Tadokoro, S., Noda, I., Matsubara, H., Takahashi, T., Shinjoh, A., & Shimada, S. (1999). Robocup rescue: Search and rescue in large-scale disasters as a domain for autonomous agents research. In Proc. of the International Conference on Systems, Man and Cybernetics, pp. 739-743.
39. Koller, D., Megiddo, N., & von Stengel, B. (1994). Fast algorithms for finding randomized strategies in game trees. In Proc. of the 26th ACM Symposium on Theory of Computing, pp. 750-759.
40. Koller, D., & Pfeffer, A. (1997). Representations and solutions for game-theoretic problems. Artificial Intelligence, 94(1-2), 167-215.
41. Kuhn, H. (1953). Extensive games and the problem of information. Annals of Mathematics Studies, 28, 193-216.
42. Lesser, V., Ortiz Jr., C. L., & Tambe, M. (Eds.). (2003). Distributed Sensor Networks: A Multiagent Perspective, Vol. 9. Kluwer Academic Publishers.
43. Littman, M., Cassandra, A., & Kaelbling, L. (1995). Learning policies for partially observable environments: Scaling up. In Proc. of the International Conference on Machine Learning, pp. 362-370.
44. Marecki, J., & Tambe, M. (2007). On opportunistic techniques for solving decentralized Markov decision processes with temporal constraints. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 1-8.
45. Nair, R., Tambe, M., & Marsella, S. (2003). Team formation for reformation in multiagent domains like RoboCupRescue. In Proc. of Robo Cup-2002 International Symposium.
46. Nair, R., Roth, M., & Yohoo, M. (2004). Communication for improving policy computation in distributed POMDPs. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 1098-1105.
47. Nair, R., Tambe, M., & Marsella, S. (2002). Team formation for reformation. In Proc. of the AAAI Spring Symposium on Intelligent Distributed and Embedded Systems.
48. Nair, R., Tambe, M., & Marsella, S. (2003a). Role allocation and reallocation in multiagent teams: towards a practical analysis. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 552-559.
49. Nair, R., Tambe, M., Yokoo, M., Pynadath, D. V., & Marsella, S. (2003b). Taming decentralized POMDPs: Towards efficient policy computation for multiagent settings. In Proc. of the International Joint Conference on Artificial Intelligence, pp. 705-711.
50. Nair, R., Varakantham, P., Tambe, M., & Yokoo, M. (2005). Networked distributed POMDPs: A synthesis of distributed constraint optimization and POMDPs. In Proc. of the National Conference on Artificial Intelligence, pp. 133-139.
51. Nash, J. F. (1950). Equilibrium points in N-person games. Proc. of the National Academy of Sciences of the United States of America, 36, 48-49.
52. Oliehoek, F., & Vlassis, N. (2006). Dec-POMDPs and extensive form games: equivalence of models and algorithms. Ias technical report IAS-UVA-06-02, University of Amsterdam, Intelligent Systems Lab, Amsterdam, The Netherlands.
53. Oliehoek, F. A., Kooij, J. F., & Vlassis, N. (2007a). A cross-entropy approach to solving Dec-POMDPs. In Proc. of the International Symposium on Intelligent and Distributed Computing, pp. 145-154.
54. Oliehoek, F. A., Spaan, M. T. J., & Vlassis, N. (2007b). Dec-POMDPs with delayed communication. In Proc. of the AAMAS Workshop on Multi-Agent Sequential Decision Making in Uncertain Domains (MSDM).
55. Oliehoek, F. A., Spaan, M. T. J., Whiteson, S., & Vlassis, N. (2008). Exploiting locality of interaction in factored Dec-POMDPs. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems.
56. Oliehoek, F. A., & Visser, A. (2006). A hierarchical model for decentralized fighting of large scale urban fires. In Proc. of the AAMAS'06 Workshop on Hierarchical Autonomous Agents and Multi-Agent Systems (H-AAMAS), pp. 14-21.
57. Oliehoek, F. A., & Vlassis, N. (2007). Q-value functions for decentralized POMDPs. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 833-840.
58. Ooi, J. M., & Wornell, G. W. (1996). Decentralized control of a multiple access broadcast channel: Performance bounds. In Proc. of the 35th Conference on Decision and Control, pp. 293-298.
59. Osborne, M. J., & Rubinstein, A. (1994). A Course in Game Theory. The MIT Press.
60. Papadimitriou, C. H., & Tsitsiklis, J. N. (1987). The complexity of Markov decision processes. Mathematics of Operations Research, 12(3), 441-451.
61. Paquet, S., Tobin, L., & Chaib-draa, B. (2005). An online POMDP algorithm for complex multiagent environments. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems.
62. Peshkin, L. (2001). Reinforcement Learning by Policy Search. Ph.D. thesis, Brown University.
63. Peshkin, L., Kim, K.-E., Meuleau, N., & Kaelbling, L. P. (2000). Learning to cooperate via policy search. In Proc. of Uncertainty in Artificial Intelligence, pp. 307-314.
64. Pineau, J., Gordon, G., & Thrun, S. (2003). Point-based value iteration: An anytime algorithm for POMDPs. In Proc. of the International Joint Conference on Artificial Intelligence, pp. 1025-1032.
65. Puterman, M. L. (1994). Markov Decision Processes-Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc.
66. Pynadath, D. V., & Tambe, M. (2002). The communicative multiagent team decision problem: Analyzing teamwork theories and models. Journal of Artificial Intelligence Research, 16, 389-423.
67. Romanovskii, I. (1962). Reduction of a game with complete memory to a. matrix game. Soviet Mathematics, 3, 678-681.
68. Roth, M., Simmons, R., & Veloso, M. (2005). Reasoning about joint beliefs for executiontime communication decisions. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 786-793.
69. Roth, M., Simmons, R., & Veloso, M. (2007). Exploiting factored representations for decentralized execution in multi-agent teams. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 467-463.
70. Russell, S., & Norvig, P. (2003). Artificial Intelligence: A Modern Approach (2nd edition). Pearson Education.
71. Schoute, F. C. (1978). Symmetric team problems and multi access wire communication. Automatica, 14, 255-269.
72. Seuken, S., & Zilberstein, S. (2007a). Improved memory-bounded dynamic programming for decentralized POMDPs. In Proc. of Uncertainty in Artificial Intelligence.
73. Seuken, S., & Zilberstein, S. (2007b). Memory-bounded dynamic programming for DECPOMDPs.. In Proc. of the International Joint Conference on Artificial Intelligence, pp. 2009-2015.
74. Sondik, E. J. (1971). The optimal control of partially observable Markov decision processes. Ph.D. thesis, Stanford University.
75. Spaan, M. T. J., Gordon, G. J., & Vlassis, N. (2006). Decentralized planning under uncertainty for teams of communicating agents. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 249-256.
76. Spaan, M. T. J., & Melo, F. S. (2008). Interaction-driven Markov games for decentralized multiagent planning under uncertainty. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems.
77. Spaan, M. T. J., & Vlassis, N. (2005). Perseus: Randomized point-based value iteration for POMDPs. Journal of Artificial Intelligence Research, 24, 195-220.
78. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. The MIT Press.
79. Szer, D., & Charpillet, F. (2005). An optimal best-first search algorithm for solving infinite horizon DEC-POMDPs. In Proc. of the European Conference on Machine Learning, pp. 389-399.
80. Szer, D., & Charpillet, F. (2006). Point-based dynamic programming for DEC-POMDPs.. In Proc. of the National Conference on Artificial Intelligence.
81. Szer, D., Charpillet, F., & Zilberstein, S. (2005). MAA*: A heuristic search algorithm for solving decentralized POMDPs. In Proc. of Uncertainty in Artificial Intelligence, pp. 576-583.
82. Tao, N., Baxter, J., & Weaver, L. (2001). A multi-agent policy-gradient approach to network routing. In Proc. of the International Conference on Machine Learning, pp. 553-560.
83. Varakantham, P., Marecki, J., Yabu, Y., Tambe, M., & Yokoo, M. (2007). Letting loose a SPIDER on a network of POMDPs: Generating quality guaranteed policies. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems.
84. Varakantham, P., Nair, R., Tambe, M., & Yokoo, M. (2006). Winning back the cup for distributed POMDPs: planning over continuous belief spaces. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 289-296.
85. Wu, J., & Durfee, E. H. (2006). Mixed-integer linear programming for transition-independent decentralized MDPs. In Proc. of the International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 1058-1060.
86. Xuan, P., Lesser, V., & Zilberstein, S. (2001). Communication decisions in multi-agent cooperation: Model and experiments. In Proc. of the International Conference on Autonomous Agents.