Hierarchical state-abstracted and socially augmented Q-Learning for reducing complexity in agent-based learning

Journal of Control Theory and Applications

Volumn 9, Issue 3, 2011, Pages 440-450

  Sun, Xueqing ^a   Mao, Tao ^a   Ray, Laura ^a   Shi, Dongqing ^b   Kralik, Jerald ^b  

^a DARTMOUTH COLLEGE   (United States)

^b Lamont Doherty Earth Observatory   (United States)

Author keywords

Decentralized Markov decision process; Multiagent systems; Reinforcement learning

Indexed keywords

ACTION SELECTION; ACTION SPACES; AGENT BASED; ANALYSIS RESULTS; BRAIN CIRCUITS; COMPLEX PROBLEMS; COMPUTATIONAL COSTS; HIERARCHICAL REPRESENTATION; LEARNING EFFICIENCY; LEARNING PROBLEM; MARKOV DECISION PROCESSES; MEMORY RESOURCES; MULTIROBOTS; NATURAL WORLD; Q-LEARNING; SOCIAL COGNITION; SOCIAL INTELLIGENCE; SOCIAL KNOWLEDGE; STATE ABSTRACTION; STATE REPRESENTATION; TASK COMPLETION TIME; TASK DOMAIN; TASK SPACE; THEORETICAL BOUNDS; UNCERTAIN ENVIRONMENTS;

ABSTRACTING; LEARNING ALGORITHMS; MAMMALS; MARKOV PROCESSES; MULTI AGENT SYSTEMS; PROBLEM SOLVING; REINFORCEMENT LEARNING;

COMPUTATIONAL COMPLEXITY;

EID: 79960460645     PISSN: 16726340     EISSN: 10008152     Source Type: Journal    
DOI: 10.1007/s11768-011-1047-6     Document Type: Article

References (33)

1. D. Bernstein, R. Givan, N. Immerman, et al. The complexity of decentralized control of Markov decision processes. Mathematics of Operations Research, 2002, 27(4): 819-840.
2. Z. Rabinovich, C. V. Goldman, J. S. Rosenschein. The complexity of multiagent systems: The price of silence. Proceedings of the 2nd International Joint Conference on Autonomous Agents and Multiagent Systems, Melbourne, Australia, 2003: 1102-1103.
3. R. Granger. Engines of the brain: the computational instruction set of human cognition. AI Magazine, 2006: 27(2): 15-32.
4. A. Rodriguez, J. Whitson, R. Granger. Derivation and analysis of basic computational operations of thalamocortical circuits. Journal of Cognitive Neuroscience, 2004: 16(5): 856-877.
5. J. B. Silk. Social components of fitness in primate groups. Science, 2007, 317(5843): 1347-1351.
6. B. B. Smuts, D. L. Cheney, R. M. Seyfarth, et al., eds. Primate Societies. Chicago: University of Chicago Press, 1987.
7. C. Boesch, H. Boesch-Achermann. The Chimpanzees of the Tai Forest: Behavioral Ecology and Evolution. Oxford: Oxford University Press, 2000.
8. M. Byron. Satisficing and optimality. Ethics, 1998: 109(1): 67-93.
9. W. Byrne, A. Whiten. Machiavellian Intelligence: Social Expertise and the Evolution of Intellect in Monkeys, Apes and Humans. Oxford: Clarendon Press, 1988.
10. C. W. Clark, M. Mangel. Dynamic State Variable Models in Ecology: Methods and Applications. New York: Oxford University Press, 2000.
11. L. A. Giraldeau, T. Caraco. Social Foraging Theory. Princeton: Princeton University Press, 2000.
12. S. A. Rands, G. Cowlishaw, R. A. Pettifor, et al. The spontaneous emergence of leaders and followers in a foraging pair. Nature, 2003, 423(6938): 432-434.
13. R. C. Connor. Dolphin social intelligence: complex alliance relationships in bottlenose dolphins and a consideration of selective environments for extreme brain size evolution in mammals. Philosophical Transactions of the Royal Society B: Biological Sciences, 2007: 362(1480): 587-602.
14. C. Boesch. Complex cooperation among tai chimpanzees. Animal Social Complexity. F. B. M. Waal, P. L. Tyack, eds. Cambridge: Harvard University Press, 2003: 93-110.
15. C. Goldman, S. Zilberstein. Decentralized control of cooperative systems: Categorization and complexity analysis. Journal of Artificial Intelligence Research, 2004: 22(1): 143-174.
16. J. Hu, M. Wellman. Nash Q-learning for general-sum stochastic games. Journal of Machine Learning Research, 2003, 4: 1039-1069.
17. D. Fudenberg, J. Tirole. Game Theory. Cambridge: MIT Press, 1991.
18. R. Vaughan, N. Sumpter, A. Frost, et al. Experiments in automatic flock control. Robotics and Autonomous Systems, 2000: 31(1/2): 109-116.
19. D. Busquets, R. L. de Mantaras, C. Sierra, et al. Reinforcement learning for landmark-based robot navigation. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems. New York: ACM, 2002: 841-843.
20. A. Sanjeev, B. Boaz. Complexity Theory: A Modern Approach. Cambridge: Cambridge University Press, 2009.
21. C. J. C. H. Watkins, P. Dayan. Technical note: Q-learning. Machine Learning, 1992: 8(3/4): 279-292.
22. J. Tsitsiklis. Asynchronous stochastic approximation and Q-learning. Machine Learning, 1994: 16(3): 185-202.
23. M. L. Littman. Markov games as a framework for multiagent reinforcement learning. Proceedings of the 11th International Conference on Machine Learning, San Francisco, CA: Morgan Kaufmann Publishers Inc., 1994: 157-163.
24. C. Claus, C. Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. Proceedings of the 15th National Conference on Artificial Intelligence, Menlo Park, CA: AAAI Press, 1998: 746-752.
25. L. D. Pyeatt, A. E. Howe. Decision Tree Function Approximation in Reinforcement Learning. Report TR CS-98-112. Fort Collins, CO: Colorado State University, 1998.
26. W. T. B. Uther, M. M. Veloso. Tree based discretization for continuous state space reinforcement learning. Proceedings of the 15th National Conference on Artificial Intelligence, Menlo Park, CA: American Association for Artificial Intelligence (AAAI), 1998: 769-774.
27. X. Sun, T. Mao, J. D. Kralik, et al. Cooperative multirobot reinforcement learning: a framework in hybrid state space. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), New York: IEEE, 2009: 1190-1196.
28. T. Mao, X. Sun, L. E. Ray. Role selection in multi-robot systems using abstract state-based reinforcement learning. Proceedings of the 14th IASTED International Conference on Robotics and Applications, Boston, MA, 2009.
29. M. Z. Sauter, D. Shi, J. D. Kralik. Multi-agent reinforcement learning and chimpanzee hunting. Proceedings of IEEE International Conferenceon Robotics and Biomimetics (ROBIO), New York: IEEE, 2009: 622-626.
30. D. Shi, M. Z. Sauter, J. D. Kralik. Distributed, heterogeneous, multiagent social coordination via reinforcement learning. Proceedings of IEEE International Conference on Robotics and Biomimetics (ROBIO), New York: IEEE, 2009: 653-658.
31. M. Lauer, M. Riedmiller. An algorithm for distributed reinforcement learning in cooperative multi-agent systems. Proceedings of the 17th International Conference on Machine Learning, San Francisco, CA: Morgan Kaufmann Publishers Inc., 2000: 535-542.
32. R. C. Miller. The significance of the gregarious habit. Ecology, 1922: 3(2): 122-126.
33. Webots Reference Manual. Professional Mobile Robot Simulation Software. Cyberbotics Ltd. http://www. cyberbotics. com.