Credit assignment in multiple goal embodied visuomotor behavior

Frontiers in Psychology

Volumn 1, Issue NOV, 2010, Pages

  Rothkopf, Constantin A ^a   Ballard, Dana H ^b  

^a FRANKFURT INSTITUTE FOR ADVANCED STUDIES   (Germany)

^b University of Texas at Austin   (United States)

Author keywords

Credit assignment; Learning; Modules; Reinforcement; Reward

Indexed keywords

EID: 80054091681     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2010.00173     Document Type: Article

References (60)

1. Anderson, J. (1983). The Architecture of Cognition. Cambridge, MA: Harvard University Press.
2. Arkin, R. (1998). Behavior Based Robotics. Cambridge, MA: MIT Press.
3. Badler, N., Palmer, M., and Bindiganavale, R. (1999). Animation control for real-time virtual humans. Commun. ACM 42, 64-73.
4. Badler, N. I., Phillips, C. B., and Webber, B. L. (1993). Simulating Humans: Computer Graphics Animation and Control. New York, NY: Oxford University Press.
5. Ballard, D. H., Hayhoe, M. M., Pook, P. K., and Rao, R. P. N. (1997). Deictic codes for the embodiment of cognition. Behav. Brain Sci. 20, 723-767.
6. Barsalou, L. (2009). Simulation, situated conceptualization, and prediction. Phil. Trans. R. Soc. B 364, 1281-1289.
7. Barto, A. G., and Mahadevan, S. (2003). Recent advances in hierarchical reinforcement learning. Discrete Event Dyn. Syst. 13, 41-77.
8. Bonasso, R. P., Firby, J. R., Gat, E., Kortenkamp, D., Miller, D. D. P., and Slack, M. G. (1997). Experiences with an architecture for intelligent, reactive agents. J. Exp. Theor. Artif. Intell. 9, 237-256.
9. Brooks, R. (1986). A robust layered control system for a mobile robot. IEEE J. Robot. Autom. 2, 14-23.
10. Bryson, J. J., and Stein, L. A. (2001). "Modularity and design in reactive intelligence," in International Joint Conference on Artificial Intelligence, Seattle, WA.
11. Daw, N. D., Niv, Y., and Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704-1711.
12. Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., and Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature 441, 876-879.
13. Dayan, P., and Hinton, G. E. (1992). "Feudal reinforcement learning," in Advances in Neural Information Processing Systems, (San Francisco, CA: Morgan Kaufmann Publishers Inc.) 5, 271-278.
14. Dearden, R. Boutilier, C., and Goldszmidt, M. (2000). Stochastic dynamic programming with factored representations. Artif. Intell. 121, 49-107.
15. Doya, K., Samejima, K., Katagiri K., and Kawato, M. (2002). Multiple model-based reinforcement learning. Neural Comput. 14, 1347-1369.
16. Firby, R. J., Kahn, R. E., Prokopowicz, P. N., and Swain, M. J. (1995). "An architecture for vision and action," in International Joint Conference on Artificial Intelligence (Montreal, Canada; Morgan Kaufmann Publishers Inc.), 72-79.
17. Franco-Watkins, A. M., Rickard, T. C., and Pashler, H. (2010). Taxing executive processes does not necessarily increase impulsive decision making. Exp. Psychol. 57, 193-201.
18. Glenberg, A. M. (2010). Embodiment as a unifying perspective for psychology. Wiley Interdiscip. Rev. Cogn. Sci. 1, 586-596.
19. Gottfried, J. A., O'Doherty, J., and Dolan, R. J. (2003). Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science 301, 1104-1107.
20. Guestrin, C. E., Koller, D., Parr, R., and Venkataraman, S. (2003). Efficient solution algorithms for factored MDPs. J. Artif. Intell. Res. 19, 399-468.
21. Haruno, M., and Kawato, M. (2006). Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. J. Neurophysiol. 95, 948-959.
22. Hikosaka, O., Bromberg-Martin, E., Hong, S., and Matsumoto, M. (2008). New insights on the subcortical representation of reward. Curr. Opin. Neurobiol. 18, 203-208.
23. Humphrys, M. (1996). "Action selection methods using reinforcement learning," in From Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior, eds P. Maes, M. Mataric, J.-A. Meyer, J. Pollack, and S. W. Wilson (Cambridge, MA: MIT Press, Bradford Books), 135-144.
24. Karlsson, J. (1997). Learning to Solve Multiple Goals. PhD thesis, University of Rochester, Rochester.
25. Laird, J. E., Newell, A., and Rosenblum, P. S. (1987). Soar: an architecture for general intelligence. Artif. Intell. 33, 1-64.
26. Langley, P., and Choi, D. (2006). Learning recursive control programs from problem solving. J. Mach. Learn. Res. 7, 493-518.
27. Luck, S. J., and Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature 390, 279-281.
28. Meuleau, N., Hauskrecht, M., Kim, K.-E., Peshkin, L., Kaelbling, L., Dean, T., and Boutilier, C. (1998). "Solving very large weakly coupled Markov decision processes," in AAAI/IAAI, 165-172.
29. Morris, G., Nevet, A., Arkadir, D., Vaadia, E., and Bergman, H. (2006). Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9, 1057-1063.
30. Neisser, U. (1967). Cognitive Psychology. New York: Appleton-Century-Crofts.
31. Newell, A. (1990). Unified Theories of Cognition. Cambridge, MA: Harvard University Press.
32. Ng, A. Y., Harada, D., and Russell, S. (1999). "Policy invariance under reward transformations: theory and application to reward shaping," in Proceedings of the Sixteenth International Conference on Machine Learning, Bled, Slovenia.
33. Parr, R., and Russell, S. (1997). "Reinforcement learning with hierarchies of machines," in Advances in Neural Information Processing Systems, M. I. Jordan, M. J. Kearns, and S. A. Solla. (Cambridge, MA: MIT Press), 1043-1049.
34. Pessiglione, M., Seymour, B., Flandin, G., Dolan, R. J., and Frith, C. D. (2006). Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442, 1042-1045.
35. Rangel, A., and Hare, T. (2010). Neural computations associated with goal-directed choice. Curr. Opin. Neurobiol. 20, 262-270.
36. Ritter, S., Anderson, J. R., Cytrynowicz, M., and Medvedeva, O. (1998). Authoring content in the pat algebra tutor. J. Interact. Media Educ. 98, 1-30.
37. Rothkopf, C. A., and Ballard, D. H. (2009). Image statistics at the point of gaze during human navigation. Vis. Neurosci. 26, 81-92.
38. Rothkopf, C. A., and Ballard, D. H. (2010). "Learning and coordinating reper-toires of behaviors: credit assignment and module activation," in Intrinsically Motivated Cumulative Learning in Natural and Artificial Systems, Eds G. Baldassarre and M. Mirolli (in press).
39. Roy, D. K., and Pentland, A. P. (2002). Learning words from sights and sounds: a computational model. Cogn. Sci. 26, 113-146.
40. Russell, S., and Zimdars, A. (2003). "Q-decomposition for reinforcement learning agents," in Proceedings of the International Conference on Machine Learning, Washington, DC.
41. Sallans, B., and Hinton, G. E. (2004). Reinforcement learning with factored states and actions. J. Mach. Learn. Res. 5, 1063-1088.
42. Samejima, K., Doya, K., and Kawato, M. (2003). Inter-module credit assignment in modular reinforcement learning. Neural Netw. 16, 985-994.
43. Schultz, W. (2000). Multiple reward signals in the brain. Nat. Rev. Neurosci. 1, 199-207.
44. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593-1599.
45. Shinoda, H., Hayhoe, M. M., and Shrivastava, A. (2001). What controls attention in natural environments? Vis. Res. 41, 3535-3546.
46. Singh, S., and Cohn, D. (1998). How to dynamically merge Markov decision processes. Neural Inf. Process. Syst. 10, 1057-1063.
47. Sprague, N., and Ballard, D. (2003). "Multiple-goal reinforcement learning with modular sarsa(0)," in International Joint Conference on Artificial Intelligence, Acapulco.
48. Sprague, N., Ballard, D., and Robinson, A. (2007). Modeling embodied visual behaviors. ACM Trans. Appl. Percept. 4, 1-23.
49. Sun, R. (2006). Cognition and Multi-Agent Interaction, chapter 4. Cambridge: Cambridge University Press, 79-99.
50. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
51. Sutton, R. S., Precup, D., and Singh, S. P. (1999). Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif. Intell. 112, 181-211.
52. Terzopoulos, D. (1999). Artificial life for computer graphics. Commun. ACM 42, 32-42.
53. Terzopoulos, D., Tu, X., and Grzeszczuk, R. (1994). Artificial fishes: autonomous locomotion, perception, behavior, and learning in a simulated physical world. Artif. Life, 1, 327-351.
54. Tolman, E. C. (1948). Cognitive maps in rats and men. Psychol. Rev. 55, 189-208.
55. Treisman, A. M. (1980). A feature-integration theory of attention. Cogn. Psychol. 12, 97-136.
56. Trick, L. M., and Pylyshyn, Z. W. (1994). Why are small and large numbers enumerated differently? A limited-capacity preattentive stage in vision. Psychol. Rev. 101, 80-102.
57. Ullman, S. (1984). Visual routines. Cognition 18, 97-157.
58. Vigorito, C. M., and Barto, A. G. (2010). Intrinsically motivated hierarchical skill learning in structured environments. IEEE Trans. Auton. Ment. Dev. 2, 132-143.
59. Warren, W. H., and Fajen, B. R. (2004). Behavioral dynamics of human locomotion. Ecol. Psychol. 16, 61-66.
60. Watkins, C. J. C. H. (1989). Learning from delayed rewards. PhD thesis, University of Cambridge, Cambridge.