Human reinforcement learning subdivides structured action spaces by learning effector-specific values

Journal of Neuroscience

Volumn 29, Issue 43, 2009, Pages 13524-13531

  Gershman, Samuel J ^a   Pesaran, Bijan ^a   Daw, Nathaniel D ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

OXYGEN;

ARTICLE; CONTROLLED STUDY; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HEMISPHERE; HUMAN; HUMAN EXPERIMENT; LEARNING; MEDICAL ERROR; OXYGEN BLOOD LEVEL; PRIORITY JOURNAL; REINFORCEMENT; RIGHT HANDEDNESS;

ALGORITHMS; ANALYSIS OF VARIANCE; BRAIN; BRAIN MAPPING; CEREBROVASCULAR CIRCULATION; CHOICE BEHAVIOR; FUNCTIONAL LATERALITY; HAND; HUMANS; LEARNING; MAGNETIC RESONANCE IMAGING; MODELS, PSYCHOLOGICAL; MOTOR ACTIVITY; NEUROPSYCHOLOGICAL TESTS; OXYGEN; PSYCHOMOTOR PERFORMANCE; REINFORCEMENT (PSYCHOLOGY);

EID: 70350521769     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.2469-09.2009     Document Type: Article

References (63)

1. Astafiev SV, Shulman GL, Stanley CM, Snyder AZ, Van Essen DC, Corbetta M (2003) Functional organization of human intraparietal and frontal cortex for attending, looking, and pointing. J Neurosci 23:4689-4699.
2. Batista AP, Buneo CA, Snyder LH, Andersen RA (1999) Reach plans in eye-centered coordinates. Science 285:257-260.
3. Bayer HM, Glimcher PW (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129-141.
4. Behrens TE, Woolrich MW, Walton ME, Rushworth MF (2007) Learning the value of information in an uncertain world. Nat Neurosci 10:1214-1221.
5. Bellman RE (1957) Dynamic programming. Princeton, NJ: Princeton UP.
6. Bertin M, Schweighofer N, Doya K (2007) Multiple model-based reinforcement learning explains dopamine neuronal activity. Neural Netw 20:668-675. (Pubitemid 47241319)
7. Boorman ED, Behrens TE, Woolrich MW, Rushworth MF (2009) How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 62:733-743.
8. Brainard DH (1997) The Psychophysics Toolbox. Spat Vis 10:433-436.
9. Brischoux F, Chakraborty S, Brierley DI, Ungless MA (2009) Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proc Natl Acad Sci U S A 106:4894-4899.
10. Camerer C, Ho TH (1999) Experience-weighted attraction in learning normal-form games. Econometrica 67:827-874.
11. Chang Y, Ho T, Kaelbling LP (2003) All learning is local: multi-agent learning in global reward games. Paper presented at 17th Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada, December.
12. Coe B, Tomihara K, Matsuzawa M, Hikosaka O (2002) Visual and anticipatory bias in three cortical eye fields of the monkey during an adaptive decision-making task. J Neurosci 22:5081-5090. (Pubitemid 37465621)
13. Cohen YE, Andersen RA (2002) A common reference frame for movement plans in the posterior parietal cortex. Nat Rev Neurosci 3:553-562.
14. Connolly JD, Andersen RA, Goodale MA (2003) FMRI evidence for a "parietal reach region" in the human brain. Exp Brain Res 153:140-145.
15. Courville AC, Daw ND, Touretzky DS (2006) Bayesian theories of conditioning in a changing world. Trends Cogn Sci 10:294-300.
16. Cui H, Andersen RA (2007) Posterior parietal cortex encodes autonomously selected motor plans. Neuron 56:552-559. (Pubitemid 350026270)
17. Daw ND, Doya K (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16:199-204.
18. Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ (2006a) Cortical substrates for exploratory decisions in humans. Nature 441:876-879. (Pubitemid 43910421)
19. Daw ND, Courville AC, Touretzky DS (2006b) Representation and timing in theories of the dopamine system. Neural Comput 18:1637-1677.
20. Dayan P, Niv Y (2008) Reinforcement learning: the good, the bad and the ugly. Curr Opin Neurobiol 18:185-196.
21. Delgado MR (2007) Reward-related responses in the human striatum. Ann N Y Acad Sci 1104:70-88.
22. Doya K, Samejima K, Katagiri K, Kawato M (2002) Multiple model-based reinforcement learning. Neural Comput 14:1347-1369.
23. Ghahramani Z, Wolpert DM (1997) Modular decomposition in visuomotor learning. Nature 386:392-395.
24. Gläscher J, Hampton AN, O'Doherty JP (2009) Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making. Cereb Cortex 19:483-495.
25. Grefkes C, Fink GR (2005) The functional organization of the intraparietal sulcus in humans and monkeys. J Anat 207:3-17. (Pubitemid 41025270)
26. Grefkes C, Ritzl A, Zilles K, Fink GR (2004) Human medial intraparietal cortex subserves visuomotor coordinate transformation. Neuroimage 23:1494-1506. (Pubitemid 40260372)
27. Griffiths TL, Tenenbaum JB (2005) Structure and strength in causal induction. Cogn Psychol 51:334-384.
28. Haber SN, Fudge JL, McFarland NR (2000) Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J Neurosci 20:2369-2382.
29. Hampton AN, Bossaerts P, O'Doherty JP (2006) The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J Neurosci 26:8360-8367. (Pubitemid 44309850)
30. Hare TA, O'Doherty J, Camerer CF, Schultz W, Rangel A (2008) Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J Neurosci 28:5623-5630.
31. Houk JC, Adams JL, Barto AG (1995) A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Models of information processing in the basal ganglia (Houk JC, Davis JL, Beiser DG, eds), pp 249-270. Cambridge, MA: MIT.
32. Kalaska JF, Cisek P, Gosselin-Kessiby N (2003) Mechanisms of selection and guidance of reaching movements in the parietal lobe. Adv Neurol 93:97-119.
33. Kass RE, Raftery AE (1995) Bayes factors. J Am Stat Assoc 90:773-794.
34. Kemp C, Tenenbaum JB (2008) The discovery of structural form. Proc Natl Acad Sci U S A 105:10687-10692.
35. Knutson B, Gibbs SE (2007) Linking nucleus accumbens dopamine and blood oxygenation. Psychopharmacology (Berl) 191:813-822. (Pubitemid 46355904)
36. Koyama M, Hasegawa I, Osada T, Adachi Y, Nakahara K, Miyashita Y (2004) Functional magnetic resonance imaging of macaque monkeys performing visually guided saccade tasks: comparison of cortical eye fields with humans. Neuron 41:795-807. (Pubitemid 38326841)
37. Kriegeskorte N, Simmons WK, Bellgowan PS, Baker CI (2009) Circular analysis in systems neuroscience: the dangers of double dipping. Nat Neurosci 12:535-540.
38. Lau B, Glimcher PW (2005) Dynamic response-by-response models of matching behavior in rhesus monkeys. J Exp Anal Behav 84:555-579.
39. Lau B, Glimcher PW (2008) Value representations in the primate striatum during Matching behavior. Neuron 58:451-463. (Pubitemid 351615183)
40. Matsuda W, Furuta T, Nakamura KC, Hioki H, Fujiyama F, Arai R, Kaneko T (2009) Single nigrostriatal dopaminergic neurons form widely spread and highly dense axonal arborizations in the neostriatum. J Neurosci 29:444-453.
41. McClure SM, Berns GS, Montague PR (2003) Temporal prediction errors in a passive learning task activate human striatum. Neuron 38:339-346.
42. Murata A, Gallese V, Kaseda M, Sakata H (1996) Parietal neurons related to memory-guided hand manipulation. J Neurophysiol 75:2180-2186. (Pubitemid 26159052)
43. Musallam S, Corneil BD, Greger B, Scherberger H, Andersen RA (2004) Cognitive control signals for neural prosthetics. Science 305:258-262. (Pubitemid 38886742)
44. O'Doherty JP (2004) Reward representations and reward-related learning in the human brain: insights from neuroimaging. Curr Opin Neurobiol 14:769-776.
45. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ (2003) Temporal difference models and reward-related learning in the human brain. Neuron 38:329-337. (Pubitemid 36555657)
46. O'Reilly RC, Frank MJ (2006) Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput 18:283-328.
47. Pagnoni G, Zink CF, Montague PR, Berns GS (2002) Activity in human ventral striatum locked to errors of reward prediction. Nat Neurosci 5:97-98. (Pubitemid 34121573)
48. Pesaran B, Nelson MJ, Andersen RA (2006) Dorsal premotor neurons encode the relative position of the hand, eye, and goal during reach planning. Neuron 51:125-134. (Pubitemid 43963337)
49. Pesaran B, Nelson MJ, Andersen RA (2008) Free choice activates a decision circuit between frontal and parietal cortex. Nature 453:406-409. (Pubitemid 351693121)
50. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD (2006) Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442:1042-1045. (Pubitemid 44330261)
51. Plassmann H, O'Doherty J, Rangel A (2007) Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J Neurosci 27:9984-9988. (Pubitemid 47404004)
52. Platt ML, Glimcher PW (1999) Neural correlates of decision variables in parietal cortex. Nature 400:233-238.
53. Roesch MR, Calu DJ, Schoenbaum G (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci 10:1615-1624.
54. Russell S, Zimdars AL (2003) Q-decomposition for reinforcement learning agents. Paper presented at International Conference on Machine Learning, Washington, DC, August.
55. Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of action-specific reward values in the striatum. Science 310:1337-1340. (Pubitemid 41698867)
56. Schönberg T, Daw ND, Joel D, O'Doherty JP (2007) Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J Neurosci 27:12860-12867. (Pubitemid 350159584)
57. Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80:1-27.
58. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593-1599.
59. Schwartz GE (1978) Estimating the dimension of a model. Ann Stat 6:461-464.
60. Seo H, Lee D (2009) Behavioral and neural changes after gains and losses of conditioned reinforcers. J Neurosci 29:3627-3641.
61. Snyder LH, Batista AP, Andersen RA (1997) Coding of intention in the posterior parietal cortex. Nature 386:167-170. (Pubitemid 27131442)
62. Sugrue LP, Corrado GS, Newsome WT (2004) Matching behavior and the representation of value in the parietal cortex. Science 304:1782-1787.
63. Sutton RS, Barto AG (1998) Reinforcement learning. Cambridge, MA: MIT.