Generalization of value in reinforcement learning by humans

European Journal of Neuroscience

Volumn 35, Issue 7, 2012, Pages 1092-1104

  Wimmer, G Elliott ^a   Daw, Nathaniel D ^b   Shohamy, Daphna ^a  

^a Och Spine at New York Presbyterian Hospitals   (United States)

^b NEW YORK UNIVERSITY   (United States)

Author keywords

Computational model; Hippocampus; Memory; Reward; Ventral striatum

Indexed keywords

ADULT; ARTICLE; BEHAVIOR; BOLD SIGNAL; CONTROLLED STUDY; CORPUS STRIATUM; DECISION MAKING; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HIPPOCAMPUS; HUMAN; HUMAN EXPERIMENT; IMAGE ANALYSIS; LEARNING; MALE; MEMORY; NORMAL HUMAN; PRIORITY JOURNAL; PROBABILITY; REINFORCEMENT; SOCIAL PSYCHOLOGY; STIMULUS RESPONSE;

ADOLESCENT; ADULT; BASAL GANGLIA; FEMALE; GENERALIZATION (PSYCHOLOGY); HUMANS; LEARNING; MALE; PHOTIC STIMULATION; PSYCHOMOTOR PERFORMANCE; REINFORCEMENT (PSYCHOLOGY); REWARD; YOUNG ADULT;

EID: 84859339117     PISSN: 0953816X     EISSN: 14609568     Source Type: Journal    
DOI: 10.1111/j.1460-9568.2012.08017.x     Document Type: Article

References (107)

1. Adcock, R.A., Thangavel, A., Whitfield-Gabrieli, S., Knutson, B. & Gabrieli, J.D. (2006) Reward-motivated learning: mesolimbic activation precedes memory formation. Neuron, 50, 507-517.
2. Akaike, H. (1974) A new look at the statistical model identification. IEEE Trans. Automat. Contr., 19, 716-723.
3. Balleine, B.W., Daw, N.D. & O'Doherty, J.P. (2008) Multiple forms of value learning and the function of dopamine. In Glimcher, P.W., Camerer, C.F., Poldrack, R.A. & Fehr, E. (Eds), Neuroeconomics: Decision Making and the Brain. Academic Press, New York, NY.
4. Boorman, E.D., Behrens, T.E., Woolrich, M.W. & Rushworth, M.F. (2009) How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action Neuron, 62, 733-743.
5. Bornstein, A.M. & Daw, N.D. (2012) Dissociating hippocampal and striatal contributions to sequential prediction learning. Eur. J. Neurosci., 35, in press.
6. Brainard, D.H. (1997) The Psychophysics Toolbox. Spat. Vis., 10, 433-436.
7. Bromberg-Martin, E.S., Matsumoto, M., Hong, S. & Hikosaka, O. (2010) A pallidus-habenula-dopamine pathway signals inferred stimulus values. J. Neurophysiol., 104, 1068-1076.
8. Buckner, R.L. (2010) The role of the hippocampus in prediction and imagination. Annu. Rev. Psychol. 61, 27-48, C21-28.
9. Camerer, C. & Ho, T.H. (1999) Experience-weighted attraction in learning normal-form games. Econometrica, 67, 827-874.
10. Cohen, N.J. & Eichenbaum, H. (1993) Memory, Amnesia, and the Hippocampal System. MIT Press, Cambridge, MA.
11. Cohen, M.X., Schoene-Bake, J.C., Elger, C.E. & Weber, B. (2009) Connectivity-based segregation of the human striatum predicts personality characteristics. Nat. Neurosci., 12, 32-34.
12. Corrado, G.S., Sugrue, L.P., Seung, H.S. & Newsome, W.T. (2005) Linear-Nonlinear-Poisson models of primate choice dynamics. J. Exp. Anal. Behav., 84, 581-617.
13. Courville, A.C., Daw, N.D. & Touretzky, D.S. (2006) Bayesian theories of conditioning in a changing world. Trends Cogn. Sci., 10, 294-300.
14. Coutureau, E., Killcross, A.S., Good, M., Marshall, V.J., Ward-Robinson, J. & Honey, R.C. (2002) Acquired equivalence and distinctiveness of cues: II. Neural manipulations and their implications. J. Exp. Psychol. Anim. Behav. Process., 28, 388-396.
15. Dahlström, A. & Fuxe, K. (1964) Localization of monoamines in the lower brain stem. Experientia, 20, 398-399.
16. Davachi, L. (2006) Item, context and relational episodic encoding in humans. Curr. Opin. Neurobiol., 16, 693-700.
17. Daw, N.D. (2011) Trial-by-trial data analysis using computational models. In: Delgado, M.R., Phelps, E.A. & Robbins, T.W. (Eds), Decision making, affect, and learning: attention and performance XXIII. Oxford University Press, Oxford, UK, pp. 3-38.
18. Daw, N.D. & Doya, K. (2006) The computational neurobiology of learning and reward. Curr. Opin. Neurobiol., 16, 199-204.
19. Daw, N.D. & Shohamy, D. (2008) The cognitive neuroscience of motivation and learning. Soc. Cogn., 26, 593-620.
20. Daw, N.D., Niv, Y. & Dayan, P. (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci., 8, 1704-1711.
21. Daw, N.D., O'Doherty, J.P., Dayan, P., Seymour, B. & Dolan, R.J. (2006) Cortical substrates for exploratory decisions in humans. Nature, 441, 876-879.
22. Daw, N.D., Gershman, S.J., Seymour, B., Dayan, P. & Dolan, R.J. (2011) Model-based influences on humans' choices and striatal prediction errors. Neuron, 69, 1204-1215.
23. Deichmann, R., Gottfried, J.A., Hutton, C. & Turner, R. (2003) Optimized EPI for fMRI studies of the orbitofrontal cortex. Neuroimage, 19, 430-441.
24. Delgado, M.R., Miller, M.M., Inati, S. & Phelps, E.A. (2005) An fMRI study of reward-related probability learning. Neuroimage, 24, 862-873.
25. Dickerson, K.C., Li, J. & Delgado, M.R. (2011) Parallel contributions of distinct human memory systems during probabilistic learning. Neuroimage, 55, 266-276.
26. Dorris, M.C. & Glimcher, P.W. (2004) Activity in posterior parietal cortex is correlated with the relative subjective desirability of action. Neuron, 44, 365-378.
27. Doya, K. (1999) What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw., 12, 961-974.
28. Dusek, J.A. & Eichenbaum, H. (1997) The hippocampus and memory for orderly stimulus relations. Proc. Natl. Acad. Sci. USA, 94, 7109-7114.
29. Eichenbaum, H. & Cohen, N.J. (2001) From Conditioning to Conscious Recollection: Memory Systems of the Brain. Oxford University Press, New York, NY.
30. Everitt, B.J. & Robbins, T.W. (2005) Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat. Neurosci., 8, 1481-1489.
31. Falck, B. & Hillarp, N.A. (1959) On the cellular localization of catechol amines in the brain. Acta Anat., 38, 277-279.
32. Foerde, K., Knowlton, B.J. & Poldrack, R.A. (2006) Modulation of competing memory systems by distraction. Proc. Natl. Acad. Sci. USA, 103, 11778-11783.
33. Frank, M.J., Seeberger, L.C. & O'Reilly R, C. (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science, 306, 1940-1943.
34. Frey, U., Schroeder, H. & Matthies, H. (1990) Dopaminergic antagonists prevent long-term maintenance of posttetanic LTP in the CA1 region of rat hippocampal slices. Brain Res., 522, 69-75.
35. Friston, K.J., Worsley, K.J., Frackowiak, S.J., Mazziotta, J.C. & Evans, A.C. (1993) Assessing the significance of focal activations using their spatial extent. Hum. Brain Mapp., 1, 210-220.
36. Friston, K.J., Buechel, C., Fink, G.R., Morris, J., Rolls, E. & Dolan, R.J. (1997) Psychophysiological and modulatory interactions in neuroimaging. Neuroimage, 6, 218-229.
37. Friston, K.J., Josephs, O., Rees, G. & Turner, R. (1998) Nonlinear event-related responses in fMRI. Magn. Reson. Med., 39, 41-52.
38. Gabrieli, J.D. (1998) Cognitive neuroscience of human memory. Annu. Rev. Psychol., 49, 87-115.
39. Gasbarri, A., Verney, C., Innocenzi, R., Campana, E. & Pacitti, C. (1994) Mesolimbic dopaminergic neurons innervating the hippocampal formation in the rat: a combined retrograde tracing and immunohistochemical study. Brain Res., 668, 71-79.
40. Gershman, S.J., Pesaran, B. & Daw, N.D. (2009) Human reinforcement learning subdivides structured action spaces by learning effector-specific values. J. Neurosci., 29, 13524-13531.
41. Gluck, M.A. & Myers, C.E. (1993) Hippocampal mediation of stimulus representation: a computational theory. Hippocampus, 3, 491-516.
42. Greene, A.J., Gross, W.L., Elsinger, C.L. & Rao, S.M. (2006) An FMRI analysis of the human hippocampus: inference, context, and task awareness. J. Cogn. Neurosci., 18, 1156-1173.
43. Griffiths, T.L. & Tenenbaum, J.B. (2005) Structure and strength in causal induction. Cogn. Psychol., 51, 334-384.
44. Hampton, A.N., Bossaerts, P. & O'Doherty, J.P. (2006) The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci., 26, 8360-8367.
45. Hampton, A.N., Bossaerts, P. & O'Doherty, J.P. (2008) Neural correlates of mentalizing-related computations during strategic interactions in humans. Proc. Natl. Acad. Sci. USA, 105, 6741-6746.
46. Hannula, D.E. & Ranganath, C. (2009) The eyes have it: hippocampal activity predicts expression of memory in eye movements. Neuron, 63, 592-599.
47. Hare, T.A., O'Doherty, J., Camerer, C.F., Schultz, W. & Rangel, A. (2008) Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J. Neurosci., 28, 5623-5630.
48. Hartley, T. & Burgess, N. (2005) Complementary memory systems: competition, cooperation and compensation. Trends Neurosci., 28, 169-170.
49. Holmes, A.P. & Friston, K.J. (1998) Generalisability, random effects and population inference. Neuroimage, 7, S754.
50. Honey, R.C. & Hall, G. (1989) Acquired equivalence and distinctiveness of cues. J. Exp. Psychol. Anim. Behav. Process., 15, 338-346.
51. Houk, J.C., Adams, J.L. & Barto, A.G. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In Houk, J.C., Davis, J.L. & Beiser, D.G. (Eds), Models of Information Processing in the Basal Ganglia. MIT Press, Cambridge, MA, pp. 249-270.
52. Huang, Y.Y. & Kandel, E.R. (1995) D1/D5 receptor agonists induce a protein synthesis-dependent late potentiation in the CA1 region of the hippocampus. Proc. Natl. Acad. Sci. USA, 92, 2446-2450.
53. Johnson, A., van der Meer, M.A. & Redish, A.D. (2007) Integrating hippocampus and striatum in decision-making. Curr. Opin. Neurobiol., 17, 692-697.
54. Kass, R.E. & Raftery, A.E. (1995) Bayes factors. J. Am. Stat. Assoc., 90, 773-795.
55. Kelley, A.E. & Domesick, V.B. (1982) The distribution of the projection from the hippocampal formation to the nucleus accumbens in the rat: an anterograde- and retrograde-horseradish peroxidase study. Neuroscience, 7, 2321-2335.
56. Kemp, C. & Tenenbaum, J.B. (2008) The discovery of structural form. Proc. Natl. Acad. Sci. USA, 105, 10687-10692.
57. Kim, H., Shimojo, S. & O'Doherty J, P. (2006) Is avoiding an aversive outcome rewarding? neural substrates of avoidance learning in the human brain. PLoS Biol., 4, e233.
58. Knowlton, B.J., Mangels, J.A. & Squire, L.R. (1996) A neostriatal habit learning system in humans. Science, 273, 1399-1402.
59. Knutson, B. & Gibbs, S.E. (2007) Linking nucleus accumbens dopamine and blood oxygenation. Psychopharmacology (Berl), 191, 813-822.
60. Knutson, B., Adams, C.M., Fong, G.W. & Hommer, D. (2001) Anticipation of increasing monetary reward selectively recruits nucleus accumbens. J. Neurosci., 21, RC159.
61. Kuhl, B.A., Shah, A.T., DuBrow, S. & Wagner, A.D. (2010) Resistance to forgetting associated with hippocampus-mediated reactivation during new learning. Nat. Neurosci., 13, 501-506.
62. Kumaran, D., Summerfield, J.J., Hassabis, D. & Maguire, E.A. (2009) Tracking the emergence of conceptual knowledge during human decision making. Neuron, 63, 889-901.
63. Lau, B. & Glimcher, P.W. (2005) Dynamic response-by-response models of matching behavior in rhesus monkeys. J. Exp. Anal. Behav., 84, 555-579.
64. Lebreton, M., Jorge, S., Michel, V., Thirion, B. & Pessiglione, M. (2009) An automatic valuation system in the human brain: evidence from functional neuroimaging. Neuron, 64, 431-439.
65. Li, J. & Daw, N.D. (2011) Signals in human striatum are appropriate for policy update rather than value prediction. J. Neurosci., 31, 5504-5511.
66. Lohrenz, T., McCabe, K., Camerer, C.F. & Montague, P.R. (2007) Neural signature of fictive learning signals in a sequential investment task. Proc. Natl. Acad. Sci. USA, 104, 9493-9498.
67. Mattfeld, A.T. & Stark, C.E. (2011) Striatal and medial temporal lobe functional interactions during visuomotor associative learning. Cereb. Cortex, 21, 647-658.
68. McClure, S.M., Berns, G.S. & Montague, P.R. (2003) Temporal prediction errors in a passive learning task activate human striatum. Neuron, 38, 339-346.
69. McFadden, D. (1974) Conditional logit analysis of qualitative choice behavior. In Zarembka, P. (Ed.), Frontiers in Econometrics. Academic Press, New York, NY, pp. 105-142.
70. van der Meer, M.A., Johnson, A., Schmitzer-Torbert, N.C. & Redish, A.D. (2010) Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron, 67, 25-32.
71. Moustafa, A.A., Myers, C.E. & Gluck, M.A. (2009) A neurocomputational model of classical conditioning phenomena: a putative role for the hippocampal region in associative learning. Brain Res., 1276, 180-195.
72. Myers, C.E., Shohamy, D., Gluck, M.A., Grossman, S., Kluger, A., Ferris, S., Golomb, J., Schnirman, G. & Schwartz, R. (2003) Dissociating hippocampal versus basal ganglia contributions to learning and transfer. J. Cogn. Neurosci., 15, 185-193.
73. Nichols, T., Brett, M., Andersson, J., Wager, T. & Poline, J.B. (2005) Valid conjunction inference with the minimum statistic. Neuroimage, 25, 653-660.
74. O'Doherty, J.P., Dayan, P., Friston, K., Critchley, H. & Dolan, R.J. (2003) Temporal difference models and reward-related learning in the human brain. Neuron, 38, 329-337.
75. O'Doherty, J.P., Hampton, A. & Kim, H. (2007) Model-based fMRI and its application to reward learning and decision making. Ann. NY Acad. Sci., 1104, 35-53.
76. Otmakhova, N.A. & Lisman, J.E. (1996) D1/D5 dopamine receptor activation increases the magnitude of early long-term potentiation at CA1 hippocampal synapses. J. Neurosci., 16, 7478-7486.
77. Pagnoni, G., Zink, C.F., Montague, P.R. & Berns, G.S. (2002) Activity in human ventral striatum locked to errors of reward prediction. Nat. Neurosci., 5, 97-98.
78. Palminteri, S., Boraud, T., Lafargue, G., Dubois, B. & Pessiglione, M. (2009) Brain hemispheres selectively track the expected value of contralateral options. J. Neurosci., 29, 13465-13472.
79. Pessiglione, M., Seymour, B., Flandin, G., Dolan, R.J. & Frith, C.D. (2006) Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature, 442, 1042-1045.
80. Plassmann, H., O'Doherty, J. & Rangel, A. (2007) Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J. Neurosci., 27, 9984-9988.
81. Platt, M.L. & Glimcher, P.W. (1999) Neural correlates of decision variables in parietal cortex. Nature, 400, 233-238.
82. Poldrack, R.A., Clark, J., Pare-Blagoev, E.J., Shohamy, D., Creso Moyano, J., Myers, C. & Gluck, M.A. (2001) Interactive memory systems in the human brain. Nature, 414, 546-550.
83. Preston, A.R., Shrager, Y., Dudukovic, N.M. & Gabrieli, J.D. (2004) Hippocampal contribution to the novel use of relational information in declarative memory. Hippocampus, 14, 148-152.
84. Rangel, A., Camerer, C. & Montague, P.R. (2008) A framework for studying the neurobiology of value-based decision making. Nat. Rev. Neurosci., 9, 545-556.
85. Redish, A.D., Jensen, S. & Johnson, A. (2008) A unified framework for addiction: vulnerabilities in the decision process. Behav. Brain Sci., 31, 415-437; discussion 437-487.
86. Schacter, D.L. (1990) Perceptual representation systems and implicit memory. Toward a resolution of the multiple memory systems debate. Ann. NY Acad. Sci., 608, 543-567; discussion 567-571.
87. Schönberg, T., Daw, N.D., Joel, D. & O'Doherty, J.P. (2007) Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J. Neurosci., 27, 12860-12867.
88. Schönberg, T., O'Doherty, J.P., Joel, D., Inzelberg, R., Segev, Y. & Daw, N.D. (2010) Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson's disease patients: evidence from a model-based fMRI study. Neuroimage, 49, 772-781.
89. Schott, B.H., Minuzzi, L., Krebs, R.M., Elmenhorst, D., Lang, M., Winz, O.H., Seidenbecher, C.I., Coenen, H.H., Heinze, H.J., Zilles, K., Duzel, E. & Bauer, A. (2008) Mesolimbic functional magnetic resonance imaging activations during reward anticipation correlate with reward-related ventral striatal dopamine release. J. Neurosci., 28, 14311-14319.
90. Schultz, W. (2006) Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol., 57, 87-115.
91. Schultz, W., Dayan, P. & Montague, P.R. (1997) A neural substrate of prediction and reward. Science, 275, 1593-1599.
92. Shohamy, D. & Adcock, R.A. (2010) Dopamine and adaptive memory. Trends Cogn. Sci., 14, 464-472.
93. Shohamy, D. & Wagner, A.D. (2008) Integrating memories in the human brain: hippocampal-midbrain encoding of overlapping events. Neuron, 60, 378-389.
94. Shohamy, D., Myers, C.E., Geghman, K.D., Sage, J. & Gluck, M.A. (2006) L-dopa impairs learning, but spares generalization, in Parkinson's disease. Neuropsychologia, 44, 774-784.
95. Shohamy, D., Myers, C.E., Kalanithi, J. & Gluck, M.A. (2008) Basal ganglia and dopamine contributions to probabilistic category learning. Neurosci. Biobehav. Rev., 32, 219-236.
96. Simon, D.A. & Daw, N.D. (2011) Neural correlates of forward planning in a spatial decision task in humans. J. Neurosci., 31, 5526-5539.
97. Smith, D.V., Hayden, B.Y., Truong, T.K., Song, A.W., Platt, M.L. & Huettel, S.A. (2010) Distinct value signals in anterior and posterior ventromedial prefrontal cortex. J. Neurosci., 30, 2490-2495.
98. Squire, L.R. (1992) Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans. Psychol. Rev., 99, 195-231.
99. Staresina, B.P. & Davachi, L. (2009) Mind the gap: binding experiences across space and time in the human hippocampus. Neuron, 63, 267-276.
100. Stephan, K.E., Penny, W.D., Daunizeau, J., Moran, R.J. & Friston, K.J. (2009) Bayesian model selection for group studies. Neuroimage, 46, 1004-1017.
101. Sugrue, L.P., Corrado, G.S. & Newsome, W.T. (2004) Matching behavior and the representation of value in the parietal cortex. Science, 304, 1782-1787.
102. Sutton, R.S. & Barto, A.G. (1998) Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA.
103. Swanson, L.W. (1982) The projections of the ventral tegmental area and adjacent regions: a combined fluorescent retrograde tracer and immunofluorescence study in the rat. Brain Res. Bull., 9, 321-353.
104. Thorndike, E.L. (1911) Animal Intelligence. Hafner, Darien, CT.
105. Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., Mazoyer, B. & Joliot, M. (2002) Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage, 15, 273-289.
106. Wittmann, B.C., Daw, N.D., Seymour, B. & Dolan, R.J. (2008) Striatal activity underlies novelty-based choice in humans. Neuron, 58, 967-973.
107. Zeithamova, D. & Preston, A.R. (2010) Flexible memories: differential roles for medial temporal lobe and prefrontal cortex in cross-episode binding. J. Neurosci., 30, 14676-14684.