Episodic memory encoding interferes with reward learning and decreases striatal prediction errors

Journal of Neuroscience

Volumn 34, Issue 45, 2014, Pages 14901-14912

  Elliott Wimmer, G ^a,b   Braun, Erin Kendall ^a   Daw, Nathaniel D ^c   Shohamy, Daphna ^a  

^a Och Spine at New York Presbyterian Hospitals   (United States)

^b UNIVERSITY MEDICAL CENTER HAMBURG EPPENDORF   (Germany)

^c NEW YORK UNIVERSITY   (United States)

Author keywords

Decision making; Hippocampus; Learning; Memory; Reward; Striatum

Indexed keywords

ADULT; ARTICLE; BEHAVIORAL SCIENCE; CORPUS STRIATUM; DECISION MAKING; DOPAMINERGIC NERVE CELL; EPISODIC MEMORY; FEEDBACK SYSTEM; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HIPPOCAMPUS; HUMAN; HUMAN EXPERIMENT; LEARNING; LEARNING AND MEMORY TEST; MALE; MEMORY TEST; QUESTIONNAIRE; REWARD; SYSTEMATIC ERROR; YOUNG ADULT; ADOLESCENT; PHYSIOLOGY;

ADOLESCENT; ADULT; ASSOCIATION LEARNING; CHOICE BEHAVIOR; CORPUS STRIATUM; FEMALE; HIPPOCAMPUS; HUMANS; MALE; MEMORY, EPISODIC; REWARD;

EID: 84908518146     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.0204-14.2014     Document Type: Article

References (86)

1. Adcock RA, Thangavel A, Whitfield-Gabrieli S, Knutson B, Gabrieli JD (2006) Reward-motivated learning: mesolimbic activation precedes memory formation. Neuron 50:507-517. CrossRef Medline
2. Barron HC, Dolan RJ, Behrens TE (2013) Online evaluation of novel choices by simultaneous representation of multiple memories. Nat Neurosci 16:1492-1498. CrossRef Medline
3. Barto AG (1995) Adaptive critics and the basal ganglia. In: Models of information processing in the basal ganglia (Davis JL, Houk JC, Beiser DG, eds), pp 215-232. Cambridge, MA: Massachusetts Institute of Technology.
4. Beckmann CF, Smith SM (2004) Probabilistic independent component analysis for functional magnetic resonance imaging. IEEE Trans Med Imaging 23:137-152. CrossRef Medline
5. Behrens TE, Woolrich MW, Walton ME, Rushworth MF (2007) Learning the value of information in an uncertain world. Nat Neurosci 10:1214-1221. CrossRef Medline
6. Behrens TE, Hunt LT, Woolrich MW, Rushworth MF (2008) Associative learning of social value. Nature 456:245-249. CrossRef Medline
7. Bialleck KA, Schaal HP, Kranz TA, Fell J, Elger CE, Axmacher N (2011) Ventromedial prefrontal cortex activation is associated with memory formation for predictable rewards. PLoS One 6:e16695. CrossRef Medline
8. Biele G, Erev I, Ert E (2009) Learning, risk attitudes and hot stoves in restless bandit problems. J Math Psychol 53:155-167.CrossRef
9. Boorman ED, Behrens TE, Woolrich MW, Rushworth MF (2009) How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 62:733-743. CrossRef Medline
10. Bornstein AM, Daw ND (2012) Dissociating hippocampal and striatal contributions to sequential prediction learning. Eur J Neurosci 35:1011-1023. CrossRef Medline
11. Box GEP, Cox DR (1964) An analysis of transformations. J R Stat Soc Ser B 26:211-252.
12. Brainard DH (1997) The Psychophysics Toolbox. Spat Vis 10:433-436. CrossRef Medline
13. Brewer JB, Zhao Z, Desmond JE, Glover GH, Gabrieli JD (1998) Making memories: brain activity that predicts how well visual experience will be remembered. Science 281:1185-1187. CrossRef Medline
14. Buckner RL (2010) The role of the hippocampus in prediction and imagination. Annu Rev Psychol 61:27-48, C1-C8. CrossRef Medline
15. Burgess N (2006) Spatial memory: how egocentric and allocentric combine. Trends Cogn Sci 10:551-557. CrossRef Medline
16. Collins AG, Frank MJ (2012) How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. Eur J Neurosci 35:1024-1035. CrossRef Medline
17. Cox RW (1996) AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res 29:162-173. CrossRef Medline
18. Daw ND (2011) Trial-by-trial data analysis using computational models In: Attention and performance, Vol XXIII (Delgado MR, Phelps EA, Robbins TW, eds), pp 3-38. Oxford: Oxford UP.
19. Daw ND, Doya K (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16:199-204. CrossRef Medline
20. Daw ND, Shohamy D (2008) The cognitive neuroscience of motivation and learning. Soc Cogn 26:593-620.CrossRef
21. Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8:1704-1711. CrossRef Medline
22. Daw ND, O’Doherty JP, Dayan P, Seymour B, Dolan RJ (2006) Cortical substrates for exploratory decisions in humans. Nature 441:876-879. CrossRef Medline
23. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011) Model-based influences on humans’ choices and striatal prediction errors. Neuron 69:1204-1215. CrossRef Medline
24. Dayan P, Kakade S, Montague PR (2000) Learning and selective attention. Nat Neurosci 3 [Suppl]:1218-1223.
25. Delgado MR, Dickerson KC (2012) Reward-related learning via multiple memory systems. Biol Psychiatry 72:134-141. CrossRef Medline
26. Delgado MR, Nystrom LE, Fissell C, Noll DC, Fiez JA (2000) Tracking the hemodynamic responses to reward and punishment in the striatum. J Neurophysiol 84:3072-3077.Medline
27. Dickerson KC, Li J, Delgado MR (2011) Parallel contributions of distinct human memory systems during probabilistic learning. Neuroimage 55: 266-276. CrossRef Medline
28. Dickinson A (1985) Actions and habits: the development of behavioural autonomy. Philos Trans R Soc Lond B 308:67-78.CrossRef
29. Doya K (1999) What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw 12:961-974. CrossRef Medline
30. Eichenbaum H, Cohen NJ (2001) From conditioning to conscious recollection: memory systems of the brain. New York: Oxford UP.
31. Frank MJ, Seeberger LC, O’Reilly RC (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306:1940-1943. CrossRef Medline
32. Friston KJ, Worsley KJ, Frackowiak RS, Mazziotta JC, Evans AC (1994) Assessing the significance of focal activations using their spatial extent.Hum Brain Mapp 1:210-220. CrossRef Medline
33. Friston KJ, Buechel C, Fink GR, Morris J, Rolls E, Dolan RJ (1997) Psychophysiological and modulatory interactions in neuroimaging. Neuroimage 6:218-229. CrossRef Medline
34. Gershman SJ, Pesaran B, Daw ND (2009) Human reinforcement learning subdivides structured action spaces by learning effector-specific values. J Neurosci 29:13524-13531. CrossRef Medline
35. Gläscher J, Daw N, Dayan P, O’Doherty JP (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66:585-595. CrossRef Medline
36. Grill-Spector K, Kourtzi Z, Kanwisher N (2001) The lateral occipital complex and its role in object recognition. Vision Res 41:1409-1422. CrossRef Medline
37. Johnson A, Redish AD (2007) Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J Neurosci 27:12176-12189. CrossRef Medline
38. Kahn I, Shohamy D (2013) Intrinsic connectivity between the hippocampus, nucleus accumbens, and ventral tegmental area in humans. Hippocampus 23:187-192. CrossRef Medline
39. Knowlton BJ, Mangels JA, Squire LR (1996) A neostriatal habit learning system in humans. Science 273:1399-1402. CrossRef Medline
40. Knutson B, Westdorp A, Kaiser E, Hommer D (2000) fMRI visualization of brain activity during a monetary incentive delay task. Neuroimage 12:20-27. CrossRef Medline
41. Kruschke JK (2001) Toward a unified model of attention in associative learning. J Math Psychol 45:812-863.CrossRef
42. Lau B, Glimcher PW (2005) Dynamic response-by-response models of matching behavior in rhesus monkeys. J Exp Anal Behav 84:555-579. CrossRef Medline
43. Lee AS, Duman RS, Pittenger C (2008) A double dissociation revealing bidirectional competition between striatum and hippocampus during learning. Proc Natl Acad Sci U S A 105:17163-17168. CrossRef Medline
44. Lengyel M, Dayan P (2005) Hippocampal contributions to control: the third way. In: Advances in neural information processing systems, Vol 20 (Platt J, Koller D, Singer Y, Roweis S, eds), pp 889-896. Cambridge, MA: Massachusetts Institute of Technology.
45. Li J, Daw ND (2011) Signals in human striatum are appropriate for policy update rather than value prediction. J Neurosci 31:5504-5511. CrossRef Medline
46. Li J, Schiller D, Schoenbaum G, Phelps EA, Daw ND (2011) Differential roles of human striatum and amygdala in associative learning. Nat Neurosci 14:1250-1252. CrossRef Medline
47. Lisman JE, Grace AA (2005) The hippocampal-VTA loop: controlling the entry of information into long-term memory. Neuron 46:703-713. CrossRef Medline
48. Luce RD (1986) Response times: Their role in inferring elementary mental organization. New York: Oxford UP.
49. Mackintosh NJ (1975) A theory of attention: variations in the associability of stimuli with reinforcement. Psychol Rev 82:276-298.CrossRef
50. Mather M, Schoeke A (2011) Positive outcomes enhance incidental learning for both younger and older adults. Front Neurosci 5:129. CrossRef Medline
51. McClure SM, Berns GS, Montague PR (2003) Temporal prediction errors in a passive learning task activate human striatum. Neuron 38:339-346. CrossRef Medline
52. Niv Y, Edlund JA, Dayan P, O’Doherty JP (2012) Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain. J Neurosci 32:551-562. CrossRef Medline
53. O’Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ (2003) Temporal difference models and reward-related learning in the human brain. Neuron 38:329-337. CrossRef Medline
54. O’Doherty JP, Hampton A, Kim H (2007) Model-based fMRI and its application to reward learning and decision making. Ann N Y Acad Sci 1104: 35-53. CrossRef Medline
55. Otten LJ, Henson RN, Rugg MD (2001) Depth of processing effects on neural correlates of memory encoding: relationship between findings from across-and within-task comparisons. Brain 124:399-412. CrossRef Medline
56. Packard MG, McGaugh JL (1996) Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiol Learn Mem 65:65-72. CrossRef Medline
57. Paller KA, Wagner AD (2002) Observing the transformation of experience into memory. Trends Cogn Sci 6:93-102. CrossRef Medline
58. Paller KA, Kutas M, Mayes AR (1987) Neural correlates of encoding in an incidental learning paradigm. Electroencephalogr Clin Neurophysiol 67: 360-371. CrossRef Medline
59. Parks TE (1966) Signal-detectability theory of recognition-memory performance. Psychol Rev 73:44-58. CrossRef Medline
60. Pearce JM, Hall G (1980) A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol Rev 87:532-552. CrossRef Medline
61. Pennartz CM, Ito R, Verschure PF, Battaglia FP, Robbins TW (2011) The hippocampal-striatal axis in learning, prediction and goal-directed behavior. Trends Neurosci 34:548-559. CrossRef Medline
62. Peters J, Büchel C (2010) Episodic future thinking reduces reward delay discounting through an enhancement of prefrontal-mediotemporal interactions. Neuron 66:138-148. CrossRef Medline
63. Pfeiffer BE, Foster DJ (2013) Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497:74-79. CrossRef Medline
64. Platt ML, Glimcher PW (1999) Neural correlates of decision variables in parietal cortex. Nature 400:233-238. CrossRef Medline
65. Poldrack RA, Packard MG (2003) Competition among multiple memory systems: converging evidence from animal and human brain studies. Neuropsychologia 41:245-251. CrossRef Medline
66. Poldrack RA, Clark J, Paré-Blagoev EJ, Shohamy D, Creso Moyano J, Myers C, Gluck MA (2001) Interactive memory systems in the human brain. Nature 414:546-550. CrossRef Medline
67. Rangel A, Camerer C, Montague PR (2008) A framework for studying the neurobiology of value-based decision making. Nat Rev Neurosci 9:545-556. CrossRef Medline
68. Robbins TW (1998) Arousal and attention: psychopharmacological and neuropsychological studies in experimental animals. In: The attentive brain (Parasuraman R, ed), pp 189-220. Cambridge, MA: Massachusetts Institute of Technology.
69. Sadeh T, Shohamy D, Levy DR, Reggev N, Maril A (2011) Cooperation between the hippocampus and the striatum during episodic encoding. J Cogn Neurosci 23:1597-1608. CrossRef Medline
70. Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of actionspecific reward values in the striatum. Science 310:1337-1340. CrossRef Medline
71. Sanquist TF, Rohrbaugh JW, Syndulko K, Lindsley DB (1980) Electrocortical signs of levels of processing: perceptual analysis and recognition memory. Psychophysiology 17:568-576. CrossRef Medline
72. Schönberg T, Daw ND, Joel D, O’Doherty JP (2007) Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J Neurosci 27:12860-12867. CrossRef Medline
73. Schultz W (2006) Behavioral theories and the neurophysiology of reward. Annu Rev Psychol 57:87-115. CrossRef Medline
74. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593-1599. CrossRef Medline
75. Scimeca JM, Badre D (2012) Striatal contributions to declarative memory retrieval. Neuron 75:380-392. CrossRef Medline
76. Shohamy D, Adcock RA (2010) Dopamine and adaptive memory. Trends Cogn Sci 14:464-472. CrossRef Medline
77. Shohamy D, Turk-Browne NB (2013) Mechanisms for widespread hippocampal involvement in cognition. J Exp Psychol Gen 142:1159-1170. CrossRef Medline
78. Squire LR, Zola SM (1996) Structure and function of declarative and nondeclarative memory systems. Proc Natl Acad Sci U S A 93:13515-13522. CrossRef Medline
79. Staresina BP, Davachi L (2009) Mind the gap: binding experiences across space and time in the human hippocampus. Neuron 63:267-276. CrossRef Medline
80. Steinberg EE, Keiflin R, Boivin JR, Witten IB, Deisseroth K, Janak PH (2013) A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci 16:966-973. CrossRef Medline
81. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. Cambridge, MA: Massachusetts Institute of Technology.
82. Tolman EC (1948) Cognitive maps in rats and men. Psychol Rev 55:189-208. CrossRef Medline
83. Wagner AD, Schacter DL, Rotte M, Koutstaal W, Maril A, Dale AM, Rosen BR, Buckner RL (1998) Building memories: remembering and forgetting of verbal experiences as predicted by brain activity. Science 281: 1188-1191. CrossRef Medline
84. Wimmer GE, Shohamy D (2012) Preference by association: how memory mechanisms in the hippocampus bias decisions. Science 338:270-273. CrossRef Medline
85. Wimmer GE, Daw ND, Shohamy D (2012) Generalization of value in reinforcement learning by humans. Eur J Neurosci 35:1092-1104. CrossRef Medline
86. Wittmann BC, Schott BH, Guderian S, Frey JU, Heinze HJ, Düzel E (2005) Reward-related fMRI activation of dopaminergic midbrain is associated with enhanced hippocampus-dependent long-term memory formation. Neuron 45:459-467. CrossRef Medline