Feedback timing modulates brain systems for learning in humans

Journal of Neuroscience

Volumn 31, Issue 37, 2011, Pages 13157-13167

  Foerde, Karin ^a   Shohamy, Daphna ^a  

^a Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; BRAIN FUNCTION; BRAIN REGION; CLINICAL ARTICLE; CONTROLLED STUDY; CORPUS STRIATUM; DOPAMINERGIC NERVE CELL; EPISODIC MEMORY; FEEDBACK TIMING; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HIPPOCAMPUS; HUMAN; IMAGE ANALYSIS; LEARNING; LEARNING DISORDER; MALE; MATHEMATICAL MODEL; MESENCEPHALON; NEUROIMAGING; NEUROMODULATION; PARKINSON DISEASE; PRIORITY JOURNAL; REINFORCEMENT; RESPONSE TIME; VISUAL FEEDBACK; VISUAL MEMORY;

AGED; BRAIN MAPPING; CORPUS STRIATUM; FEEDBACK, PSYCHOLOGICAL; HIPPOCAMPUS; HUMANS; LEARNING; MAGNETIC RESONANCE IMAGING; MEMORY, EPISODIC; MIDDLE AGED; MODELS, NEUROLOGICAL; PARKINSON DISEASE; PSYCHOMOTOR PERFORMANCE; TIME FACTORS; YOUNG ADULT;

EID: 80052942848     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.2701-11.2011     Document Type: Article

References (62)

1. Adcock RA, Thangavel A, Whitfield-Gabrieli S, Knutson B, Gabrieli JD (2006) Reward-motivated learning: mesolimbic activation precedes memory formation. Neuron 50:507-517.
2. Agid Y, Cervera P, Hirsch E, Javoy-Agid F, Lehericy S, Raisman R, Ruberg M (1989) Biochemistry of Parkinson's disease 28 years later: a critical review. Mov Disord 4 [Suppl 1]:S126-S144.
3. Cheng DT, Disterhoft JF, Power JM, Ellis DA, Desmond JE (2008) Neural substrates underlying human delay and trace eyeblink conditioning. Proc Natl Acad Sci U S A 105:8108-8113.
4. Cheung TH, Cardinal RN (2005) Hippocampal lesions facilitate instrumental learning with delayed reinforcement but induce impulsive choice in rats. BMC Neurosci 6:36.
5. Clark RE, Squire LR (1998) Classical conditioning and brain systems: the role of awareness. Science 280:77-81.
6. Cohen NJ, Eichenbaum H (1993) Memory, amnesia, and the hippocampal system. Cambridge, MA: MIT.
7. D'Ardenne K, McClure SM, Nystrom LE, Cohen JD (2008) BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science 319:1264-1267.
8. Davachi L (2006) Item, context and relational episodic encoding in humans. Curr Opin Neurobiol 16:693-700.
9. Daw ND (2011) Trial-by-trial data analysis using computational models. In: Affect, learning and decision making, attention and performance (pp. 3-38) (Phelps EA, Robbins TW, Delgado M, eds). New York: Oxford UP.
10. Daw ND, Doya K (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16:199-204.
11. Dickerson KC, Li J, Delgado MR (2011) Parallel contributions of distinct human memory systems during probabilistic learning. Neuroimage 55:266-276.
12. Dickinson A, Watt A, Griffiths WJ (1992) Free-operant acquisition with delayed reinforcement. Q J Exp Psychol B 45B:241-258.
13. Dickinson A, Watt A, Varga ZI (1996) Context conditioning and free- operant acquisition under delayed reinforcement. Q J Exp Psychol B 49B:97-110.
14. Doll BB, Jacobs WJ, Sanfey AG, Frank MJ (2009) Instructional control of reinforcement learning: a behavioral and neurocomputational investigation. Brain Res 1299:74-94.
15. Duncan K, Curtis C, Davachi L (2009) Distinct memory signatures in the hippocampus: intentional states distinguish match and mismatch enhancement signals. J Neurosci 29:131-139.
16. Düzel E, Bunzeck N, Guitart-Masip M, Wittmann B, Schott BH, Tobler PN (2009) Functional imaging of the human dopaminergic midbrain. Trends Neurosci 32:321-328.
17. Epstein R, Kanwisher N (1998) A cortical representation of the local visual environment. Nature 392:598-601.
18. Figner B, Knoch D, Johnson EJ, Krosch AR, Lisanby SH, Fehr E, Weber EU (2010) Lateral prefrontal cortex and self-control in inter temporal choice. Nat Neurosci 13:538-539.
19. Fiorillo CD, Newsome WT, Schultz W (2008) The temporal precision of reward prediction in dopamine neurons. Nat Neurosci 11:966-973.
20. Foerde K, Knowlton BJ, Poldrack RA (2006) Modulation of competing memory systems by distraction. Proc Natl Acad Sci U S A 103:11778-11783.
21. Frank MJ, Seeberger LC, O'Reilly RC (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306:1940-1943.
22. Gläscher J, Daw N, Dayan P, O'Doherty JP (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66:585-595.
23. Glover GH, Law CS (2001) Spiral-in/out BOLD fMRI for increased SNR and reduced susceptibility artifacts. Magn Reson Med 46:515-522.
24. Gregorios-PippasL, Tobler PN, SchultzW (2009) Short-term temporal discounting of reward value in human ventral striatum. J Neurophysiol 101:1507-1523.
25. Grice GR (1948) The relationof secondary reinforcement to delayed reward in visual discrimination learning. J Exp Psychol 38:1-16.
26. Johnson A, Redish AD (2007) Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J Neurosci 27:12176-12189.
27. Kable JW, Glimcher PW (2007) The neural correlates of subjective value during intertemporal choice. Nat Neurosci 10:1625-1633.
28. Kable JW, Glimcher PW (2009) The neurobiology of decision: consensus and controversy. Neuron 63:733-745.
29. Knowlton BJ, Mangels JA, Squire LR (1996) A neostriatal habit learning system in humans. Science 273:1399-1402.
30. Kobayashi S, Schultz W (2008) Influence of reward delays on responses of dopamine neurons. J Neurosci 28:7837-7846.
31. Kumaran D, Maguire EA (2006) An unexpected sequence of events: mismatch detection in the human hippocampus. PLoS Biol 4:e424.
32. Kumaran D, Maguire EA (2007) Match mismatch processes underlie human hippocampal responses to associative novelty. J Neurosci 27:8517-8524.
33. Lattal KA, Gleeson S (1990) Response acquisition with delayed reinforcement. J Exp Psychol Anim Behav Process 16:27-39.
34. Lisman JE, Grace AA (2005) The hippocampal-VTA loop: controlling the entry of information into long-term memory. Neuron 46:703-713.
35. LiuZ,RichmondBJ (2000) Response differencesinmonkeyTEand perirhi-nal cortex: stimulus association related to reward schedules. J Neuro-physiol 83:1677-1692.
36. Luo S, Ainslie G, Giragosian L, Monterosso JR (2009) Behavioral and neural evidence of incentive bias for immediate rewards relative to preference-matched delayed rewards. J Neurosci 29:14820-14827.
37. Maddox WT, Ing AD (2005) Delayed feedback disrupts the procedural-learning system but not the hypothesis-testing system in perceptual category learning. J Exp Psychol Learn Mem Cogn 31:100-107.
38. Maddox WT, Ashby FG, Bohil CJ (2003) Delayed feedback effects on rule-based and information-integration category learning. JExp Psychol Learn Mem Cogn 29:650-662.
39. McClure SM, Laibson DI, Loewenstein G, Cohen JD (2004) Separate neural systems value immediate and delayed monetary rewards. Science 306:503-507.
40. O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452-454.
41. Otmakhova NA, Lisman JE (1998) D1/D5 dopamine receptors inhibit depotentiation at CA1 synapses via cAMP-dependent mechanism. J Neurosci 18:1270-1279.
42. Perin CT (1943) The effect of delayed reinforcement upon the differentiation of bar responses in white rats. J Exp Psychol 32:95-109.
43. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD (2006) Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442:1042-1045.
44. Peters J, Büchel C (2010) Episodic future thinking reduces reward delay discounting through an enhancement of prefrontal-mediotemporal interactions. Neuron 66:138-148.
45. Pine A, Shiner T, Seymour B, Dolan RJ (2010) Dopamine, time, and impulsivity in humans. J Neurosci 30:8888-8896.
46. Poldrack RA, Clark J, Paré-Blagoev EJ, Shohamy D, Creso Moyano J, Myers C, Gluck MA (2001) Interactive memory systems in the human brain. Nature 414:546-550.
47. Port R, Curtis K, Inoue C, Briggs J, Seybold K (1993) Hippocampal damage does not impair instrumental appetitive conditioning with delayed reinforcement. Brain Res Bull 30:41-44.
48. Roesch MR, Calu DJ, Schoenbaum G (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci 10:1615-1624.
49. Schönberg T, Daw ND, Joel D, O'Doherty JP (2007) Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J Neurosci 27:12860-12867.
50. Schultz W (1998) Predictive reward signal of dopamine neurons. J Neuro-physiol 80:1-27.
51. Sherry DF, Schacter DL (1987) The evolution of multiple memory systems. Psychol Rev 94:439-454.
52. Shohamy D, Adcock RA (2010) Dopamine and adaptive memory. Trends Cogn Sci 14:464-472.
53. Shohamy D, Wagner AD (2008) Integrating memories in the human brain: hippocampal-midbrain encoding of overlapping events. Neuron 60: 378-389.
54. Shohamy D, Myers CE, Grossman S, Sage J, Gluck MA, Poldrack RA (2004) Cortico-striatal contributions to feedback-based learning: converging data from neuroimaging and neuropsychology. Brain 127:851-859.
55. Squire LR (2004) Memory systems of the brain: a brief history and current perspective. Neurobiol Learn Mem 82:171-177.
56. Staresina BP, Davachi L (2009) Mind the gap: binding experiences across space and time in the human hippocampus. Neuron 63:267-276.
57. Sutton R, Barto AG (1998) Reinforcement learning. Cambridge, MA: MIT.
58. Thompson RF, Kim JJ (1996) Memory systems in the brain and localization of a memory. Proc Natl Acad Sci U S A 93:13438-13444.
59. Watanabe T, Niki H (1985) Hippocampal unit activity and delayed response in the monkey. Brain Res 325:241-254.
60. White NM, McDonald RJ (2002) Multiple parallel memory systems in the brain of the rat. Neurobiol Learn Mem 77:125-184.
61. Wirth S, Avsar E, Chiu CC, Sharma V, Smith AC, Brown E, Suzuki WA (2009) Trial outcome and associative learning signals in the monkey hippocampus. Neuron 61:930-940.
62. Wittmann BC, Schott BH, Guderian S, Frey JU, Heinze HJ, Düzel E (2005) Reward-related FMRI activation of dopaminergic midbrain is associated with enhanced hippocampus-dependent long-term memory formation. Neuron 45:459-467.