Changes in corticostriatal connectivity during reinforcement learning in humans

Human Brain Mapping

Volumn 36, Issue 2, 2015, Pages 793-803

  Horga, Guillermo ^a   Maia, Tiago V ^a,b   Marsh, Rachel ^a   Hao, Xuejun ^a   Xu, Dongrong ^a   Duan, Yunsuo ^a   Tau, Gregory Z ^a   Graniello, Barbara ^a   Wang, Zhishun ^a   Kangarlu, Alayar ^a   Martinez, Diana ^a   Packard, Mark G ^c   Peterson, Bradley S ^d  

^a NEW YORK STATE PSYCHIATRIC INSTITUTE   (United States)

^b INSTITUTO DE MEDICINA MOLECULAR   (Portugal)

^c TEXAS A AND M UNIVERSITY   (United States)

^d UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

Computational model based fMRI; Functional connectivity; Putamen; Reinforcement learning

Indexed keywords

ADOLESCENT; ADULT; ALGORITHM; ARTICLE; BOLD SIGNAL; BRAIN FUNCTION; CORPUS STRIATUM; CORTICOSTRIATAL CONNECTIVITY; DECISION MAKING; FEMALE; FUNCTIONAL ANATOMY; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; IMAGE ANALYSIS; INTELLIGENCE QUOTIENT; MALE; NORMAL HUMAN; NUCLEAR MAGNETIC RESONANCE SCANNER; PREMOTOR CORTEX; PRIORITY JOURNAL; PUTAMEN; REINFORCEMENT; REWARD; SENSORIMOTOR CORTEX; SUPPLEMENTARY MOTOR AREA; TASK PERFORMANCE; THALAMUS; THALAMUS LATERAL NUCLEUS; VENTROMEDIAL PREFRONTAL CORTEX; BIOLOGICAL MODEL; BRAIN CIRCULATION; BRAIN MAPPING; COMPUTER INTERFACE; HEMODYNAMICS; MAZE TEST; NERVE TRACT; NEUROPSYCHOLOGICAL TEST; NUCLEAR MAGNETIC RESONANCE IMAGING; PHYSIOLOGY; PSYCHOPHYSICS; VASCULARIZATION;

ADULT; ALGORITHMS; BRAIN MAPPING; CEREBROVASCULAR CIRCULATION; CHOICE BEHAVIOR; FEMALE; HEMODYNAMICS; HUMANS; MAGNETIC RESONANCE IMAGING; MALE; MAZE LEARNING; MODELS, NEUROLOGICAL; NEURAL PATHWAYS; NEUROPSYCHOLOGICAL TESTS; PSYCHOPHYSICS; PUTAMEN; REINFORCEMENT (PSYCHOLOGY); SENSORIMOTOR CORTEX; USER-COMPUTER INTERFACE;

EID: 84922261945     PISSN: 10659471     EISSN: 10970193     Source Type: Journal    
DOI: 10.1002/hbm.22665     Document Type: Article

References (49)

1. Balleine BW, O'Doherty JP (2010): Human and rodent homologies in action control: Corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 35:48-69.
2. Barto AG (1995): Adaptive critics and the basal ganglia. In: Houk JC, Davis JL, Beiser DG, editors. Models of Information Processing in the Basal Ganglia. Cambridge, MA: MIT Press.
3. Belin D, Jonkman S, Dickinson A, Robbins TW, Everitt BJ (2009): Parallel and interactive learning processes within the basal ganglia: Relevance for the understanding of addiction. Behav Brain Res 199:89-102.
4. Centonze D, Picconi B, Gubellini P, Bernardi G, Calabresi P (2001): Dopaminergic control of synaptic plasticity in the dorsal striatum. Eur J Neurosci 13:1071-1077.
5. Charpier S, Deniau JM (1997): In vivo activity-dependent plasticity at cortico-striatal connections: Evidence for physiological long-term potentiation. Proc Natl Acad Sci USA 94:7036-7040.
6. Daw ND, Niv Y, Dayan P (2005): Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8:1704-1711.
7. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011): Model-based influences on humans' choices and striatal prediction errors. Neuron 69:1204-1215.
8. de Wit S, Watson P, Harsay HA, Cohen MX, van de Vijver I, Ridderinkhof KR (2012): Corticostriatal connectivity underlies individual differences in the balance between habitual and goal-directed action control. J Neurosci 32:12066-12075.
9. den Ouden HE, Friston KJ, Daw ND, McIntosh AR, Stephan KE (2009): A dual role for prediction error in associative learning. Cereb Cortex 19:1175-1185.
10. den Ouden HE, Daunizeau J, Roiser J, Friston KJ, Stephan KE (2010): Striatal prediction error modulates cortical coupling. J Neurosci 30:3210-3219.
11. Draganski B, Kherif F, Kloppel S, Cook PA, Alexander DC, Parker GJ, Deichmann R, Ashburner J, Frackowiak RS (2008): Evidence for segregated and integrative connectivity patterns in the human Basal Ganglia. J Neurosci 28:7143-7152.
12. Frank MJ (2005): Dynamic dopamine modulation in the basal ganglia: A neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J Cogn Neurosci 17:51-72.
13. Frank MJ, Moustafa AA, Haughey HM, Curran T, Hutchison KE (2007): Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc Natl Acad Sci USA 104:16311-16316.
14. Friston KJ, Buechel C, Fink GR, Morris J, Rolls E, Dolan RJ (1997): Psychophysiological and modulatory interactions in neuroimaging. NeuroImage 6:218-229.
15. Friston KJ, Josephs O, Rees G, Turner R (1998): Nonlinear event-related responses in fMRI. Magn Reson Med 39:41-52.
16. Glimcher PW (2011): Understanding dopamine and reinforcement learning: The dopamine reward prediction error hypothesis. Proc Natl Acad Sci USA 108(Suppl 3):15647-15654.
17. Guitart-Masip M, Fuentemilla L, Bach DR, Huys QJ, Dayan P, Dolan RJ, Duzel E (2011): Action dominates valence in anticipatory representations in the human striatum and dopaminergic midbrain. J Neurosci 31:7867-7875.
18. Haber SN, Fudge JL, McFarland NR (2000): Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J Neurosci 20:2369-2382.
19. Klein TA, Neumann J, Reuter M, Hennig J, von Cramon DY, Ullsperger M (2007): Genetically determined differences in learning from errors. Science 318:1642-1645.
20. Knowlton BJ, Mangels JA, Squire LR (1996): A neostriatal habit learning system in humans. Science 273:1399-1402.
21. Lehericy S, Ducros M, Van de Moortele PF, Francois C, Thivard L, Poupon C, Swindale N, Ugurbil K, Kim DS (2004): Diffusion tensor fiber tracking shows distinct corticostriatal circuits in humans. Ann Neurol 55:522-529.
22. Maia TV (2009): Reinforcement learning, conditioning, and the brain: Successes and challenges. Cogn Affect Behav Neurosci 9:343-364.
23. Maia TV, Frank MJ (2011): From reinforcement learning models to psychiatric and neurological disorders. Nat Neurosci 14:154-162.
24. Maldjian JA, Laurienti PJ, Kraft RA, Burdette JH (2003): An automated method for neuroanatomic and cytoarchitectonic atlas-based interrogation of fMRI data sets. NeuroImage 19:1233-1239.
25. Marsh R, Hao X, Xu D, Wang Z, Duan Y, Liu J, Kangarlu A, Martinez D, Garcia F, Tau GZ, Yu S, Packard MG, Peterson BS (2010): A virtual reality-based FMRI study of reward-based spatial learning. Neuropsychologia 48:2912-2921.
26. McDonald RJ, Hong NS (2004): A dissociation of dorso-lateral striatum and amygdala function on the same stimulus-response habit task. Neuroscience 124:507-513.
27. McDonald RJ, White NM (1993): A triple dissociation of memory systems: Hippocampus, amygdala, and dorsal striatum. Behav Neurosci 107:3-22.
28. Neumann J, Lohmann G (2003): Bayesian second-level analysis of functional magnetic resonance images. NeuroImage 20:1346-1355.
29. O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ (2004): Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452-454.
30. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ (2003): Temporal difference models and reward-related learning in the human brain. Neuron 38:329-337.
31. O'Doherty JP, Hampton A, Kim H (2007): Model-based fMRI and its application to reward learning and decision making. Ann N Y Acad Sci 1104:35-53.
32. Packard MG (1999): Glutamate infused posttraining into the hippocampus or caudate-putamen differentially strengthens place and response learning. Proc Natl Acad Sci USA 96:12881-12886.
33. Packard MG, Knowlton BJ (2002): Learning and memory functions of the Basal Ganglia. Annu Rev Neurosci 25:563-593.
34. Packard MG, Hirsh R, White NM (1989): Differential effects of fornix and caudate nucleus lesions on two radial maze tasks: Evidence for multiple memory systems. J Neurosci 9:1465-1472.
35. Paik SB, Glaser DA (2010): Synaptic plasticity controls sensory responses through frequency-dependent gamma oscillation resonance. PLoS Comput Biol 6:e1000927.
36. Pawlak V, Kerr JN (2008): Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J Neurosci 28:2435-2446.
37. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD (2006): Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442:1042-1045.
38. Postuma RB, Dagher A (2006): Basal ganglia functional connectivity based on a meta-analysis of 126 positron emission tomography and functional magnetic resonance imaging publications. Cereb Cortex 16:1508-1521.
39. Sage JR, Knowlton BJ (2000): Effects of US devaluation on win-stay and win-shift radial maze performance in rats. Behav Neurosci 114:295-306.
40. Schonberg T, Daw ND, Joel D, O'Doherty JP (2007): Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J Neurosci 27:12860-12867.
41. Schultz W, Dayan P, Montague PR (1997): A neural substrate of prediction and reward. Science 275:1593-1599.
42. Tricomi E, Balleine BW, O'Doherty JP (2009): A specific role for posterior dorsolateral striatum in human habit learning. Eur J Neurosci 29:2225-2232.
43. van den Bos W, Cohen MX, Kahnt T, Crone EA (2012): Striatum-medial prefrontal cortex connectivity predicts developmental changes in reinforcement learning. Cereb Cortex 22:1247-1255.
44. Wager TD, Waugh CE, Lindquist M, Noll DC, Fredrickson BL, Taylor SF (2009): Brain mediators of cardiovascular responses to social threat: Part I: Reciprocal dorsal and ventral sub-regions of the medial prefrontal cortex and heart-rate reactivity. NeuroImage 47:821-835.
45. Watkins CJCH, Dayan P (1992): Q-Learning. Mach Learn 8:279-292.
46. Wimmer GE, Daw ND, Shohamy D (2012): Generalization of value in reinforcement learning by humans. Eur J Neurosci 35:1092-1104.
47. Wunderlich K, Dayan P, Dolan RJ (2012): Mapping value based planning and extensively trained choice in the human brain. Nat Neurosci 15:786-791.
48. Xu T, Yu X, Perlik AJ, Tobin WF, Zweig JA, Tennant K, Jones T, Zuo Y (2009): Rapid formation and selective stabilization of synapses for enduring motor memories. Nature 462:915-919.
49. Yin HH, Knowlton BJ (2006): The role of the basal ganglia in habit formation. Nat Rev Neurosci 7:464-476.