Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits 2: Evidence from fMRI

Cerebral Cortex

Volumn 22, Issue 3, 2012, Pages 527-536

  Badre, David ^a   Frank, Michael J ^a  

^a BROWN UNIVERSITY   (United States)

Author keywords

basal ganglia; cognitive control; fMRI; prefrontal cortex, reinforcement learning

Indexed keywords

ALGORITHM; ARTICLE; BAYES THEOREM; BRAIN REGION; COGNITION; CORPUS STRIATUM; FRONTAL LOBE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; LEARNING; MATHEMATICAL COMPUTING; PRIORITY JOURNAL; QUANTITATIVE ANALYSIS; REINFORCEMENT;

BRAIN MAPPING; CEREBRAL CORTEX; CLINICAL TRIALS AS TOPIC; CORPUS STRIATUM; HUMANS; LEARNING; MAGNETIC RESONANCE IMAGING; MODELS, NEUROLOGICAL; NERVE NET; NEURAL PATHWAYS; REINFORCEMENT (PSYCHOLOGY);

EID: 84857207526     PISSN: 10473211     EISSN: 14602199     Source Type: Journal    
DOI: 10.1093/cercor/bhr117     Document Type: Article

References (38)

1. Badre D. 2008. Cognitive control, hierarchy, and the rostro-caudal organization of the frontal lobes. Trends Cogn Sci. 12:193-200.
2. Badre D, D'Esposito M. 2007. Functional magnetic resonance imaging evidence for a hierarchical organization of the prefrontal cortex. J Cogn Neurosci. 19:2082-2099. (Pubitemid 350220518)
3. Badre D, D'Esposito M. 2009. Is the rostro-caudal axis of the frontal lobe hierarchical? Nat Rev Neurosci. 10:659-669.
4. Badre D, Hoffman J, Cooney JW, D'Esposito M. 2009. Hierarchical cognitive control deficits following damage to the human frontal lobe. Nat Neurosci. 12:515-522.
5. Badre D, Kayser AS, D'Esposito M. 2010. Frontal cortex and the discovery of abstract action rules. Neuron. 66:315-326.
6. Badre D, Wagner AD. 2004. Selection, integration, and conflict monitoring; assessing the nature and generality of prefrontal . cognitive control mechanisms. Neuron. 41:473-487. (Pubitemid 38221049)
7. Brown JW, Bullock D, Grossberg S. 2004. How laminar frontal cortex and basal ganglia circuits interact to control planned and reactive saccades. Neural Netw. 17:471-510. (Pubitemid 38510335)
8. Bunge SA. 2004. How we use rules to select actions: a review of evidence from cognitive neuroscience. Cogn Affect Behav Neurosci. 4:564-579. (Pubitemid 40342901)
9. Cohen MX, Frank MJ. 2009. Neurocomputational models of basal ganglia function in learning, memory and choice. Behav Brain Res. 199:141-156.
10. Cools R, Ivry RB, D'Esposito M. 2006. The human striatum is necessary for responding to changes in stimulus relevance. J Cogn Neurosci. 18:1973-1983. (Pubitemid 44913354)
11. Dale AM. 1999. Optimal experimental design for event-related fMRI. Hum Brain Mapp. 8:109-114. (Pubitemid 29480297)
12. Draganski B, Kherif F, Kloppel S, Cook PA, Alexander DC, Parker GJ, Deichmann R, Ashburner J, Frackowiak RS. 2008. Evidence for segregated and integrative connectivity patterns in the human Basal Ganglia. J Neurosci. 28:7143-7152.
13. Frank MJ. 2005. Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J Cogn Neurosci. 17:51-72.
14. Frank MJ, Badre D. 2011. Mechanisms of hierarchical reinforcement learning in corticostriatal circuits I: computational analysis 22:509-526.
15. Frank MJ, Loughry B, O'Reilly RC. 2001. Interactions between frontal cortex and basal ganglia in working memory: a computational model. Cogn Affect Behav Neurosci. 1:137-160. (Pubitemid 38392490)
16. Frank MJ, O'Reilly RC. 2006. A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidol. Behav Neurosci. 120:497-517. (Pubitemid 44050530)
17. Fuster JM. 2001. The prefrontal cortex-an update: time is of the essence. Neuron. 30:319-333. (Pubitemid 32536780)
18. Gershman SJ, Pesaran B, Daw ND. 2009. Human reinforcement learning subdivides structured action spaces by learning effector-specific values. J Neurosci. 29:13524-13531.
19. Gruber AJ, Dayan P, Gutkin BS, Solla SA. 2006. Dopamine modulation in the basal ganglia locks the gate to working memory. J Comput Neurosci. 20:153-166.
20. Inase M, Tokuno H, Nambu A, Akazawa T, Takada M. 1999. Corticostriatal and corticosubthalamic input zones from the presupplementary motor area in the macaque monkey: comparison with the input zones from the supplementary motor area. Brain Res. 833:191-201. (Pubitemid 29333793)
21. Kemp JM, Powell TP. 1970. The cortico-striate projection in the monkey. Brain. 93:525-546.
22. Koechlin E, Jubault T. 2006. Broca's area and the hierarchical organization of human behavior. Neuron. 50:963-974. (Pubitemid 43850692)
23. Koechlin E, Ody C, Kouneiher F. 2003. The architecture of cognitive control in the human prefrontal cortex. Science. 302:1181-1185. (Pubitemid 37421016)
24. Koechlin E, Summerfield C. 2007. An information theoretical approach to prefrontal executive function. Trends Cogn Sci. 11:229-235. (Pubitemid 46809561)
25. Lehericy S, Ducros M, Krainik A, Francois C, Van de Moortele PF, Ugurbil K, Kim DS. 2004. 3-D diffusion tensor axonal tracking shows distinct SMA and pre-SMA projections to the human striatum. Cereb Cortex. 14:1302-1309. (Pubitemid 39507281)
26. Lehericy S, Ducros M, Van de Moortele PF, Francois C, Thivard L, Poupon C, Swindale N, Ugurbil K, Kim DS. 2004. Diffusion tensor fiber tracking shows distinct corticostriatal circuits in humans. Ann Neurol. 55:522-529. (Pubitemid 38391969)
27. McClure SM, Berns GS, Montague PR. 2003. Temporal prediction errors in a passive learning task activate human striatum. Neuron. 38:339-346. (Pubitemid 36555658)
28. Miller EK, Cohen JD. 2001. An integrative theory of prefrontal cortex function. Annu Rev Neurosci. 24:167-202. (Pubitemid 32695227)
29. Montague PR, Dayan P, Sejnowski TJ. 1996. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci. 16:1936-1947. (Pubitemid 26145969)
30. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ. 2003. Temporal difference models and reward-related learning in the human brain. Neuron. 38:329-337. (Pubitemid 36555657)
31. O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ. 2004. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science. 304:452-454. (Pubitemid 38495943)
32. O'Reilly RC, Frank MJ. 2006. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput. 18:283-328. (Pubitemid 43543839)
33. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD. 2006. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature. 442:1042-1045. (Pubitemid 44330261)
34. Postuma RB, Dagher A. 2006. Basal ganglia functional connectivity based on a meta-analysis of 126 positron emission tomography and functional magnetic resonance imaging publications. Cereb Cortex. 16:1508-1521. (Pubitemid 44526974)
35. Rutledge RB, Dean M, Caplin A, Glimcher PW. 2010. Testing the reward prediction error hypothesis with an axiomatic model. J Neurosci. 30:13525-13536.
36. Schonberg T, O'Doherty JP, Joel D, Inzelberg R, Segev Y, Daw ND. 2010. Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson's disease patients: evidence from a model-based fMRI study. Neuroimage. 49:772-781.
37. Shen W, FlajoletM, Greengard P, Surmeier DJ. 2008.Dichotomous dopaminergic control of striatal synaptic plasticity. Science. 321:848-851.
38. Voon V, Pessiglione M, Brezing C, Gallea C, Fernandez HH, Dolan RJ, Hallett M. 2010. Mechanisms underlying dopamine-mediated reward bias in compulsive behaviors. Neuron. 65:135-142.