Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: Computational analysis

Cerebral Cortex

Volumn 22, Issue 3, 2012, Pages 509-526

  Frank, Michael J ^a   Badre, David ^a  

^a BROWN UNIVERSITY   (United States)

Author keywords

basal ganglia; computational model; hierarchical reinforcement learning; prefrontal cortex

Indexed keywords

ACCURACY; ATTENTION; BASAL GANGLION; BAYES THEOREM; BRAIN REGION; COGNITION; CORPUS STRIATUM; DOPAMINERGIC SYSTEM; FUNCTIONAL MAGNETIC RESONANCE IMAGING; LEARNING; MATHEMATICAL COMPUTING; MATHEMATICAL MODEL; PREFRONTAL CORTEX; PRIORITY JOURNAL; QUANTITATIVE ANALYSIS; REINFORCEMENT; REVIEW; REWARD;

COMPUTER SIMULATION; CORPUS STRIATUM; HUMANS; LEARNING; MODELS, NEUROLOGICAL; NEURAL PATHWAYS; NEURONS; PREFRONTAL CORTEX; REINFORCEMENT (PSYCHOLOGY);

EID: 84857211334     PISSN: 10473211     EISSN: 14602199     Source Type: Journal    
DOI: 10.1093/cercor/bhr114     Document Type: Review

References (92)

1. Akaike H. 1974. A new look at the statistical mode identification. IEEE Trans Automat Contr. 19:716-723.
2. Alexander G, DeLong M, Strick P. 1986. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu Rev Neurosci. 9:357-381. (Pubitemid 16055790)
3. Badre D. 2008. Cognitive control, hierarchy, and the rostro-caudal organization of the frontal lobes. Trends Cog Sci. 12:193-200.
4. Badre D, D'Esposito M. 2007. Functional magnetic resonance imaging evidence for a hierarchical organization of the prefrontal cortex. J Cogn Neurosci. 19:2082-2099. (Pubitemid 350220518)
5. Badre D, Hoffman J, Cooney J, D'Esposito M. 2009. Hierarchical cognitive control deficits following damage to the human frontal lobe. Nat Neurosci. 12:515-522.
6. Badre D, Kayser A, D'Esposito M. 2010. Frontal cortex and the discovery of abstract action rules. Neuron. 66:315-326.
7. Baier B, Karnath HO, Dieterich M, Birklein F, Heinze C, Muller NG. 2010. Keeping memory clear and stable-the contribution of human basal ganglia and prefrontal cortex to working memory. J Neurosci. 30:9788-9792.
8. Balleine BW, O'Doherty JP. 2010. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology. 35:48-69.
9. Botvinick M, Niv Y, Barto AC. 2009. Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition. 113:262-280.
10. Botvinick MM. 2007. Multilevel structure in behaviour and in the brain: a model of Fuster's hierarchy. Philos Trans R Soc Lond B Biol Sci. 362:1615-1626. (Pubitemid 47395853)
11. Botvinick MM. 2008. Hierarchical models of behavior and prefrontal function. Trends Cog Sci. 12:201-208.
12. Brown J, Bullock D, Grossberg S. 2004. How laminar frontal cortex and basal ganglia circuits interact to control planned and reactive saccades. Neural Netw. 17:471-510. (Pubitemid 38510335)
13. Calzavara R, Mailly P, Haber S. 2007. Relationship between the corticostriatal terminals from areas 9 and 46 and those from area 8a dorsal and rostral premotor cortex and area 24c: an anatomical substrate for cognition to action. Eur J Neurosci. 26:2005-2024. (Pubitemid 47493207)
14. Camerer C, Ho TH. 1999. Experienced-weighted attraction learning in normal form games. Econometrica. 67:827-874.
15. Cavanagh JF, Frank MJ, Klein TJ, Allen JB. 2010. Frontal theta links prediction errors to behavioral adaptation in reinforcement learning. Neuroimage. 49:3198-3209.
16. Christoff K, Keramatian K, Gordon AM, Smith R, Madler B. 2009. Prefrontal organization of cognitive control according to levels of abstraction. Brain Res. 1286:94-105.
17. Christoff K, Ream JM, Geddes LPT, Gabrieli JDE. 2003. Evaluating selfgenerated information: anterior prefrontal contributions to human cognition. Behav Neurosci. 117:1161-1168. (Pubitemid 37549419)
18. Cools R, Altamirano L, D'Esposito M. 2006. Reversal learning in Parkinson's disease depends on medication status and outcome valence. Neuropsychologia. 44:1663-1673. (Pubitemid 43928015)
19. Cools R, Barker RA, Sahakian BJ, Robbins TW. 2001. Mechanisms of cognitive set flexibility in Parkinson's disease. Brain. 124:2503-2512. (Pubitemid 33134938)
20. Cools R, Frank MJ, Gibbs SE, Miyakawa A, Jagust W, D'Esposito M. 2009. Striatal dopamine predicts outcome-specific reversal learning and its sensitivity to dopaminergic drug administration. J Neurosci. 29:1538.
21. Cools R, Gibbs SE, Miyakawa A, Jagust W, D'Esposito M. 2008. Working memory capacity predicts dopamine synthesis capacity in the human striatum. J Neurosci. 28:1208-1212. (Pubitemid 351186336)
22. Cools R, Lewis SJG, Clark L, Barker RA, Robbins TW. 2007. L-dopa disrupts activity in the nucleus accumbens during reversal learning in Parkinson's disease. Neuropsychopharmacology. 32:180-189. (Pubitemid 44967796)
23. Cools R, Sheridan M, Jacobs E, D'Esposito M. 2007. Impulsive personality predicts dopamine-dependent changes in frontostriatal activity during component processes of working memory. J Neurosci. 27:5506-5514. (Pubitemid 46790174)
24. Crone EA, Wendelken C, Donohue SE, Bunge SA. 2006. Neural evidence for dissociable components of task-switching. Cereb cortex. 16:475-486.
25. Dagher A, Robbins T. 2009. Personality addiction dopamine: insights from Parkinson's disease. Neuron. 61:502-510.
26. Daw ND, Niv Y, Dayan P. 2005. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci. 8:1704-1711. (Pubitemid 41683198)
27. Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ. 2006. Cortical substrates for exploratory decisions in humans. Nature. 441:876-879.
28. Doll BB, Jacobs WJ, Sanfey AG, Frank MJ. 2009. Instructional control of reinforcement learning: a behavioral and neurocomputational investigation. Brain Res. 1299:74-94.
29. Doya K, Samejima K, Katagiri K, Kawato M. 2002. Multiple model-based reinforcement learning. Neural Comput. 14:1347-1370.
30. Draganski B, Kherif F, Klöppel S, Cook PA, Alexander DC, Parker GJM, Deichmann R, Ashburner J, Frackowiak RSJ. 2008. Evidence for segregated and integrative connectivity patterns in the human basal ganglia. J Neurosci. 28:7143-7152.
31. Frank MJ. 2005. Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and non-medicated parkinsonism. J Cogn Neurosci. 17:51-72.
32. Frank MJ. 2006. Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making. Neural Netw. 19:1120-1136. (Pubitemid 44466333)
33. Frank MJ, Claus ED. 2006. Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychol Rev. 113:300-326. (Pubitemid 43815199)
34. Frank MJ, Fossella JA. 2011. Neurogenetics and pharmacology of learning, motivation, and cognition. Neuropsychopharmacology. 36:133-152.
35. Frank MJ, Loughry B, O'Reilly RC. 2001. Interactions between the frontal cortex and basal ganglia in working memory: a computational model. Cogn Affect Behav Neurosci. 1:137-160. (Pubitemid 38392490)
36. Frank MJ, Moustafa AA, Haughey HM, Curran T, Hutchison KE. 2007. Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc Natl Acad Sci USA. 104:16311-16316. (Pubitemid 350099356)
37. Frank MJ, O'Reilly RC. 2006. A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidol. Behav Neurosci. 120:497-517. (Pubitemid 44050530)
38. Frank MJ, Seeberger LC, O'Reilly RC. 2004. By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science. 306:1940-1943. (Pubitemid 39627859)
39. Fuster JM. 1997. The prefrontal cortex: anatomy, physiology and neuropsychology of the frontal lobe. 3rd ed. New York: Lippincott-Raven.
40. Gershman SJ, Pesaran B, Daw ND. 2009. Human reinforcement learning subdivides structured action spaces by learning effector-specific values. J Neurosci. 29:13524-13531.
41. Graybiel AM. 1995. Building action repertoires: memory and learning functions of the basal ganglia. Curr Opin Neurobiol. 5:733.
42. Gruber AJ, Dayan P, Gutkin BS, Solla SA. 2006. Dopamine modulation in the basal ganglia locks the gate to working memory. J Comput Neurosci. 20:153-166.
43. Gurney K, Prescott TJ, Redgrave P. 2001a. A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol Cybern. 84:401-410. (Pubitemid 33605190)
44. Gurney K, Prescott TJ, Redgrave P. 2001b. A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour. Biol Cybern. 84:411-424.
45. Haber SN. 2004. The primate basal ganglia: parallel and integrative networks. J Chem Neuroanat. 26:317-330. (Pubitemid 38077155)
46. Hampton AN, Bossaerts P, O'Doherty JP. 2006. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J Neurosci. 26:8360-8367. (Pubitemid 44309850)
47. Hazy TE, Frank MJ, O'Reilly RC. 2007. Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system. Philos Trans R Soc B Biol Sci. 362:105-118.
48. Hazy TE, Frank MJ, O'Reilly RC. 2010. Neural mechanisms of acquired phasic dopamine responses in learning. Neurosci Biobehav Rev. 34:701-720.
49. Hochreiter S, Schmidhuber J. 1997. Long short term memory. Neural Comput. 9:1735-1780.
50. Holroyd CB, Coles MGH. 2002. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol Rev. 109:679-709.
51. Houk JC. 2005. Agents of the mind. Biol Cybern. 92:427-437. (Pubitemid 40834171)
52. Houk JC, Wise SP. 1995. Distributed modular architectures linking basal ganglia, cerebellum, and cerebral cortex: their role in planning and controlling action. Cereb Cortex. 5:95-110.
53. Humphries MD, Stewart RD, Gurney KN. 2006. A physiologically plausible model of action selection and oscillatory activity in the basal ganglia. J Neurosci. 26:12921-12942. (Pubitemid 44954612)
54. Inase M, Tokuno H, Nambu A, Akazawa T, Takada M. 1999. Corticostriatal and corticosubthalamic input zones from the presupplementary motor area in the macaque monkey: comparison with the input zones from the supplementary motor area. Brain Res. 833:191-201. (Pubitemid 29333793)
55. Jacobs RA, Jordan MI, Nowlan SJ, Hinton GE. 1991. Adaptive mixtures of local experts. Neural Comput. 3:79-87.
56. Joel D, Weiner I. 2000. The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum. Neuroscience. 96:451.
57. Koechlin E, Corrado G, Pietrini P, Grafman J. 2000. Dissociating the role of the medial and lateral anterior prefrontal cortex in human planning. Proc Natl Acad Sci USA. 97:7651-7656. (Pubitemid 30431518)
58. Koechlin E, Hyafil A. 2007. Anterior prefrontal function and the limits of human decision-making. Science. 318:594-598. (Pubitemid 350014799)
59. Koechlin E, Ody C, Kouneiher F. 2003. The architecture of cognitive control in the human prefrontal cortex. Science. 302:1181-1184.
60. Koechlin E, Summerfield C. 2007. An information theoretical approach to prefrontal executive function. Trends Cogn Sci. 11:229-235. (Pubitemid 46809561)
61. Kouneiher F, Charron S, Koechlin E. 2009. Motivation and cognitive control in the human prefrontal cortex. Nat Neurosci. 12:659-669.
62. Kruschke JK. 2001. Toward a unified model of attention in associative learning. J Math Psychol. 45:812-863.
63. Lehericy S, Ducros M, Krainik A, Francois C, Vande Moortele P, Ugurbi LK, Kim D. 2004. 3-D diffusion tensor axonal tracking shows distinct SMA and pre-SMA projections to the human striatum. Cereb Cortex. 14:1302.
64. Lehericy S, Ducros M, Vande Moortele PF, Francois C, Thivard L, Poupon C, Swindale N, Ugurbil K, Kim DS. 2004. Diffusion tensor fiber tracking shows distinct corticostriatal circuits in humans. Ann Neurol. 55:522-529. (Pubitemid 38391969)
65. McNab F, Klingberg T. 2008. Prefrontal cortex and basal ganglia control access to working memory. Nat Neurosci. 11:103-107.
66. Miller EK, Cohen JD. 2001. An integrative theory of prefrontal cortex function. Annu Rev Neurosci. 24:167-202. (Pubitemid 32695227)
67. Mink JW. 1996. The basal ganglia: focused selection and inhibition of competing motor programs. Prog Neurobiol. 50:381-425. (Pubitemid 27028538)
68. Montague PR, Dayan P, Sejnowski TJ. 1997. A framework for mesencephalic dopamine systems based on predictive hebbian learning. J Neurosci. 16:1936-1947. (Pubitemid 26145969)
69. Moustafa AA, Sherman SJ, Frank MJ. 2008. A dopaminergic basis for working memory, learning, and attentional shifting in Parkinson's disease. Neuropsychologia. 46:3144-3156.
70. O'Reilly RC, Frank MJ. 2006. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput. 18:283-328. (Pubitemid 43543839)
71. Palminteri S, Lebreton M, Worbe Y, Grabli D, Hartmann A, Pessiglione M. 2009. Pharmacological modulation of subliminal learning in Parkinson's and Tourette's syndromes. Proc Natl Acad Sci USA. 106:19179-19184.
72. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD. 2006. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature. 442:1042-1045. (Pubitemid 44330261)
73. Postuma RB, Dagher A. 2006. Basal ganglia functional connectivity based on a meta-analysis of 126 positron emission tomography and functional magnetic resonance imaging publications. Cereb Cortex. 16:1508-1521. (Pubitemid 44526974)
74. Pucak ML, Levitt JB, Lund JS, Lewis DA. 1996. Patterns of intrinsic and associational circuitry in monkey prefrontal cortex. J Comp Neurol. 376:614-630. (Pubitemid 27009844)
75. Reynolds JN, Wickens JR. 2002. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15:507-521. (Pubitemid 34947461)
76. Reynolds JR, O'Reilly RC. 2009. Developing PFC representations using reinforcement learning. Cognition. 113:281-292.
77. Rougier NP, Noelle D, Braver TS, Cohen JD, O'Reilly RC. 2005. Prefrontal cortex and the flexibility of cognitive control: rules without symbols. Proc Natl Acad Sci USA. 102:7338-7343. (Pubitemid 40696380)
78. Rushworth MFS, Hadland KA, Gaffan D, Passingham RE. 2003. The effect of cingulate cortex lesions on task switching and working memory. J Cogn Neurosci. 15:338-353. (Pubitemid 36438945)
79. Rushworth MFS, Hadland KA, Paus T, Sipila PK. 2002. Role of the human medial frontal cortex in task switching: a combined fMRI and TMS study. J Neurophysiol. 87:2577-2592. (Pubitemid 34670728)
80. Samejima K, Doya K. 2007. Multiple representations of belief states and action values in corticobasal ganglia loops. Ann N Y Acad Sci. 1104:213-228. (Pubitemid 47092747)
81. Samejima K, Ueda Y, Doya K, Kimura M. 2005. Representation of actionspecific reward values in the striatum. Science. 310:1337-1340. (Pubitemid 41698867)
82. Schonberg T, O'Doherty J, Joel D, Inzelberg R, Segev Y, Daw N. 2010. Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson's disease patients: evidence from a model-based fMRI study. Neuroimage. 49:772-781.
83. Schultz W, Dayan P, Montague PR. 1997. A neural substrate of prediction and reward. Science. 275:1593.
84. Shen W, Flajolet M, Greengard P, Surmeier DJ. 2008. Dichotomous dopaminergic control of striatal synaptic plasticity. Science. 321(5890):848-851.
85. Siessmeier T, Kienast T, Wrase J, Larsen JL, Braus DF, Smolka MN, Buchholz HG, Schreckenberger M, Rosch F, Cumming P, et al. 2006. Net influx of plasma 6-18f-fluoro-l-dopa (fdopa) to the ventral striatum correlates with prefrontal processing of affective stimuli. Eur J Neurosci. 24:305-313. (Pubitemid 44050688)
86. Smith AC, Frank LM, Wirth S, Yanike M, Hu D, Kubota Y, Graybiel AM, Suzuki WA, Brown EN. 2004. Dynamic analysis of learning in behavioral experiments. J Neurosci. 24:447-461. (Pubitemid 38112376)
87. Stephan KE, Penny WD, Daunizeau J, Moran RJ, Friston KJ. 2009. Bayesian model selection for group studies. Neuroimage. 46:1004-1017.
88. Stollstorff M, Foss-Feig J, Cook EH, Stein MA, Gaillard WD, Vaidya CJ. 2010. Neural response to working memory load varies by dopamine transporter genotype in children. Neuroimage. 53:970-977.
89. Surmeier DJ, Shen W, Day M, Gertler T, Chan S, Tian X, Plotkin JL. 2010. The role of dopamine in modulating the structure and function of striatal circuits. Prog Brain Res. 183:148-167.
90. Voon V, Pessiglione M, Brezing C, Gallea C, Fernandez HH, Dolan RJ, Hallett M. 2010. Mechanisms underlying dopamine-mediated reward bias in compulsive behaviors. Neuron. 65:135-142.
91. Voytek B, Knight RT. 2010. Prefrontal cortex and basal ganglia contributions to visual working memory. Proc Natl Acad Sci USA. 107:18167-18172.
92. Xu Y. 2002. Limitations of object-based feature encoding in visual shortterm memory. J Exp Psychol. 28:458-468.