Action Selection and Action Value in Frontal-Striatal Circuits

Neuron

Volumn 74, Issue 5, 2012, Pages 947-960

  Seo, Moonsang ^a   Lee, Eunjeong ^a   Averbeck, Bruno B ^a  

^a NATIONAL INSTITUTE OF MENTAL HEALTH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACTION POTENTIAL; ACTION SELECTION; ACTION VALUE; ANIMAL BEHAVIOR; ANIMAL EXPERIMENT; ARTICLE; BRAIN NERVE CELL; BRAIN REGION; CONTROLLED STUDY; DECISION MAKING; DORSA STRIATUM; FRONTAL CORTEX; HAPLORHINI; LATERAL PREFRONTAL CORTEX; MALE; NONHUMAN; PREFRONTAL CORTEX; PRIORITY JOURNAL; SACCADIC EYE MOVEMENT; STRIATE CORTEX; TASK PERFORMANCE;

ACTION POTENTIALS; ANALYSIS OF VARIANCE; ANIMALS; BIAS (EPIDEMIOLOGY); COLOR PERCEPTION; CORPUS STRIATUM; DECISION MAKING; EYE MOVEMENTS; FRONTAL LOBE; HAPLORHINI; MALE; MODELS, NEUROLOGICAL; NEURONS; PHOTIC STIMULATION; REWARD; TIME FACTORS;

EID: 84861949192     PISSN: 08966273     EISSN: 10974199     Source Type: Journal    
DOI: 10.1016/j.neuron.2012.03.037     Document Type: Article

References (87)

1. Albin R.L., Young A.B., Penney J.B. The functional anatomy of basal ganglia disorders. Trends Neurosci. 1989, 12:366-375.
2. Alexander G.E., DeLong M.R., Strick P.L. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu. Rev. Neurosci. 1986, 9:357-381.
3. Amemori K., Gibb L.G., Graybiel A.M. Shifting responsibly: the importance of striatal modularity to reinforcement learning in uncertain environments. Front. Hum. Neurosci. 2011, 5:47.
4. Antzoulatos E.G., Miller E.K. Differences between neural activity in prefrontal cortex and striatum during learning of novel abstract categories. Neuron 2011, 71:243-249.
5. Apicella P., Ravel S., Deffains M., Legallet E. The role of striatal tonically active neurons in reward prediction error signaling during instrumental task performance. J. Neurosci. 2011, 31:1507-1515.
6. Averbeck B.B., Lee D. Prefrontal neural correlates of memory for sequences. J. Neurosci. 2007, 27:2204-2211.
7. Averbeck B.B., Chafee M.V., Crowe D.A., Georgopoulos A.P. Parallel processing of serial movements in prefrontal cortex. Proc. Natl. Acad. Sci. USA 2002, 99:13172-13177.
8. Averbeck B.B., Chafee M.V., Crowe D.A., Georgopoulos A.P. Neural activity in prefrontal cortex during copying geometrical shapes. I. Single cells encode shape, sequence, and metric parameters. Exp. Brain Res. 2003, 150:127-141.
9. Averbeck B.B., Sohn J.W., Lee D. Activity in prefrontal cortex during dynamic selection of action sequences. Nat. Neurosci. 2006, 9:276-282.
10. Bar-Gad I., Morris G., Bergman H. Information processing, dimensionality reduction and reinforcement learning in the basal ganglia. Prog. Neurobiol. 2003, 71:439-473.
11. Barnes T.D., Kubota Y., Hu D., Jin D.Z., Graybiel A.M. Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories. Nature 2005, 437:1158-1161.
12. Barraclough D.J., Conroy M.L., Lee D. Prefrontal cortex and decision making in a mixed-strategy game. Nat. Neurosci. 2004, 7:404-410.
13. Benjamini Y., Yekutieli D. The control of the false discovery rate in multiple testing under dependency. Ann. Stat. 2001, 29:1165-1188.
14. Berns G.S., Sejnowski T.J. A computational model of how the basal ganglia produce sequences. J. Cogn. Neurosci. 1998, 10:108-121.
15. Brasted P.J., Wise S.P. Comparison of learning-related neuronal activity in the dorsal premotor cortex and striatum. Eur. J. Neurosci. 2004, 19:721-740.
16. Crutcher M.D., Alexander G.E. Movement-related neuronal activity selectively coding either direction or muscle pattern in three motor areas of the monkey. J. Neurophysiol. 1990, 64:151-163.
17. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8:1704-1711.
18. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
19. Dayan P., Daw N.D. Decision theory, reinforcement learning, and the brain. Cogn. Affect. Behav. Neurosci. 2008, 8:429-453.
20. DeLong M.R. Primate models of movement disorders of basal ganglia origin. Trends Neurosci. 1990, 13:281-285.
21. Denny-Brown D., Yanagisawa N. The role of the basal ganglia in the initiation of movement. The Basal Ganglia 1976, 115-149. Raven Press, New York. M.D. Yahr (Ed.).
22. Ding L., Gold J.I. Caudate encodes multiple computations for perceptual decisions. J. Neurosci. 2010, 30:15747-15759.
23. Djamshidian A., O'Sullivan S.S., Wittmann B.C., Lees A.J., Averbeck B.B. Novelty seeking behaviour in Parkinson's disease. Neuropsychologia 2011, 49:2483-2488.
24. Doya K. Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr. Opin. Neurobiol. 2000, 10:732-739.
25. Doyon J., Bellec P., Amsel R., Penhune V., Monchi O., Carrier J., Lehéricy S., Benali H. Contributions of the basal ganglia and functionally related brain structures to motor learning. Behav. Brain Res. 2009, 199:61-75.
26. Frank M.J. Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J. Cogn. Neurosci. 2005, 17:51-72.
27. Frank M.J., Seeberger L.C., O'Reilly R.C. By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 2004, 306:1940-1943.
28. Fujii N., Graybiel A.M. Representation of action sequence boundaries by macaque prefrontal cortical neurons. Science 2003, 301:1246-1249.
29. Fujii N., Graybiel A.M. Time-varying covariance of neural activities recorded in striatum and frontal cortex as monkeys perform sequential-saccade tasks. Proc. Natl. Acad. Sci. USA 2005, 102:9032-9037.
30. Funahashi S., Bruce C.J., Goldman-Rakic P.S. Neuronal activity related to saccadic eye movements in the monkey's dorsolateral prefrontal cortex. J. Neurophysiol. 1991, 65:1464-1483.
31. Fuster J. The Prefrontal Cortex 2008, Academic Press, London. Fourth Edition.
32. Gazi L., Strange P.G. Dopamine receptors. Understanding G Protein-Coupled Receptors and Their Role in the CNS 2002, 264-285. Oxford University Press, New York. M.N. Pangalos, C.H. Davies (Eds.).
33. Graybiel A.M. Habits, rituals, and the evaluative brain. Annu. Rev. Neurosci. 2008, 31:359-387.
34. Grillner S., Hellgren J., Ménard A., Saitoh K., Wikström M.A. Mechanisms for selection of basic motor programs-roles for the striatum and pallidum. Trends Neurosci. 2005, 28:364-370.
35. Haber S.N., Fudge J.L., McFarland N.R. Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J. Neurosci. 2000, 20:2369-2382.
36. Haber S.N., Kim K.S., Mailly P., Calzavara R. Reward-related cortical inputs define a large striatal region in primates that interface with associative cortical connections, providing a substrate for incentive-based learning. J. Neurosci. 2006, 26:8368-8376.
37. Hayden B.Y., Pearson J.M., Platt M.L. Fictive reward signals in the anterior cingulate cortex. Science 2009, 324:948-950.
38. Hazy T.E., Frank M.J., O'reilly R.C. Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2007, 362:1601-1613.
39. Histed M.H., Pasupathy A., Miller E.K. Learning substrates in the primate prefrontal cortex and striatum: sustained activity related to successful actions. Neuron 2009, 63:244-253.
40. Hollerman J.R., Tremblay L., Schultz W. Influence of reward expectation on behavior-related neuronal activity in primate striatum. J. Neurophysiol. 1998, 80:947-963.
41. Houk J.C., Bastianen C., Fansler D., Fishbach A., Fraser D., Reber P.J., Roy S.A., Simo L.S. Action selection and refinement in subcortical loops through basal ganglia and cerebellum. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2007, 362:1573-1583.
42. Humphries M.D., Stewart R.D., Gurney K.N. A physiologically plausible model of action selection and oscillatory activity in the basal ganglia. J. Neurosci. 2006, 26:12921-12942.
43. Jin D.Z., Fujii N., Graybiel A.M. Neural representation of time in cortico-basal ganglia circuits. Proc. Natl. Acad. Sci. USA 2009, 106:19156-19161.
44. Joshua M., Adler A., Mitelman R., Vaadia E., Bergman H. Midbrain dopaminergic neurons and striatal cholinergic interneurons encode the difference between reward and aversive events at different epochs of probabilistic classical conditioning trials. J. Neurosci. 2008, 28:11673-11684.
45. Kamali Sarvestani I., Lindahl M., Hellgren-Kotaleski J., Ekeberg O. The arbitration-extension hypothesis: a hierarchical interpretation of the functional organization of the Basal Ganglia. Front. Syst. Neurosci. 2011, 5:13.
46. Kennerley S.W., Wallis J.D. Evaluating choices by single neurons in the frontal lobe: outcome value encoded across multiple decision variables. Eur. J. Neurosci. 2009, 29:2061-2073.
47. Lauwereyns J., Takikawa Y., Kawagoe R., Kobayashi S., Koizumi M., Coe B., Sakagami M., Hikosaka O. Feature-based anticipation of cues that predict reward in monkey caudate nucleus. Neuron 2002, 33:463-473.
48. Lauwereyns J., Watanabe K., Coe B., Hikosaka O. A neural correlate of response bias in monkey caudate nucleus. Nature 2002, 418:413-417.
49. Lavin A., Nogueira L., Lapish C.C., Wightman R.M., Phillips P.E., Seamans J.K. Mesocortical dopamine neurons operate in distinct temporal domains using multimodal signaling. J. Neurosci. 2005, 25:5013-5023.
50. Leon M.I., Shadlen M.N. Effect of expected reward magnitude on the response of neurons in the dorsolateral prefrontal cortex of the macaque. Neuron 1999, 24:415-425.
51. Marsden C.D. The mysterious motor function of the basal ganglia: the Robert Wartenberg Lecture. Neurology 1982, 32:514-539.
52. Marsden C.D., Obeso J.A. The functions of the basal ganglia and the paradox of stereotaxic surgery in Parkinson's disease. Brain 1994, 117:877-897.
53. Matsumoto M., Hikosaka O. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 2009, 459:837-841.
54. Matsumoto N., Hanakawa T., Maki S., Graybiel A.M., Kimura M. Role of nigrostriatal dopamine system in learning to perform sequential motor tasks in a predictive manner. J. Neurophysiol. 1999, 82:978-998.
55. Matsumoto N., Minamimoto T., Graybiel A.M., Kimura M. Neurons in the thalamic CM-Pf complex supply striatal neurons with information about behaviorally significant sensory events. J. Neurophysiol. 2001, 85:960-976.
56. Middleton F.A., Strick P.L. Basal ganglia and cerebellar loops: motor and cognitive circuits. Brain Res. Brain Res. Rev. 2000, 31:236-250.
57. Miller E.K., Cohen J.D. An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 2001, 24:167-202.
58. Minamimoto T., Hori Y., Kimura M. Complementary process to response bias in the centromedian nucleus of the thalamus. Science 2005, 308:1798-1801.
59. Mink J.W. The basal ganglia: focused selection and inhibition of competing motor programs. Prog. Neurobiol. 1996, 50:381-425.
60. Nakamura K., Hikosaka O. Role of dopamine in the primate caudate nucleus in reward modulation of saccades. J. Neurosci. 2006, 26:5360-5369.
61. Nakano K., Hasegawa Y., Tokushige A., Nakagawa S., Kayahara T., Mizuno N. Topographical projections from the thalamus, subthalamic nucleus and pedunculopontine tegmental nucleus to the striatum in the Japanese monkey, Macaca fuscata. Brain Res. 1990, 537:54-68.
62. Niv Y., Daw N.D., Joel D., Dayan P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl.) 2007, 191:507-520.
63. Norman D.A., Shallice T. Attention to action willed and automatic control of behavior. Consciousness and Self-Regulation. Advances in Research and Theory 1986, 1-18. Plenum Press, New York. R.J. Davidson, G.E. Schwartz, D. Shapiro (Eds.).
64. O'Doherty J., Dayan P., Schultz J., Deichmann R., Friston K., Dolan R.J. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 2004, 304:452-454.
65. Parent A., Hazrati L.N. Anatomical aspects of information processing in primate basal ganglia. Trends Neurosci. 1993, 16:111-116.
66. Parent A., Hazrati L.N. Functional anatomy of the basal ganglia. I. The cortico-basal ganglia-thalamo-cortical loop. Brain Res. Brain Res. Rev. 1995, 20:91-127.
67. Parthasarathy H.B., Schall J.D., Graybiel A.M. Distributed but convergent ordering of corticostriatal projections: analysis of the frontal eye field and the supplementary eye field in the macaque monkey. J. Neurosci. 1992, 12:4468-4488.
68. Parush N., Tishby N., Bergman H. Dopaminergic balance between reward maximization and policy complexity. Front. Syst. Neurosci. 2011, 5:22.
69. Pasupathy A., Miller E.K. Different time courses of learning-related activity in the prefrontal cortex and striatum. Nature 2005, 433:873-876.
70. Pessiglione M., Seymour B., Flandin G., Dolan R.J., Frith C.D. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 2006, 442:1042-1045.
71. Platt M.L., Glimcher P.W. Neural correlates of decision variables in parietal cortex. Nature 1999, 400:233-238.
72. Redgrave P., Prescott T.J., Gurney K. The basal ganglia: a vertebrate solution to the selection problem?. Neuroscience 1999, 89:1009-1023.
73. Rubchinsky L.L., Kopell N., Sigvardt K.A. Modeling facilitation and inhibition of competing motor programs in basal ganglia subthalamic nucleus-pallidal circuits. Proc. Natl. Acad. Sci. USA 2003, 100:14427-14432.
74. Samejima K., Ueda Y., Doya K., Kimura M. Representation of action-specific reward values in the striatum. Science 2005, 310:1337-1340.
75. Schultz W. Getting formal with dopamine and reward. Neuron 2002, 36:241-263.
76. Schultz W. Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 2006, 57:87-115.
77. Seamans J.K., Robbins T.W. Dopamine modulation of the prefrontal cortex and cognitive function. The Dopamine Receptors 2010, 373-398. Humana Press, New York. K.A. Neve (Ed.).
78. Seo M., Beigi M., Jahanshahi M., Averbeck B.B. Effects of dopamine medication on sequence learning with stochastic feedback in Parkinson's disease. Front. Syst. Neurosci. 2010, 4. 10.3389/fnsys.2010.00036.
79. Shidara M., Aigner T.G., Richmond B.J. Neuronal signals in the monkey ventral striatum related to progress through a predictable series of trials. J. Neurosci. 1998, 18:2613-2625.
80. Simmons J.M., Ravel S., Shidara M., Richmond B.J. A comparison of reward-contingent neuronal activity in monkey orbitofrontal cortex and ventral striatum: guiding actions toward rewards. Ann. N Y Acad. Sci. 2007, 1121:376-394.
81. Sul J.H., Jo S., Lee D., Jung M.W. Role of rodent secondary motor cortex in value-based action selection. Nat. Neurosci. 2011, 14:1202-1208.
82. Surmeier D.J., Song W.J., Yan Z. Coordinated expression of dopamine receptors in neostriatal medium spiny neurons. J. Neurosci. 1996, 16:6579-6591.
83. Surmeier D.J., Ding J., Day M., Wang Z., Shen W. D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends Neurosci. 2007, 30:228-235.
84. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge, MA.
85. Turner R.S., Desmurget M. Basal ganglia contributions to motor control: a vigorous tutor. Curr. Opin. Neurobiol. 2010, 20:704-716.
86. Williams Z.M., Eskandar E.N. Selective enhancement of associative learning by microstimulation of the anterior caudate. Nat. Neurosci. 2006, 9:562-568.
87. Zar J.H. Biostatistical Analysis 1999, Prentice Hall, Upper Saddle River.