Computational perspectives on forebrain microcircuits implicated in reinforcement learning, action selection, and cognitive control

Neural Networks

Volumn 22, Issue 5-6, 2009, Pages 757-765

  Bullock, Daniel ^a   Tan, Can Ozan ^b,c   John, Yohan J ^a  

^a BOSTON UNIVERSITY   (United States)

^b HARVARD MEDICAL SCHOOL   (United States)

^c and Red Sox Foundation and Massachusetts General Hospital Home Base Program   (United States)

Author keywords

Acetylcholine; Basal ganglia; Decision making; Dopamine; Striatum

Indexed keywords

ACETYLCHOLINE; ACTION SELECTION; BASAL GANGLIA; COGNITIVE CONTROL; CONTROL PROBLEMS; DOPAMINE; FLEXIBLE CONTROL; FRONTAL LOBES; PARALLEL DESIGN; SIGNALING PATHWAYS; STRIATUM; SYNAPTIC MODIFICATION; VOLUNTARY ACTION;

BRAIN; EDUCATION; REINFORCEMENT; REINFORCEMENT LEARNING;

DECISION MAKING;

ACETYLCHOLINE; DOPAMINE; DOPAMINE 2 RECEPTOR; MUSCARINIC M1 RECEPTOR; MUSCARINIC M2 RECEPTOR; MUSCARINIC M4 RECEPTOR;

ACETYLCHOLINE RELEASE; AMYGDALOID NUCLEUS; ARTICLE; BASAL GANGLION; BRAIN FUNCTION; CHOLINERGIC TRANSMISSION; COGNITION; CORPUS STRIATUM; DOPAMINERGIC NERVE CELL; FOREBRAIN; LEARNING; LONG TERM DEPRESSION; LONG TERM POTENTIATION; MOTIVATION; NEUROPHYSIOLOGY; NONHUMAN; PARKINSON DISEASE; PREDICTION; PRIORITY JOURNAL; PROTEIN EXPRESSION; REINFORCEMENT; REWARD; SPEECH; STIMULUS RESPONSE; THALAMUS NUCLEUS;

ACETYLCHOLINE; AMYGDALA; BASAL GANGLIA; CEREBRAL CORTEX; COGNITION; CORPUS STRIATUM; DOPAMINE; HUMANS; LEARNING; NEURAL NETWORKS (COMPUTER); NEURAL PATHWAYS; NEURONS; PROSENCEPHALON; PSYCHOMOTOR PERFORMANCE; REINFORCEMENT (PSYCHOLOGY); SPEECH; SYNAPSES; THALAMUS; TIME FACTORS; UNCERTAINTY;

EID: 68149182676     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neunet.2009.06.008     Document Type: Article

References (90)

1. Albin R.L., Young A.B., and Penney J.B. The functional anatomy of basal ganglia disorders. Trends in Neurosciences 12 (1989) 366
2. Alcantara A.A., Mrzljak L., Jakab R.L., Levey A.I., Hersch S.M., and Goldman-Rakic P.S. Muscarinic m1 and m2 receptor proteins in local circuit and projection neurons of the primate striatum: Anatomical evidence for cholinergic modulation of glutamatergic prefronto-striatal pathways. Journal of Comparative Neurology 434 (2001) 445-460
3. Alm P.A. Stuttering and the basal ganglia: A critical review of possible relations. Journal of Communication Disorders 37 (2004) 325-369
4. Averbeck B.B., Chafee M.V., Crowe D.A., and Georgopoulos A.P. Parallel processing of serial movements in prefrontal cortex. Proceedings of the National Academy of Sciences of the United States of America 99 (2002) 13172-13177
5. Bevan M.D., Booth P.A., Eaton S.A., and Bolam J.P. Selective innervation of neostriatal interneurons by a subclass of neuron in the globus pallidus of the rat. Journal of Neuroscience 18 (1998) 9438-9452
6. Blomeley C.P., Kehoe L.A., and Bracci E. Substance P mediates excitatory interactions between striatal projection neurons. Journal of Neuroscience 29 (2009) 4953-4963
7. Bohland, J. W., Bullock, D., & Guenther, F. H. (2009). Neural representations and mechanisms for the performance of simple speech sequences. Journal of Cognitive Neuroscience (in press)
8. Bonsi P., Martella G., Cuomo D., Platania P., Sciamanna G., Bernardi G., et al. Loss of muscarinic autoreceptor function impairs long-term depression but not long-term potentiation in the striatum. Journal of Neuroscience 28 (2008) 6258
9. Bornkessel I., and Schlesewsky M. The extended argument dependency model: A neurocognitive approach to sentence comprehension across languages. Psychological Review 113 (2006) 787-821
10. Brown J.W., Bullock D., and Grossberg S. How laminar frontal cortex and basal ganglia circuits interact to control planned and reactive saccades. Neural Networks 17 (2004) 471-510
11. Buhusi C.V., and Meck W.H. What makes us tick? Functional and neural mechanisms of interval timing. Nature Reviews Neuroscience 6 (2005) 755-765
12. Bullock D. Adaptive neural models of queuing and timing in fluent action. Trends in Cognitive Science 8 (2004) 426-433
13. Bullock D. From parallel sequence representations to calligraphic control: A conspiracy of neural circuits. Motor Control 8 (2004) 371-391
14. Bullock D., and Tan C.O. Role of basal ganglia in decision sequences: A computational model of dopamine and acetylcholine regulation of action selection, interruption, resumptions, and switching. Society for Neuroscience Abstracts (2007) S514
15. Bullock D., and Tan C.O. Computational implications of microcircuit specializations in forebrain circuits for motivated action selection. Proceedings of the IEEE International Joint Conference on Neural Networks (2009)
16. Calabresi P., Maj R., Pisani A., Mercuri N.B., and Bernardi G. Long-term synaptic depression in the striatum: Physiological and pharmacological characterization. Journal of Neuroscience 12 (1992) 4224-4233
17. Calabresi P., Pisani A., Mercuri N.B., and Bernardi G. Long-term potentiation in the striatum is unmasked by removing the voltage-dependent magnesium block of NMDA receptor channels. European Journal of Neuroscience 4 (1992) 929-935
18. Centonze D., Gubellini P., Bernardi G., and Calabresi P. Permissive role of interneurons in cortical synaptic plasticity. Brain Research Reviews 31 (1999) 1-5
19. Civier, A., Bullock, D., Max, L., & Guenther, F. (2009). Simulating neural impairments to syllable-level command generation in stuttering. Sixth World Congress on Fluency Disorders
20. de Olmos J.S., and Heimer L. The concepts of the ventral striatopallidal system and extended amygdala. Annals of the New York Academy of Sciences 877 (1999) 1-32
21. DeMartino B., Kumaran D., Seymour B., and Dolan R.J. Frames, biases, and rational decision-making in the human brain. Science 313 (2006) 684-687
22. Dominey P.F., Hoen M., and Inui T. A neurolinguistic model of grammatical construction processing. Journal of Cognitive Neuroscience 18 (2006) 2088-2107
23. Dranias M.R., Grossberg S., and Bullock D. Dopaminergic and non-dopaminergic value systems in conditioning and outcome-specific revaluation. Brain Research 1238 (2008) 239-287
24. Farrell S., and Lewandowsky S. Modeling transposition latencies: Constraints for theories of serial order memory. Journal of Memory and Language 51 (2004) 115-135
25. Fino E., Glowinski J., and Venance L. Bidirectional activity-dependent plasticity at corticostriatal synapses. Journal of Neuroscience 25 (2005) 11279-11287
26. Fiorillo C.D., Tobler P.N., and Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299 (2003) 1898-1902
27. Fiorillo C.D., Tobler P.N., and Schultz W. Evidence that the delay-period activity of dopamine neurons corresponds to reward uncertainty rather than backpropagating TD errors. Behavioral and Brain Functions 1 (2005) 7
28. Flores-Hernandez J., Cepeda C., Hernandez-Echeagaray E., Calvert C.R., Jokel E.S., Fienberg A.A., et al. Dopamine enhancement of NMDA currents in dissociated medium-sized striatal neurons: Role of D1 receptors and DARPP-32. Journal of Neurophysiology 88 (2002) 3010-3020
29. Frank M.J. Dynamic dopamine modulation in the basal ganglia: A neurocomputational account of cognitive deficits in medicated and nonmedicated parkinsonism. Journal of Cognitive Neuroscience 17 (2005) 51-72
30. Frank M.J., and Claus E.D. Anatomy of a decision: Striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychological Review 113 (2006) 300-326
31. Frank M.J., and O'Reilly R.C. A mechanistic account of striatal dopamine function in human cognition: Psychopharmacological studies with cabergoline and haloperidol. Behavioral Neuroscience 120 (2006) 497-517
32. Gabel L.A., and Nisenbaum E.S. Muscarinic receptors differentially modulate the persistent potassium currents in striatal spiny projection neurons. Journal of Neurophysiology 81 (1999) 1418-1423
33. Goldberg A.E. Constructions at work: The nature of generalization in language (2006), Oxford, New York
34. Grossberg S. In: Rosen R., and Snell F. (Eds). A theory of human memory: Self-organization and performance of sensory-motor codes, maps, and plans. Progress in theoretical biology Vol. 5 (1978), Academic Press, New York 233-374
35. Gruber A.J., Solla S.A., Surmeier D.J., and Houk J.C. Modulation of striatal single units by expected reward: A spiny neurons model displaying dopamine-induced bistability. Journal of Neurophysiology 90 (2003) 1095-1114
36. Gurney K., Prescott T.J., and Redgrave P. A computational model of action selection in the basal ganglia. i. A new functional anatomy. Biological Cybernetics 84 (2001) 401-410
37. Gurney K., Prescott T.J., and Redgrave P. A computational model of action selection in the basal ganglia. ii. Analysis and simulation of behavior. Biological Cybernetics 84 (2001) 411-423
38. Hasselmo M.E. The role of acetylcholine in learning and memory. Current Opinion in Neurobiology 16 (2006) 710-715
39. Houk J., Adams J.L., and Barto A.G. A model of how the basal ganglia generate and use signals that predict reinforcement (1995), MIT Press, Cambridge, MA 249-270
40. +2 currents in rat striatal neurons through parallel pathways. Journal of Neuroscience 15 (1995) 458-469
41. Humpries M.D., Stewart R.D., and Gurney K.N. A physiologically plausible model of action selection and oscillatory activity in the basal ganglia. Journal of Neuroscience 26 (2006) 12921-12942
42. Ivey, R., Bullock, D., & Grossberg, S. 2008. A neuromorphic model of spatial lookahead planning. In Proceedings of the 12th international conference on cognitive and neural systems
43. Jaeger D., Kita H., and Wilson C.J. Surround inhibition among projection neurons is weak or nonexistent in the rat neostriatum. Journal of Neurophysiology 72 (1994) 2555-2558
44. Kakade S., and Dayan P. Dopamine: Generalization and bonuses. Neural Networks 15 (2002) 549-559
45. Kelley A.E., Cador M., Stinus L., and Le M.M. Neurotensin, substance P, neurokinin-alpha, and enkephalin: Injection into ventral tegmental area in the rat produces differential effects on operant responding. Psychopharmacology (Berlin) 97 (1989) 243-252
46. Kerr J.N.D., and Wickens J.R. Dopamine D-1/D-5 receptor activation is required for long-term potentiation in the rat neostriatum in vitro. Journal of Neurophysiology 85 (2001) 117-124
47. Kilgard M.P., and Merzenich M.M. Cortical map reorganization enabled by nucleus basalis activity. Science 279 (1998) 1714-1718
48. Koos T., and Tepper J.M. Inhibitory control of neostriatal projection neurons by GABAergic interneurons. Nature Neuroscience 2 (1999) 467-472
49. Laurent P.A. The emergence of saliency and novelty responses from reinforcement learning principles. Neural Networks 21 (2008) 1493-1499
50. McDonald A.J. Is there an amygdala and how far does it extend? An anatomical perspective. Annals of the New York Academy of Sciences 985 (2003) 1-21
51. Meck W.H. Selective adjustment of the speed of internal clock and memory processes. Journal of Experimental Psychology: Animal Behaviour Process 9 (1983) 171
52. Meck W.H. Affinity for the dopamine D2 receptor predicts neuroleptic potency in decreasing the speed of an internal clock. Pharmacology, Biochemistry and Behaviour 25 (1986) 1185
53. Meck W.H. Neuropharmacology of timing and time perception. Cognitive Brain Research 3 (1996) 227-242
54. Meck W.H., and Church R.M. Cholinergic modulation of the content of temporal memory. Behavioral Neuroscience 101 (1987) 457-464
55. Nadeau S.E., and Crosson B. Subcortical aphasia. Brain and Language 58 (1997) 355-402
56. Niv Y., Duff M.O., and Dayan P. Dopamine, uncertainty and TD learning. Behavioral and Brain Functions 1 (2005) 6
57. Onn S.-P., and Grace A.A. Dye coupling between rat striatal neurons recorded in vivo: Compartmental organization and modulation by dopamine. Journal of Neurophysiology 71 (1994) 1917-1934
58. Paré D., Royer S., Smith Y., and Lang E.J. Contextual inhibitory gating of impulse traffic in the intra-amygdaloid network. Annals of the New York Academy of Sciences 985 (2003) 78-91
59. Passingham R.E. From where does the motor cortex get its instructions?. In: Wise S.P. (Ed). Higher brain functions (1987), Wiley, New York 67-97
60. Pawlak V., and Kerr J.N. Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. Journal of Neuroscience 28 (2008) 2435-2446
61. Placenza F.M., Fletcher P.J., Rotzinger S., and Vaccarino F.J. Infusion of the substance P analogue, DiMe-C7, into the ventral tegmental area induces reinstatement of cocaine-seeking behaviour in rats. Psychopharmacology (Berlin) 177 (2004) 111-120
62. Redgrave P., Prescott T.J., and Gurney K. The basal ganglia: A vertebrate solution to the selection problem?. Neuroscience 89 (1999) 1009-1023
63. Redgrave P., Prescott T.J., and Gurney K. Is the short-latency dopamine response too short to signal reward error?. Trends in Neurosciences 22 (1999) 146-151
64. Regoli D., Boudon A., and Fauchere J.-L. Receptors and antagonists for substance P and related peptides. Pharmacological Reviews 45 (1994) 551-599
65. Rhodes B.J., Bullock D., Verwey W.B., Averbeck B.B., and Page M.P. Learning and production of movement sequences: Behavioral, neurophysiological, and modeling perspectives. Human Movement Science 23 (2004) 699-746
66. Schäfer M.K., Eiden L.E., and Weihe E. Cholinergic neurons and terminal fields revealed by immunohistochemistry for the vesicular acetylcholine transporter. I. Central nervous system. Neuroscience 1998 (1988) 331-359
67. Schotanus S.M., and Chergui K. Dopamine D1 receptors and group I metabotropic glutamate receptors contribute to the induction of long-term potentiation in the nucleus accumbens. Neuropharmacology 54 (2008) 837-844
68. Sjöström P.J., and Nelson S.B. Spike timing, calcium signals, and synaptic plasticity. Current Opinion in Neurobiology 12 (2002) 305-314
69. Srihasam K., Bullock D., and Grossberg S. Target selection by frontal cortex during coordinated saccadic and smooth pursuit eye movements. Journal of Cognitive Neuroscience 21 (2009) 1611-1627
70. Surmeier D.J., Mercer J.N., and Chan C.S. Autonomous pacemakers in the basal ganglia: Who needs excitatory synapses anyway?. Current Opinion in Neurobiology 15 (2005) 312-318
71. Sutton R.S., and Barto A.G. Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review 88 (1981) 135-170
72. Swanson L.W. Cerebral hemisphere regulation of motivated behavior. Brain Research 886 (2000) 113-164
73. Tan C.O., Anderson E., Dranias M., and Bullock D. Can the apparent adaptation of dopamine neurons' mismatch sensitivities be reconciled with their computation of reward prediction errors?. Neuroscience Letters 438 (2008) 14-16
74. Tan C.O., and Bullock D. A dopamine-acetylcholine cascade: Simulating learned and lesion-induced behavior of striatal cholinergic interneurons. Journal of Neurophysiology 100 (2008) 2409-2421
75. Tan C.O., and Bullock D. A local circuit model of learned striatal and dopamine cell responses under probabilistic schedules of reward. Journal of Neuroscience 28 (2008) 10062-10074
76. Tan C.O., and Bullock D. Neuropeptide co-release with GABA may explain functional non-monotonic uncertainty responses in dopamine neurons. Neuroscience Letters 430 (2008) 218-223
77. Taverna S., Ilijic E., and Surmeier D.J. Recurrent collateral connections of striatal medium spiny neurons are disrupted in models of parkinson's disease. Journal of Neuroscience 28 (2008) 5504-5512
78. Teichmann M., Guara V., Démonet J.F., Supiot F., Delliaux M., Verny C., et al. Language processing within the striatum: Evidence from a PET correlation study in Huntington's disease. Brain 131 (2008) 1046-1056
79. Tepper J.M., Koos T., and Wilson C.J. GABAergic microcircuits in the neostriatum. Trends in Neurosciences 27 (2004) 662-669
80. Tobler P.N., Fiorillo C.D., and Schultz W. Adaptive coding of reward value by dopamine neurons. Science 307 (2005) 1642-1645
81. Tunstall M.J., Oorschot D.E., Kean A., and Wickens J.R. Inhibitory interactions between spiny projection neurons in the rat striatum. Journal of Neurophysiology 88 (2002) 1263-1269
82. Turner R.S., and DeLong M.R. Corticostriatal activity in primary motor cortex of the macaque. Journal of Neuroscience 20 (2000) 7096-7108
83. Ullman M.T., Corkin S., Coppola M., Hickok G., Growdon J.H., Koroshetz W.J., et al. A neural dissociation within language: Evidence that the mental dictionary is part of declarative memory, and that grammatical rules are processed by the procedural system. Journal of Cognitive Neuroscience 9 (1997) 266-276
84. Ungless M.A., Magill P.J., and Bolam J.P. Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science 303 (2004) 2040-2042
85. Venance L., Glowinski J., and Giaume C. Electrical and chemical transmission between striatal GABAergic output neurones in rat brain slices. Journal of Physiology 559 (2004) 215-230
86. Wahl M., Marzinzik F., Friederici A.D., Hahne A., Kupsch A., Schneider G.H., et al. The human thalamus processes syntactic and semantic language violations. Neuron 59 (2008) 695-707
87. Wang J.Q., and McGinty J.F. The full D1 dopamine receptor agonist SKF-82958 induces neuropeptide mRNA in the normosensitive striatum of rats: Regulation of D1/D2 interactions by muscarinic receptors. Journal of Pharmacology and Experimental Therapeutics 281 (1997) 972-982
88. Wang Z., Kai L., Day M., Ronesi J., Yin H.H., Ding J., et al. Dopaminergic control of corticostriatal long-term synaptic depression in medium spiny neurons is mediated by cholinergic interneurons. Neuron 50 (2006) 443-452
89. Ward N. A connectionist language generator (1994), Ablex Publishing, Norwood, NJ
90. Zahm D.S., Grosu S., Irving J.C., and Williams E.A. Discrimination of striatopallidum and extended amygdala in the rat: A role for parvalbumin immunoreactive neurons?. Brain Research 978 (2003) 141-154