A local circuit model of learned striatal and dopamine cell responses under probabilistic schedules of reward

Journal of Neuroscience

Volumn 28, Issue 40, 2008, Pages 10062-10074

  Can, Ozan Tan ^a   Bullock, Daniel ^a  

^a BOSTON UNIVERSITY   (United States)

Author keywords

Computational model; Dopamine; Neuropeptide; Reinforcement learning; Substance P; Substantia nigra

Indexed keywords

4 AMINOBUTYRIC ACID; SUBSTANCE P; DOPAMINE;

ARTICLE; ASSOCIATION; BASAL GANGLION; BRAIN FUNCTION; CHOLINERGIC TRANSMISSION; CONTROLLED STUDY; CORPUS STRIATUM; DENDRITE; DOPAMINERGIC NERVE CELL; GABAERGIC TRANSMISSION; LEARNING; LOCAL CIRCUIT MODEL; MATHEMATICAL COMPUTING; PRIORITY JOURNAL; PROBABILITY; PROCESS MODEL; REWARD; SIMULATION; SUBSTANTIA NIGRA; UNCERTAINTY; ARTIFICIAL NEURAL NETWORK; COMPARATIVE STUDY; NERVE CELL NETWORK; PHYSIOLOGY; REINFORCEMENT;

CORPUS STRIATUM; DOPAMINE; LEARNING; NERVE NET; NEURAL NETWORKS (COMPUTER); REINFORCEMENT SCHEDULE; REWARD;

EID: 54049117252     PISSN: 02706474     EISSN: 02706474     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.0259-08.2008     Document Type: Article

References (81)

1. Beaujouan JC, Torrens Y, Saffroy M, Kemel M-L, Glowinski J (2004) A 25 year adventure in the field of tachykinins. Peptides 25:339-357.
2. Bernard V, Dumartin B, Lamy E, Bloch B (1993) Fos immunoreactivity after stimulation or inhibition of muscarinic receptors indicates anatomical specificity for cholinergic control of striatal efferent neurons and cortical neurons in the rat. Eur J Neurosci 5:1218-1225.
3. Betarbet R, Greenamyre JT (2004) Regulation of dopamine receptor and neuropeptide expression in the basal ganglia of monkeys treated with MPTP. Exp Neurol 189:393-403.
4. Bonanno G, Gemignani A, Schmid G, Severi P, Cavazzani P, Raiteri M (1996) Human brain somatostatin release from isolated cortical nerve endings and its modulation through GABAB receptors. Br J Pharmacol 118:1441-1446.
5. Bowery NG (1993) GABAB receptor pharmacology. Annu Rev Pharmacol Toxicol 33:109-147.
6. Brown J, Bullock D, Grossberg S (1999) How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J Neurosci 19:10502-10511.
7. Buck SH, Burks TF, Brown MR, Yamamura HI (1981) Reduction in basal ganglia and substantia nigra substance P levels in Huntington's disease. Brain Res 209:464-469.
8. Cicchetti F, Prensa L, Wu Y, Parent A (2000) Chemical anatomy of striatal interneurons in normal individuals and in patients with Huntington's disease. Brain Res Rev 34:80-101.
9. Condé H (1992) Organization and physiology of substantia nigra. Exp Brain Res 88:233-248.
10. DeLong MR, Crutcher MD, Georgopoulos AP (1983) Relations between movement and single cell discharge in the substantia nigra of the behaving monkey. J Neurosci 3:1599-1606.
11. De Martino B, Kumaran D, Seymour B, Dolan RJ (2006) Frames, biases, and rational decision-making in the human brain. Science 313:684-687.
12. de Rover M, Lodder JC, Kits KS, Schoffelmeer ANM, Brussaard AB (2002) Cholinergic modulation of nucleus accumbens medium spiny neurons. Eur J Neurosci 16:2279-2290.
13. Di Chiara G, Morelli M, Consolo S (1994) Modulatory functions of neurotransmitters in the striatum - ACh/dopamine/NMDA interactions. Trends Neurosci 17:228-233.
14. Doya K (2002) Metalearning and neuromodulation. Neural Netw 15:495-506.
15. Fiala JC, Grossberg S, Bullock D (1996) Metabotropic glutamate receptor activation in cerebellar Purkinje cells as substrate for adaptive timing of the classically conditioned eye-blink response. J Neurosci 16:3760-3774.
16. Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299:1898-1902.
17. Fiorillo CD, Tobler PN, Schultz W (2005) Evidence that the delay-period activity of dopamine neurons corresponds to reward uncertainty rather than backpropagating TD errors. Behav Brain Funct 1:7.
18. Flaherty AW, Graybiel AM (1994) Input-output organization of the sensorimotor striatum in the squirrel monkey. J Neurosci 14:599-610.
19. Floresco SB, West AR, Ash B, Moore H, Grace AA (2003) Afferent modulation of dopamine neuron firing differentially regulates tonic and phasic dopamine transmission. Nat Neurosci 6:968-973.
20. Futami T, Takakusaki K, Kitai ST (1995) Glutamatergic and cholinergic inputs from the pedunculopontine tegmental nucleus to dopamine neurons in the substantia nigra pars compacta. Neurosci Res 21:331-342.
21. Gerfen CR (1992) The neostriatal mosaic: multiple levels of compartmental organization in the basal ganglia. Annu Rev Neurosci 15:285-320.
22. Gerfen C, Wilson CJ (1996) The basal ganglia. In: Handbook of chemical neuroanatomy, Vol 12, Integrated systems of the CNS, Pt III (Swanson LW, Bjorklund A, Hokfelt T, eds), pp 371-468. Amsterdam: Elsevier.
23. Grossberg S, Bullock D, Dranias M (2008) Neural dynamics underlying impaired autonomic and conditioned responses following amygdala and orbitofrontal lesions. Behav Neurosci, in press.
24. Hanson GR, Bush L, Keefe KA, Alburges ME (2002) Distinct responses of basal ganglia substance P systems to low and high doses of methamphetamine. J Neurochem 82:1171-1178.
25. Houk J, Adams J, Barto A (1995) A model of how basal ganglia generate and use neural signals that predict reinforcement. In: Models of information processing in the basal ganglia (Houk J, Davis J, Beiser D, eds), pp 249-270. Cambridge, MA: MIT.
26. Humpel C, Saria A (1989) Effects of GABA and L-glutamic acid on the potassium-evoked in vitro release of substance P- and neurokinin A-like immunoreactivities are different in the rat striatum and substantia nigra. Neurosci Lett 105:159-163.
27. Iversen LL, Lee CM, Gilbert RF, Hunt S, Emson PC (1980) Regulation of neuropeptide release. Proc R Soc Lond B Biol Sci 210:91-111.
28. Jessell TM (1978) Substance P release from the rat substantia nigra. Brain Res 151:469-478.
29. Joel D, Weiner I (2000) The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum. Neuroscience 96:451-474.
30. Kamin L (1968) "Attention-like" processes in classical conditioning. In: Miami symposium on the prediction of behavior: aversive simulation (Jones MR, ed), pp 9-31. Miami: University of Miami.
31. Kaneko S, Hikida T, Watanabe D, Ichinose H, Nagatsu T, Kreitman RJ, Pastan I, Nakanishi S (2000) Synaptic integration mediated by striatal cholinergic interneurons in basal ganglia function. Science 289:633-637.
32. Kawagoe R, Takikawa Y, Hikosaka O (1998) Expectation of reward modulates cognitive signals in the basal ganglia. Nat Neurosci 1:411-416.
33. Khan S, Whelpton R, Michael-Titus AT (1996) Evidence for modulatory effects of substance P fragments (1- 4) and (8 -11) on endogenous outflow in rat striatal slices. Neurosci Lett 205:33-36.
34. Koós T, Tepper JM (1999) Inhibitory control of neostriatal projection neurons by GABAergic interneurons. Nat Neurosci 2:467-472.
35. Koós T, Tepper JM (2002) Dual cholinergic control of fast-spiking interneurons in the neostriatum. J Neurosci 22:529-535.
36. Kötter R (1994) Postsynaptic integration of glutamatergic and dopaminergic signals in the striatum. Prog Neurobiol 44:163-196.
37. Kubota Y, Kawaguchi Y (1993) Spatial distributions of chemically identified intrinsic neurons in relation to patch and matrix compartments of rat neostriatum. J Comp Neurol 332:499-513.
38. Lévesque M, Wallman MJ, Parent R, Sík A, Parent A (2007) Neurokinin-1 and neurokinin-3 receptors in primate substantia nigra. Neurosci Res 57:362-371.
39. Ljungberg T, Apicella P, Schultz W (1992) Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophysiol 67:145-163.
40. Malcangio M, Bowery NG (1999) Peptide autoreceptors: does an autoreceptor for substance P exist? Trends Pharmacol Sci 20:405-407.
41. Martorana A, Fusco FR, D'Angelo V, Sancesario G, Bernardi G (2003) Enkephalin, neurotensin and substance P immunoreactivity neurones of the rat GP following 6-hydroxydopamine lesion of the substantia nigra. Exp Neurol 183:311-319.
42. Mendez I, Elisevich K, Flumerfelt BA (1993) GABAergic synaptic interactions in the substantia nigra. Brain Res 617:274-284.
43. Nakahara H, Itoh H, Kawagoe R, Takikawa Y, Hikosaka O (2004) Dopamine neurons can represent context-dependent prediction error. Neuron 41:269-280.
44. Nakamura K, Ono T (1986) Lateral hypothalamus neuron involvement in integration of natural and artificial rewards and cue signals. J Neurophysiol 55:163-181.
45. Niv Y, Duff MO, Dayan P (2005) Dopamine, uncertainty and TD learning. Behav Brain Funct 1:6.
46. O'Reilly RC, Frank MJ, Hazy TE, Watz B (2007) PVLV: the primary value and learned value pavlovian learning algorithm. Behav Neurosci 121:31-49.
47. Otsuka M, Yoshioka K (1993) Neurotransmitter functions of mammalian tachykinins. Physiol Rev 73:229-308.
48. Paladini CA, Tepper JM (1999) GABA(A) and GABA(B) antagonists differentially affect the firing pattern of substantia nigra dopaminergic neurons in vivo. Synapse 32:165-176.
49. Pan W-X, Hyland BI (2005) Pedunculopontine tegmental nucleus controls conditioned responses of midbrain dopamine neurons in behaving rats. J Neurosci 25:4725-4732.
50. Pan WX, Schmidt R, Wickens JR, Hyland BI (2005) Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J Neurosci 25:6235-6242.
51. Parent A (1986) Comparative neurobiology of the basal ganglia. Wiley series in neurobiology. New York: Wiley.
52. Pasupathy A, Miller EK (2005) Different time courses of learning-related activity in the prefrontal cortex and striatum. Nature 433:873-876.
53. Prensa L, Giménez-Amaya JM, Parent A (1999) Chemical heterogeneity of the striosomal compartment in the human striatum. J Comp Neurol 413:603-618.
54. Ragsdale CW Jr, Graybiel AM (1990) A simple ordering of neocortical area established by the compartmental organization of their striatal projections. Proc Natl Acad Sci U S A 87:6196-6199.
55. Ravel S, Legallet E, Apicella P (2003) Responses of tonically active neurons in the monkey striatum discriminate between motivationally opposing stimuli. J Neurosci 23:8489-8497.
56. Reid MS, Herrera-Marschitz M, Terenius L, Ungerstedt U (1990) Intranigral substance P modulation of striatal dopamine: interaction with N-terminal and C-terminal substance P fragments. Brain Res 526:228-234.
57. Riley RC, Trafton JA, Chi SI, Basbaum AI (2001) Presynaptic regulation of spinal cord tachykinin signaling via GABA(B) but not GABA(A) receptor activation. Neuroscience 103:725-737.
58. Romo R, Schultz W (1990) Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements. J Neurophysiol 63:592-606.
59. Salio C, Lossi L, Ferrini F, Merighi A (2005) Ultrastructural evidence for a pre- and postsynaptic localization of full-length trkB receptors in substantia gelatinosa (lamina II) of rat and mouse spinal cord. Eur J Neurosci 22:1951-1966.
60. Salio C, Lossi L, Ferrini F, Merighi A (2006) Neuropeptides as synaptic transmitters. Cell Tissue Res 326:583-598.
61. Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of actionspecific reward values in the striatum. Science 310:1337-1340.
62. Scarnati E, Hajdu F, Pacitti C, Tömböl T (1988) An EM and Golgi study on the connection between the nucleus tegmenti pedunculopontinus and the pars compacta of the substantia nigra in the rat. J Hirnforsch 29:95-105.
63. Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80:1-27.
64. Schultz W (2004) Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioral ecology. Curr Opin Neurobiol 14:139-147.
65. Schultz W, Apicella P, Scarnati E, Ljungberg T (1992) Neuronal activity in monkey ventral striatum related to the expectation of reward. J Neurosci 12:4595-4610.
66. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593-1599.
67. Seamans JK, Yang CR (2004) The principal features and mechanisms of dopamine modulation in the prefrontal cortex. Prog Neurobiol 74:1-58.
68. Semba K, Fibiger HC (1992) Afferent connections of the laterodorsal and the pedunculopontine tegmental nuclei in the rat: a retro- and anterograde transport and immunohistological study. J Comp Neurol 323:387-410.
69. Smith CUM 1996 Elements of molecular neurobiology, Ed 2. New York: Wiley.
70. Tan CO, Anderson E, Dranias M, Bullock D (2008) Can the apparent adaptation of dopamine neurons' mismatch sensitivities be reconciled with their computation of reward prediction errors? Neurosci Lett 438:14-16.
71. Tan CO, Bullock D (2008a) Neuropeptide co-release with GABA may explain functional non-monotonic uncertainty responses in dopamine neurons. Neurosci Lett 430:218-223.
72. Tan CO, Bullock D (2008b) A dopamine-acetylcholine cascade: simulating learned and lesion-induced behavior of striatal cholinergic interneurons. J Neurophysiol, in press.
73. Teoh H, Malcangio M, Bowery NG (1996) GABA, glutamate and substance P-like immunoreactivity release: effects of novel GABAB antagonists. Br J Pharmacol 118:1153-1160.
74. Tepper JM, Lee CR (2007) GABAergic control of substantia nigra dopaminergic neurons. Prog Brain Res 160:189-208.
75. Tobler PN, Fiorillo CD, Schultz W (2005) Adaptive coding of reward value by dopamine neurons. Science 307:1642-1645.
76. Waelti P, Dickinson A, Schultz W (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature 412:43-48.
77. Watanabe K, Lauwereyns J, Hikosaka O (2003) Neural correlates of rewarded and unrewarded eye movements in the primate caudate nucleus. J Neurosci 23:10052-10057.
78. Whitty CJ, Paul MA, Bannon MJ (1997) Neurokinin receptor mRNA localization in human midbrain dopamine neurons. J Comp Neurol 382:394-400.
79. Wickens JR, Begg AJ, Arbuthnott GW (1996) Dopamine reverses the depression of rat corticostriatal synapses which normally follows highfrequency stimulation of cortex in vitro. Neuroscience 70:1-5.
80. Wörgötter F, Porr B (2005) Temporal sequence learning, prediction, and control: a review of different models and their relation to biological mechanisms. Neural Comput 17:245-319.
81. Yang C, Mogenson G (1987) Hippocampal signal transmission to the pedunculopontine nucleus and its regulation by dopamine D2 receptors in the nucleus accumbens: an electrophysiological and behavioural study. Neuroscience 23:1041-1055.