A New Framework for Cortico-Striatal Plasticity: Behavioural Theory Meets In Vitro Data at the Reinforcement-Action Interface

PLoS Biology

Volumn 13, Issue 1, 2015, Pages

  Gurney, Kevin N ^a,b   Humphries, Mark D ^c   Redgrave, Peter ^a  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

^b CISTIB Centre for Computational Imaging and Simulation Technologies in Biomedicine INSIGNEO Institute for in silico Medicine Department of Mechanical Engineering The University of Sheffield   (United Kingdom)

^c UNIVERSITY OF MANCHESTER   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE; DOPAMINE RECEPTOR;

ARTICLE; BEHAVIORAL SCIENCE; CONCEPTUAL FRAMEWORK; CONTROLLED STUDY; CORRELATIONAL STUDY; CORTICOSTRIATAL TRACT; DOPAMINE BRAIN LEVEL; IN VITRO STUDY; LEARNING; MATHEMATICAL MODEL; NERVE CELL NETWORK; NERVE CELL PLASTICITY; NERVE TRACT; OPERANT BEHAVIORAL TEST; REINFORCEMENT; SIGNAL DETECTION; SPIKE WAVE; SYNAPTIC TRANSMISSION; TASK PERFORMANCE; THEORETICAL MODEL; VALIDATION STUDY; ANIMAL; BIOLOGICAL MODEL; BRAIN CORTEX; CORPUS STRIATUM; HUMAN; PHYSIOLOGY; PSYCHOLOGICAL MODEL;

ANIMALS; CEREBRAL CORTEX; CORPUS STRIATUM; HUMANS; MODELS, NEUROLOGICAL; MODELS, PSYCHOLOGICAL; NEURONAL PLASTICITY; REINFORCEMENT (PSYCHOLOGY);

EID: 84922228056     PISSN: 15449173     EISSN: 15457885     Source Type: Journal    
DOI: 10.1371/journal.pbio.1002034     Document Type: Article

References (114)

1. Mogenson GJ, Jones DL, Yim CY, (1980) From motivation to action: functional interface between the limbic system and the motor system. Progress in Neurobiology 14: 69–97.
2. Schultz W, Dayan P, Montague P, (1997) A neural substrate of prediction and reward. Science 275: 1593–1599.
3. Bayer HM, Glimcher PW, (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47: 129–141.
4. Redgrave P, Gurney K, (2006) The short-latency dopamine signal: a role in discovering novel actions? Nat Rev Neurosci 7: 967–975.
5. Tsai HC, Zhang F, Adamantidis A, Stuber GD, Bonci A, et al. (2009) Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 324: 1080–1084.
6. Centonze D, Picconi B, Gubellini P, Bernardi G, Calabresi P, (2001) Dopaminergic control of synaptic plasticity in the dorsal striatum. Eur J Neurosci 13: 1071–1077.
7. Reynolds JN, Hyland BI, Wickens JR, (2001) A cellular mechanism of reward-related learning. Nature 413: 67–70.
8. Calabresi P, Picconi B, Tozzi A, Di FM, (2007) Dopamine-mediated regulation of corticostriatal synaptic plasticity. Trends Neurosci 30: 211–219.
9. Yin HH, Knowlton BJ, (2006) The role of the basal ganglia in habit formation. Nat Rev Neurosci 7: 464–476.
10. Wickens JR, Budd CS, Hyland BI, Arbuthnott GW, (2007) Striatal contributions to reward and decision making: making sense of regional variations in a reiterated processing matrix. Ann N Y Acad Sci 1104: 192–212.
11. Khamassi M, Humphries MD, (2012) Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies. Front Behav Neurosci 6: 79.
12. Mink J, Thach W, (1993) Basal ganglia intrinsic circuits and their role in behavior. Curr Opin Neurobiol 3: 950–957.
13. Redgrave P, Prescott T, Gurney K, (1999) The basal ganglia: a vertebrate solution to the selection problem? Neuroscience 89: 1009–1023.
14. Hikosaka O, Takikawa Y, Kawagoe R, (2000) Role of the basal ganglia in the control of purposive saccadic eye movements. Physiol Rev 80: 953–978.
15. Reynolds JNJ, Wickens JR, (2002) Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks 15: 507–521.
16. Frank M, (2005) Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated parkinsonism. J Cogn Neurosci 17: 51–72.
17. Shen W, Flajolet M, Greengard P, Surmeier DJ, (2008) Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321: 848–851.
18. Alexander GE, Crutcher MD, (1990) Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci 13: 266–272.
19. Kravitz AV, Freeze BS, Parker PR, Kay K, Thwin MT, et al. (2010) Regulation of parkinsonian motor behaviours by optogenetic control of basal ganglia circuitry. Nature 466: 622–626.
20. Freeze BS, Kravitz AV, Hammack N, Berke JD, Kreitzer AC, (2013) Control of basal ganglia output by direct and indirect pathway projection neurons. J Neurosci 33: 18531–18539.
21. Fino E, Glowinski J, Venance L, (2005) Bidirectional activity-dependent plasticity at corticostriatal synapses. J Neurosci 25: 11279–11287.
22. Pawlak V, Kerr JN, (2008) Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J Neurosci 28: 24–35.
23. Houk J, Adams J, Barto A (1995) A model of how the basal ganglia generate and use neural signals that predict reinforcement. Houk JC, Davis JL, Beiser DG, editors. Models of information processing in the basal ganglia. Cambridge: MIT Press. pp. 249–270.
24. Berke JD, Hyman SE, (2000) Addiction, dopamine, and the molecular mechanisms of memory. Neuron 25: 515–532.
25. Cui G, Jun SB, Jin X, Pham MD, Vogel SS, et al. (2013) Concurrent activation of striatal direct and indirect pathways during action initiation. Nature 494: 238–242.
26. Barnes TD, Kubota Y, Hu D, Jin DZ, Graybiel AM, (2005) Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories. Nature 437: 1158–1161.
27. Tang C, Pawlak AP, Prokopenko V, West MO, (2007) Changes in activity of the striatum during formation of a motor habit. Eur J Neurosci 25: 1212–1227.
28. Kimchi EY, Laubach M, (2009) The dorsomedial striatum reflects response bias during learning. J Neurosci 29: 14891–14902.
29. Kimchi EY, Torregrossa MM, Taylor JR, Laubach M, (2009) Neuronal correlates of instrumental learning in the dorsal striatum. J Neurophysiol 102: 475–489.
30. Thorn CA, Atallah H, Howe M, Graybiel AM, (2010) Differential dynamics of activity changes in dorsolateral and dorsomedial striatal loops during learning. Neuron 66: 781–795.
31. Thorn CA, Graybiel AM, (2014) Differential entrainment and learning-related dynamics of spike and local field potential activity in the sensorimotor and associative striatum. J Neurosci 34: 2845–2859.
32. Kreitzer A, Berke J, (2011) Investigating striatal function through cell-type-specific manipulations. Neuroscience 198: 19–26.
33. Kravitz AV, Tye LD, Kreitzer AC, (2012) Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat Neurosci 15: 816–818.
34. Gurney K, Prescott TJ, Redgrave P, (2001) A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol Cybern 84: 401–410.
35. Gurney K, Prescott TJ, Redgrave P, (2001) A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour. Biol Cybern 84: 411–423.
36. Gurney K, Lepora N, Shah A, Koene A, Redgrave P (2013) Action discovery and intrinsic motivation: a biologically constrained formalisation. Baldassarre G, Mirolli M, editors. Intrinsically motivated learning in natural and artificial systems. Berlin; Heidelberg: Springer. pp. 151–181.
37. Hart AS, Rutledge RB, Glimcher PW, Phillips PEM, (2014) Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. J Neurosci 34: 698–704.
38. Lak A, Stauffer WR, Schultz W, (2014) Dopamine prediction error responses integrate subjective value from different reward dimensions. Proc Natl Acad Sci U S A 111: 2343–2348.
39. Schultz W, (2010) Dopamine signals for reward value and risk: basic and recent data. Behav Brain Funct 6: 24.
40. Bayer HM, Lau B, Glimcher PW, (2007) Statistics of midbrain dopamine neuron spike trains in the awake primate. J Neurophysiol 98: 1428–1439.
41. Pan WX, Brown J, Dudman JT, (2013) Neural signals of extinction in the inhibitory microcircuit of the ventral midbrain. Nat Neurosci 16: 71–78.
42. van Rossum MCW, Bi GQ, Turrigiano GG, (2000) Stable hebbian learning from spike timing-dependent plasticity. J Neurosci 20: 8812–8821.
43. Elsner B, Hommel B, (2004) Contiguity and contingency in action-effect learning. Psychological Research 68: 138–154.
44. Izhikevich EM, (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17: 2443–2452.
45. Humphries M, Lepora N, Wood R, Gurney K, (2009) Capturing dopaminergic modulation and bimodal membrane behaviour of striatal medium spiny neurons in accurate, reduced models. Front Comput Neurosci 3: 26.
46. Bouton ME, (2004) Context and behavioral processes in extinction. Learn Memory 11: 485–494.
47. Bouton ME, Todd TP, Vurbic D, Winterbauer NE, (2011) Renewal after the extinction of free operant behavior. Learn Behav 39: 57–67.
48. Crombag HS, Shaham Y, (2002) Renewal of drug seeking by contextual cues after prolonged extinction in rats. Behav Neurosci 116: 169.
49. Nakajima S, Tanaka S, Urushihara K, Imada H, (2000) Renewal of extinguished lever-press responses upon return to the training context. Learn Motiv 31: 416–431.
50. Todd TP, Winterbauer NE, Bouton ME, (2012) Contextual control of appetite. renewal of inhibited food-seeking behavior in sated rats after extinction. Appetite 58: 484–489.
51. Frank MJ, (2005) Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated parkinsonism. J Cogn Neurosci 17: 51–72.
52. Tecuapetla F, Matias S, Dugue GP, Mainen ZF, Costa RM, (2014) Balanced activity in basal ganglia projection pathways is critical for contraversive movements. Nat Commun 5: 4315.
53. Paille V, Fino E, Du K, Morera-Herreras T, Perez S, et al. (2013) GABAergic circuits control spike-timing-dependent plasticity. J Neurosci 33: 9353–9363.
54. Wang Z, Kai L, Day M, Ronesi J, Yin HH, et al. (2006) Dopaminergic control of corticostriatal long-term synaptic depression in medium spiny neurons is mediated by cholinergic interneurons. Neuron 50: 443–452.
55. Threlfell S, Lalic T, Platt NJ, Jennings K, Katie A, et al. (2012) Striatal dopamine release is triggered by synchronized activity in cholinergic interneurons. Neuron 75: 58–64.
56. Lindskog M, Kim M, Wikstrm MA, Blackwell KT, Kotaleski JH, (2006) Transient calcium and dopamine increase PKA activity and DARPP-32 phosphorylation. PLoS Comput Biol 2: e119.
57. Nakano T, Doi T, Yoshimoto J, Doya K, (2010) A kinetic model of dopamine- and calcium-dependent striatal synaptic plasticity. PLoS Comput Biol 6: e1000670.
58. Evans RC, Maniar YM, Blackwell KT, (2013) Dynamic modulation of spike timing-dependent calcium influx during corticostriatal upstates. J Neurophysiol 110: 1631–1645.
59. Kim B, Hawes SL, Gillani F, Wallace LJ, Blackwell KT, (2013) Signaling pathways involved in striatal synaptic plasticity are sensitive to temporal pattern and exhibit spatial specificity. PLoS Comput Biol 9: e1002953.
60. Redgrave P, Gurney K, Reynolds J, (2008) What is reinforced by phasic dopamine signals? Brain Res Rev 58: 322–339.
61. Bolado-Gomez R, Gurney K, (2013) A biologically plausible embodied model of action discovery. Frontiers in Neurorobotics 7: 4.
62. Samson RD, Frank MJ, Fellous JM, (2010) Computational models of reinforcement learning: the role of dopamine as a reward signal. Cognitive neurodynamics 4: 91–105.
63. Sutton R, Barto A (1998) Reinforcement learning: an introduction. Cambridge: MIT Press.
64. Montague PR, Dayan P, Sejnowski TJ, (1996) A framework for mesencephalic dopamine systems based on predictive hebbian learning. J Neurosci 16: 1936–1947.
65. Worgotter F, Porr B, (2005) Temporal sequence learning, prediction, and control: a review of different models and their relation to biological mechanisms. Neural Comput 17: 245–319.
66. Bienenstock EL, Cooper LN, Munro PW, (1982) Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. J Neurosci 2: 32–48.
67. Cooper LN, Intrator N, Blais SB, Shouval ZH (2004) Theory of cortical plasticity. London: World Scientific Publishing.
68. Pfister JP, Gerstner W, (2006) Triplets of spikes in a model of spike timing-dependent plasticity. J Neurosci 26: 9673–9682.
69. Dickinson A, (1985) Actions and habits: the development of behavioural autonomy. Phil Trans R Soc Lond B 308: 67–78.
70. Shiflett MW, Balleine BW, (2011) Molecular substrates of action control in cortico-striatal circuits. Prog Neurobiol 95: 1–13.
71. Daw ND, Niv Y, Dayan P, (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8: 1704–1711.
72. Bornstein AM, Daw ND, (2011) Multiplicity of control in the basal ganglia: computational roles of striatal subregions. Curr Opin Neurobiol 21: 374–380.
73. Yin HH, Knowlton BJ, Balleine BW, (2004) Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur J Neurosci 19: 181–189.
74. De Leonibus E, Costantini VJA, Massaro A, Mandolesi G, Vanni V, et al. (2011) Cognitive and neural determinants of response strategy in the dual-solution plus-maze task. Learn Mem 18: 241–244.
75. Gremel CM, Costa RM, (2013) Orbitofrontal and striatal circuits dynamically encode the shift between goal-directed and habitual actions. Nat Commun 4: 2264.
76. Faure A, Haberland U, Cond F, Massioui NE, (2005) Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation. J Neurosci 25: 2771–2780.
77. Yin HH, Ostlund SB, Knowlton BJ, Balleine BW, (2005) The role of the dorsomedial striatum in instrumental conditioning. Eur J Neurosci 22: 513–523.
78. Shiflett MW, Brown RA, Balleine BW, (2010) Acquisition and performance of goal-directed instrumental actions depends on ERK signaling in distinct regions of dorsal striatum in rats. J Neurosci 30: 2951–2959.
79. Shan Q, Ge M, Christie MJ, Balleine BW, (2014) The acquisition of goal-directed actions generates opposing plasticity in direct and indirect pathways in dorsomedial striatum. J Neurosci 34: 9196–9201.
80. Humphries MD, Prescott TJ, (2010) The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward. Prog Neurobiol 90: 385–417.
81. Joel D, Weiner I, (2000) The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum. Neuroscience 96: 451–474.
82. McGeorge AJ, Faull RL, (1989) The organization of the projection from the cerebral cortex to the striatum in the rat. Neuroscience 29: 503–537.
83. Schilman EA, Uylings HBM, Galis-de Graaf Y, Joel D, Groenewegen HJ, (2008) The orbital cortex in rats topographically projects to central parts of the caudate-putamen complex. Neurosci Lett 432: 40–45.
84. Schoenbaum G, Roesch MR, Stalnaker TA, Takahashi YK, (2009) A new perspective on the role of the orbitofrontal cortex in adaptive behaviour. Nat Rev Neurosci 10: 885–892.
85. Sul JH, Kim H, Huh N, Lee D, Jung MW, (2010) Distinct roles of rodent orbitofrontal and medial prefrontal cortex in decision making. Neuron 66: 449–460.
86. Ostlund SB, Balline BW, (2007) Selective reinstatement of instrumental performance depends on the discriminative stimulus properties of the mediating outcome. Learn Behav 35: 43–52.
87. Maurin Y, Banrezes B, Menetrey A, Mailly P, Deniau JM, (1999) Three-dimensional distribution of nigrostriatal neurons in the rat: relation to the topography of striatonigral projections. Neuroscience 91: 891–909.
88. Haber SN, Fudge JL, McFarland NR, (2000) Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J Neurosci 20: 2369–2382.
89. Wang LP, Li F, Wang D, Xie K, Wang D, et al. (2011) NMDA receptors in dopaminergic neurons are crucial for habit learning. Neuron 72: 1055–1066.
90. Ilango A, Kesner AJ, Keller KL, Stuber GD, Bonci A, et al. (2014) Similar roles of substantia nigra and ventral tegmental dopamine neurons in reward and aversion. J Neurosci 34: 817–822.
91. Matsumoto M, Hikosaka O, (2009) Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459: 837–841.
92. Yin HH, Mulcare SP, Hilrio MRF, Clouse E, Holloway T, et al. (2009) Dynamic reorganization of striatal circuits during the acquisition and consolidation of a skill. Nat Neurosci 12: 333–341.
93. Smith R, Musleh W, Akopian G, Buckwalter G, Walsh JP, (2001) Regional differences in the expression of corticostriatal synaptic plasticity. Neuroscience 106: 95–101.
94. Redish AD, Jensen S, Johnson A, Kurth-Nelson Z, (2007) Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol Rev 114: 784.
95. Humphries MD, Stewart RD, Gurney KN, (2006) A physiologically plausible model of action selection and oscillatory activity in the basal ganglia. J Neurosci 26: 12921–12942.
96. Alexander GE, DeLong MR, Strick PL, (1986) Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu Rev Neurosci 9: 357–381.
97. Middleton FA, Strick PL, (2000) Basal ganglia and cerebellar loops: motor and cognitive circuits. Brain Res Brain Res Rev 31: 236–250.
98. Romanelli P, Esposito V, Schaal DW, Heit G, (2005) Somatotopy in the basal ganglia: experimental and clinical evidence for segregated sensorimotor channels. Brain Res Brain Res Rev 48: 112–128.
99. Alexander GE, DeLong MR, (1985) Microstimulation of the primate neostriatum. II. Somatotopic organization of striatal microexcitable zones and their relation to neuronal response properties. J Neurophysiol 53: 1417–1430.
100. Brown LL, Sharp FR, (1995) Metabolic mapping of rat striatum: somatotopic organization of sensorimotor activity. Brain Res 686: 207–222.
101. Fan D, Rossi MA, Yin HH, (2012) Mechanisms of action selection and timing in substantia nigra neurons. J Neurosci 32: 5534–5548.
102. Fee MS, (2012) Oculomotor learning revisited: a model of reinforcement learning in the basal ganglia incorporating an efference copy of motor actions. Front Neural Circuits 6: 38.
103. Znamenskiy P, Zador AM, (2013) Corticostriatal neurons in auditory cortex drive decisions during auditory discrimination. Nature 497: 482–485.
104. Surmeier DJ, Ding J, Day M, Wang Z, Shen W, (2007) D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends Neurosci 30: 228–235.
105. Moyer JT, Wolf JA, Finkel LH, (2007) Effects of dopaminergic modulation on the integrative properties of the ventral striatal medium spiny neuron. J Neurophysiol 98: 3731–3748.
106. Izhikevich E, (2003) Simple model of spiking neurons. IEEE Trans Neural Netw 14: 1569–1572.
107. Izhikevich EM (2007) Dynamical systems in neuroscience. Cambridge: MIT Press.
108. Jahr CE, Stevens CF, (1990) Voltage dependence of NMDA-activated macroscopic conductances predicted by single-channel kinetics. J Neurosci 10: 3178–3182.
109. Bauswein E, Fromm C, Preuss A, (1989) Corticostriatal cells in comparison with pyramidal tract neurons: contrasting properties in the behaving monkey. Brain Res 493: 198–203.
110. Shinomoto S, Shima K, Tanji J, (2003) Differences in spiking patterns among cortical neurons. Neural Computation 15: 2823–2842.
111. Ponzi A, Wickens JR, (2013) Optimal balance of the striatal medium spiny neuron network. PLoS Comput Biol 9: e1002954.
112. Cragg S, Rice M, (2004) DAncing past the DAT at a DA synapse. Trends Neurosci 27: 270–277.
113. Abbott L, Nelson S, (2000) Synaptic plasticity: taming the beast. Nat Neurosci 3 Suppl:1178–1183.
114. Mahon S, Deniau JM, Charpier S, Delord B, (2000) Role of a striatal slowly inactivating potassium current in short-term facilitation of corticostriatal inputs: a computer simulation study. Learn Mem 7: 357–362.