Striatal dopamine ramping may indicate flexible reinforcement learning with forgetting in the cortico-basal ganglia circuits

Frontiers in Neural Circuits

Volumn 8, Issue APR, 2014, Pages

  Morita, Kenji ^a   Kato, Ayaka ^a  

^a UNIVERSITY OF TOKYO   (Japan)

Author keywords

Basal ganglia; Computational modeling; Corticostriatal; Dopamine; Flexibility; Reinforcement learning; Reward prediction error; Synaptic plasticity

Indexed keywords

DOPAMINE;

ARTICLE; BASAL GANGLION; CONNECTOME; CORTICO BASAL GANGLION; DOPAMINE RELEASE; MATHEMATICAL MODEL; NERVE CELL PLASTICITY; REINFORCEMENT; REWARD; SIGNAL TRANSDUCTION; SIMULATION; ANIMAL; BIOLOGICAL MODEL; BRAIN CORTEX; COMPUTATIONAL MODELING; COMPUTER SIMULATION; CORPUS STRIATUM; CORTICOSTRIATAL; DECISION MAKING; FLEXIBILITY; METABOLISM; NERVE CELL; NERVE TRACT; PHYSIOLOGY; REINFORCEMENT LEARNING; REWARD PREDICTION ERROR;

BASAL GANGLIA; COMPUTATIONAL MODELING; CORTICOSTRIATAL; DOPAMINE; FLEXIBILITY; REINFORCEMENT LEARNING; REWARD PREDICTION ERROR; SYNAPTIC PLASTICITY;

ANIMALS; BASAL GANGLIA; CEREBRAL CORTEX; CHOICE BEHAVIOR; COMPUTER SIMULATION; CORPUS STRIATUM; DOPAMINE; MODELS, NEUROLOGICAL; NEURAL PATHWAYS; NEURONS; REINFORCEMENT (PSYCHOLOGY);

EID: 84898426520     PISSN: 16625110     EISSN: None     Source Type: Journal    
DOI: 10.3389/fncir.2014.00036     Document Type: Article

References (49)

1. Bayer, H. M., and Glimcher, P. W. (2005). Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129-141. doi: 10.1016/j.neuron.2005.05.020
2. Bayer, H. M., Lau, B., and Glimcher, P. W. (2007). Statistics of midbrain dopamine neuron spike trains in the awake primate. J. Neurophysiol. 98, 1428-1439. doi: 10.1152/jn.01140.2006
3. Björklund, A., and Dunnett, S. B. (2007). Dopamine neuron systems in the brain: an update. Trends Neurosci. 30, 194-202. doi: 10.1016/j.tins.2007.03.006
4. Boeijinga, P. H., Mulder, A. B., Pennartz, C. M., Manshanden, I., and Lopes da Silva, F. H. (1993). Responses of the nucleus accumbens following fornix/fimbria stimulation in the rat. Identification and long-term potentiation of mono-and polysynaptic pathways. Neuroscience 53, 1049-1058. doi: 10.1016/0306-4522(93)90488-2
5. Bolam, J. P., and Pissadaki, E. K. (2012). Living on the edge with too many mouths to feed: why dopamine neurons die. Mov. Disord. 27, 1478-1483. doi: 10.1002/mds.25135
6. Bromberg-Martin, E. S., Matsumoto, M., and Hikosaka, O. (2010). Dopamine in motivational control: rewarding, aversive, and alerting. Neuron 68, 815-834. doi: 10.1016/j.neuron.2010.11.022
7. Calabresi, P., Maj, R., Pisani, A., Mercuri, N. B., and Bernardi, G. (1992). Long-term synaptic depression in the striatum: physiological and pharmacological characterization. J. Neurosci. 12, 4224-4233.
8. Doya, K. (2000). Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr. Opin. Neurobiol. 10, 732-739. doi: 10.1016/S0959-4388(00)00153-7
9. Fiorillo, C. D., Tobler, P. N., and Schultz, W. (2003). Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898-1902. doi: 10.1126/science.1077349
10. Gerfen, C. R., and Surmeier, D. J. (2011). Modulation of striatal projection systems by dopamine. Annu. Rev. Neurosci. 34, 441-466. doi: 10.1146/annurev-neuro-061010-113641
11. Gershman, S. J. (2014). Dopamine ramps are a consequence of reward prediction errors. Neural Comput. 26, 467-471. doi: 10.1162/NECO_a_00559
12. Glimcher, P. W. (2011). Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc. Natl. Acad. Sci. U.S.A. 108(Suppl. 3), 15647-15654. doi: 10.1073/pnas.1014269108
13. Gustafsson, B., Asztely, F., Hanse, E., and Wigström, H. (1989). Onset characteristics of long-term potentiation in the Guinea-Pig hippocampal CA1 region in vitro. Eur. J. Neurosci. 1, 382-394. doi: 10.1111/j.1460-9568.1989.tb00803.x
14. Hardt, O., Nader, K., and Nadel, L. (2013). Decay happens: the role of active forgetting in memory. Trends Cogn. Sci. 17, 111-120. doi: 10.1016/j.tics.2013.01.001
15. Hardt, O., Nader, K., and Wang, Y. T. (2014). GluA2-dependent AMPA receptor endocytosis and the decay of early and late long-term potentiation: possible mechanisms for forgetting of short-and long-term memories. Philos. Trans. R. Soc. Lond. B Biol. Sci. 369:20130141. doi: 10.1098/rstb.2013.0141
16. Hart, A. S., Rutledge, R. B., Glimcher, P. W., and Phillips, P. E. (2014). Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. J. Neurosci. 34, 698-704. doi: 10.1523/JNEUROSCI.2489-13.2014
17. Howe, M. W., Tierney, P. L., Sandberg, S. G., Phillips, P. E., and Graybiel, A. M. (2013). Prolonged dopamine signalling in striatum signals proximity and value of distant rewards. Nature 500, 575-579. doi: 10.1038/nature12475. Available online at:http://www.nature.com/nature/journal/v500/n7464/full/nature12475.html
18. Ito, M., and Doya, K. (2011). Multiple representations and algorithms for reinforcement learning in the cortico-basal ganglia circuit. Curr. Opin. Neurobiol. 21, 368-373. doi: 10.1016/j.conb.2011.04.001
19. Kawagoe, R., Takikawa, Y., and Hikosaka, O. (2004). Reward-predicting activity of dopamine and caudate neurons-a possible mechanism of motivational control of saccadic eye movement. J. Neurophysiol. 91, 1013-1024. doi: 10.1152/jn.00721.2003
20. Laughlin, S. (2001). Energy as a constraint on the coding and processing of sensory information. Curr. Opin. Neurobiol. 11, 475-480. doi: 10.1016/S0959-4388(00)00237-3
21. Matsuzaki, M., Honkura, N., Ellis-Davies, G., and Kasai, H. (2004). Structural basis of long-term potentiation in single dendritic spines. Nature 429, 761-766. doi: 10.1038/nature02617
22. Montague, P. R., Dayan, P., and Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936-1947.
23. Montague, P. R., Hyman, S. E., and Cohen, J. D. (2004). Computational roles for dopamine in behavioural control. Nature 431, 760-767. doi: 10.1038/nature03015
24. Morita, K. (2014). Differential cortical activation of the striatal direct and indirect-pathway cells: reconciling the anatomical and optogenetic results by a computational method. J. Neurophysiol. doi: 10.1152/jn.00625.2013. [Epub ahead of print].
25. Morita, K., Morishima, M., Sakai, K., and Kawaguchi, Y. (2012). Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways. Trends Neurosci. 35, 457-467. doi: 10.1016/j.tins.2012.04.009
26. Morita, K., Morishima, M., Sakai, K., and Kawaguchi, Y. (2013). Dopaminergic control of motivation and reinforcement learning: a closed-circuit account for reward-oriented behavior. J. Neurosci. 33, 8866-8890. doi: 10.1523/JNEUROSCI.4614-12.2013
27. Morris, G., Nevet, A., Arkadir, D., Vaadia, E., and Bergman, H. (2006). Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9, 1057-1063. doi: 10.1038/nn1743
28. Niv, Y. (2013). Neuroscience: dopamine ramps up. Nature 500, 533-535. doi: 10.1038/500533a
29. Niv, Y., Daw, N. D., and Dayan, P. (2006). Choice values. Nat. Neurosci. 9, 987-988. doi: 10.1038/nn0806-987
30. Niv, Y., Daw, N. D., Joel, D., and Dayan, P. (2007). Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl.) 191, 507-520. doi: 10.1007/s00213-006-0502-4
31. Niv, Y., Duff, M. O., and Dayan, P. (2005). Dopamine, uncertainty and TD learning. Behav. Brain Funct. 1:6. doi: 10.1186/1744-9081-1-6
32. O'Doherty, J. P., Hampton, A., and Kim, H. (2007). Model-based fMRI and its application to reward learning and decision making. Ann. N.Y. Acad. Sci. 1104, 35-53. doi: 10.1196/annals.1390.022
33. Pennartz, C. M., Ito, R., Verschure, P. F., Battaglia, F. P., and Robbins, T. W. (2011). The hippocampal-striatal axis in learning, prediction and goal-directed behavior. Trends Neurosci. 34, 548-559. doi: 10.1016/j.tins.2011.08.001
34. Pissadaki, E. K., and Bolam, J. P. (2013). The energy cost of action potential propagation in dopamine neurons: clues to susceptibility in Parkinson's disease. Front. Comput. Neurosci. 7:13. doi: 10.3389/fncom.2013.00013
35. Potjans, W., Diesmann, M., and Morrison, A. (2011). An imperfect dopaminergic error signal can drive temporal-difference learning. PLoS Comput. Biol. 7:e1001133. doi: 10.1371/journal.pcbi.1001133
36. Rangel, A., Camerer, C., and Montague, P. R. (2008). A framework for studying the neurobiology of value-based decision making. Nat. Rev. Neurosci. 9, 545-556. doi: 10.1038/nrn2357
37. Reynolds, J. N., Hyland, B. I., and Wickens, J. R. (2001). A cellular mechanism of reward-related learning. Nature413, 67-70. doi: 10.1038/35092560
38. Roesch, M. R., Calu, D. J., and Schoenbaum, G. (2007). Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 10, 1615-1624. doi: 10.1038/nn2013
39. Rummery, G. A., and Niranjan, M. (1994). "On-line Q-learning using connectionist systems, " in Technical Report CUED/F-INFENG/TR 166 (Cambridge: Cambridge University Engineering Department).
40. Samejima, K., Ueda, Y., Doya, K., and Kimura, M. (2005). Representation of action-specific reward values in the striatum. Science 310, 1337-1340. doi: 10.1126/science.1115270
41. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593-1599. doi: 10.1126/science.275.5306.1593
42. Shen, W., Flajolet, M., Greengard, P., and Surmeier, D. J. (2008). Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321, 848-851. doi: 10.1126/science.1160575
43. Steinberg, E. E., Keiflin, R., Boivin, J. R., Witten, I. B., Deisseroth, K., and Janak, P. H. (2013). A causal link between prediction errors, dopamine neurons and learning. Nat. Neurosci. 16, 966-973. doi: 10.1038/nn.3413
44. Sutton, R., and Barto, A. (1998). Reinforcement Learning. Cambridge, MA: MIT Press.
45. Threlfell, S., Lalic, T., Platt, N. J., Jennings, K. A., Deisseroth, K., and Cragg, S. J. (2012). Striatal dopamine release is triggered by synchronized activity in cholinergic interneurons. Neuron 75, 58-64. doi: 10.1016/j.neuron.2012.04.038
46. Ungerstedt, U. (1971). Stereotaxic mapping of the monoamine pathways in the rat brain. Acta Physiol. Scand. Suppl. 367, 1-48.
47. Watabe-Uchida, M., Zhu, L., Ogawa, S. K., Vamanrao, A., and Uchida, N. (2012). Whole-brain mapping of direct inputs to midbrain dopamine neurons. Neuron 74, 858-873. doi: 10.1016/j.neuron.2012.03.017
48. Watkins, C. (1989). Learning from Delayed Rewards. thesis, University of Cambridge, Cambridge.
49. Xiao, M. Y., Niu, Y. P., and Wigström, H. (1996). Activity-dependent decay of early LTP revealed by dual EPSP recording in hippocampal slices from young rats. Eur. J. Neurosci. 8, 1916-1923. doi: 10.1111/j.1460-9568.1996.tb01335.x