An Imperfect Dopaminergic Error Signal Can Drive Temporal-Difference Learning

PLoS Computational Biology

Volumn 7, Issue 5, 2011, Pages

  Potjans, Wiebke ^a,b,c   Diesmann, Markus ^a,b,c,d   Morrison, Abigail ^b,c  

^a FORSCHUNGSZENTRUM JÜLICH   (Germany)

^b RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

^c UNIVERSITY OF FREIBURG   (Germany)

^d RIKEN   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

AMINES; BEHAVIORAL RESEARCH; BIOINFORMATICS; BRAIN; ERRORS; LEARNING ALGORITHMS; MAMMALS; NEURAL NETWORKS;

COMPUTATIONAL NEUROSCIENCE; DISCRETE TIME; DOPAMINE; ERROR SIGNAL; NEURONAL NETWORKS; PLASTICITY DYNAMICS; SYNAPTIC PLASTICITY; TEMPORAL DIFFERENCE ERRORS; TEMPORAL DIFFERENCE LEARNING; TEMPORAL-DIFFERENCE ALGORITHM;

NEURONS;

DOPAMINE;

ARTICLE; CONTROLLED STUDY; DOPAMINERGIC SYSTEM; EXPERIMENTAL STUDY; LEARNING; LEARNING ALGORITHM; MOLECULAR DYNAMICS; MOLECULAR MODEL; NERVE CELL NETWORK; NERVE CELL PLASTICITY; POSTSYNAPTIC POTENTIAL; PRESYNAPTIC POTENTIAL; REINFORCEMENT; REWARD; SIGNAL DETECTION; SYSTEM ANALYSIS; TASK PERFORMANCE; TEMPORAL DIFFERENCE LEARNING; ACTION POTENTIAL; ALGORITHM; ANIMAL; BIOLOGICAL MODEL; HUMAN; PHYSIOLOGY; RAT;

MAMMALIA;

ACTION POTENTIALS; ALGORITHMS; ANIMALS; DOPAMINE; HUMANS; LEARNING; MODELS, NEUROLOGICAL; NERVE NET; NEURONAL PLASTICITY; RATS; REWARD;

EID: 79958078227     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1001133     Document Type: Article

References (107)

1. Sutton RS, Barto AG, (1998) Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning The MIT Press.
2. Schultz W, Dayan P, Montague PR, (1997) A neural substrate of prediction and reward. Science 275: 1593-1599.
3. Schultz W, (2002) Getting formal with dopamine and reward. Neuron 36: 241-263.
4. Montague PR, Dayan P, Sejowski TJ, (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16: 1936-1947.
5. Redgrave P, Gurney K, (2006) The short-latency dopamine signal: a role in discovering novel actions? Nat Rev Neurosci 7: 967-975.
6. Pessiglione M, Seymour B, Flandin G, Dolan R, Frith C, (2006) Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442: 1042-1045.
7. Reynolds JNJ, Hyland BI, Wickens JR, (2001) A cellular mechanism of reward-related learning. Nature 413: 67-70.
8. Pawlak V, Kerr JN, (2008) Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J Neurosci 28: 2435-2446.
9. Reynolds JN, Wickens JR, (2002) Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw 15: 507-521.
10. O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, et al. (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304: 452-454.
11. Witten IH, (1977) An adaptive optimal controller for discrete-time markov environments. Information and Control 34: 286-295.
12. Barto A, Sutton RS, Anderson CW, (1983) Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans Syst Man Cybern 13: 834-846.
13. Morris G, Nevet A, Arkadir D, Vaadia E, Bergman H, (2006) Midbrain dopamine neurons encode decisions for future action. Nat Neurosci 9: 1057-1063.
14. Attalah HE, Lopez-Paniagua D, Rudy JW, O'Reilly RC, (2007) Separate neural substrates for skill-learning and performance in the ventral and dorsal striatum. Nat Neurosci 10: 126-131.
15. Fiorillo CD, Tobler PN, Schultz W, (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299: 1898-1902.
16. Tobler PN, Fiorillo CD, Schultz W, (2005) Adaptive coding of reward value by dopamine neurons. Science 307: 1642-1645.
17. Morris G, Arkadir D, Nevet A, Vaadia E, Bergman H, (2004) Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 1: 133-143.
18. Houk JC, Adams JL, Barto AG, (1995) A model of how the basal ganglia generate and use neural signals that predict reinforcement MIT Press.
19. Montague P, Dayan P, Person C, Sejnowski T, (1995) Bee foraging in uncertain environments using predictive Hebbian learning. Nature 377: 725-728.
20. Suri R, Schultz W, (1999) A neural network model with dopamine-like reinforcement signal that learns a spatial delayed reponse task. Neuroscience 91: 871-890.
21. Suri RE, Schultz W, (2001) Temporal difference model reproduces anticipatory neural activity. Neural Comput 13: 841-862.
22. Joel D, Niv J, Ruppin E, (2002) Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw 15: 535-547.
23. Wörgötter F, Porr B, (2005) Temporal sequence learning, prediction, and control: A review of different models and their relation to biological mechanisms. Neural Comput 17: 245-319.
24. Seung HS, (2003) Learning spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron 40: 1063-1073.
25. Xie X, Seung HS, (2004) Learning in neural networks by reinforcement of irregular spiking. Phys Rev E 69: 41909.
26. Baras D, Meir R, (2007) Reinforcement learning, spike-time-dependent plasticity, and the BCM rule. Neural Comput 19: 2245-2279.
27. Florian RV, (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput 19: 1468-1502.
28. Legenstein R, Pecevski D, Maass W, (2008) A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput Biol 4: e1000180.
29. Vasilaki E, Frémaux N, Urbanczik R, Senn W, Gerstner W, (2009) Spike-based reinforcement learning in continuous state and action space: When policy gradient methods fail. PLoS Comput Biol 5: e1000586 doi:10.1371/journal.pcbi.1000586.
30. Frémaux N, Sprekeler H, Gerstner W, (2010) Functional requirements for reward-modulated spike-timing-dependent plasticity. J Neurosci 30: 13326-13337.
31. Rao RPN, Sejnowski TJ, (2001) Spike-timing-dependent Hebbian plasticity as temporal difference learning. Neural Comput 13: 2221-2237.
32. Farries MA, Fairhall AL, (2007) Reinforcement learning with modulated spike timing-dependent synaptic plasticity. J Neurophysiol 98: 3648-3665.
33. Izhikevich EM, (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17: 2443-2452.
34. Potjans W, Morrison A, Diesmann M, (2009) A spiking neural network model of an actor-critic learning agent. Neural Comput 21: 301-339.
35. Dennett DC, (1998) Brainchildren: Essays on Designing Minds The MIT Press, 1 edition.
36. Barto AG, (1995) Adaptive critic and the basal ganglia. In: Houk JC, Davis J, Beiser D, editors. Models of Information Processing in the Basal Ganglia Cambridge (Massachusetts) MIT Press pp. 215-232.
37. Dayan P, (1992) The convergence of td(λ) for general λ. Mach Learn 8: 341-362.
38. Dayan P, Sejnowski T, (1994) Td(λ) converges with probability 1. Mach Learn 14: 295-301.
39. Foster DJ, Morris RGM, Dayan P, (2000) A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus 10: 1-16.
40. VanRullen R, Guyonneau R, Thorpe SJ, (2005) Spike times make sense. Trends Neurosci 28: 1-4.
41. Gurney K, Prescott TJ, Redgrave P, (2001) A computational model of action selection in the basal ganglia. i. a new functional anatomy. Biol Cybern 84: 401-410.
42. Humphries MD, Stewart RD, Gurney KN, (2006) A physiologically plausible model of action selection and oscillatory activity in the basal ganglia. J Neurosci 26: 12921-12942.
43. Prinz AA, Bucher D, Marder E, (2004) Similar network activity from disparate circuit parameters. Nat Neurosci 7: 1345-1352.
44. Dai M, Tepper JM, (1998) Do silent dopaminergic neurons exist in rat substantia nigra in vivo? Neuroscience 85: 1089-1099.
45. Hyland BI, Reynolds JNJ, Hay J, Perk CG, Miller R, (2002) Firing modes of midbrain dopamine cells in the freely moving rat. Neuroscience 114: 475-492.
46. Bayer HM, Lau B, Glimcher PW, (2007) Statistics of midbrain dopamine neuron spike trains in the awake primate. J Neurophysiol 98: 1428-1439.
47. Ljungberg T, Apicella P, Schultz W, (1992) Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophysiol 67: 145-163.
48. Schultz W, (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80: 1-27.
49. Helias M, Deger M, Rotter S, Diesmann M, (2010) Instantaneous non-linear processing by pulse-coupled threshold units. PLoS Comput Biol 6: e1000929.
50. Froemke RC, Dan Y, (2002) Spike-timing-dependent synaptic modification induced by natural spike trains. Nature 416: 433-438.
51. Garris PA, Ciolkowski EL, Pastore P, Wightman RM, (1994) Efflux of dopamine from the synaptic cleft in the nucleus accumbens of the rat brain. J Neurosci 14: 6084-6093.
52. Montague PR, McClure SM, Baldwin P, Phillips PE, Budygin EA, et al. (2004) Dynamic gain control of dopamine delivery in freely moving animals. J Neurosci 24: 1754-1759.
53. Soltani A, Lee D, Wang XJ, (2006) Neural mechanism for stochastic behavior during a competitive game. Neural Netw 19: 1075-1090.
54. Schweighofer N, Doya K, (2003) Meta-learning in reinforcement learning. Neural Comput 16: 5-9.
55. Friston KJ, Tononi G, Reeke GN Jr, Sporns O, Edelman GM, (1994) Value-dependent selection in the brain: Simulation in a synthetic neural model. Neuroscience 59: 229-243.
56. Calabresi P, Fedele E, Pisani A, Fontana G, Mercuri N, et al. (1995) Transmitter release associated with long-term synaptic depression in rat corticostriatal slices. Eur J Neurosci 7: 1889-1894.
57. Wickens J, (1993) A Theory of the Striatum. Pergamon Studies in Neuroscience. Pergamon.
58. Pawlak V, Wickens JR, Kirkwood A, Kerr JND, (2010) Timing is not everything: neuromodulation opens the STDP gate. Front Syn Neurosci 2 doi:10.3389/fnsyn.2010.00146.
59. Nakano T, Doi T, Yoshimoto J, Doya K, (2010) A kinetic model of dopamine- and calcium-dependent striatal synaptic plasticity. PLoS Comput Biol 6: e1000670.
60. Loewenstein Y, Seung HS, (2006) Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proc Natl Acad Sci USA 103: 15224-9.
61. Fusi S, Asaad WF, Miller EK, Wang XJ, (2007) A neural circuit model of flexible sensorimotor mapping: learning and forgetting on multiple timescales. Neuron 54: 319-33.
62. Soltani A, Wang XJ, (2010) Synaptic computation underlying probabilistic inference. Nat Neurosci 13: 112-9.
63. Steele RJ, Morris RGMM, (1999) Delay-dependent impairment of a matching-to-place task with chronic and intrahippocampal infusion of the nmda-antagonist d-ap5. Hippocampus 9: 118-136.
64. Garthe A, Behr J, Kempermann G, (2009) Adult-generated hippocampal neurons allow the flexible use of spatially precise learning strategies. PLoS ONE 4: e5464.
65. Ludvig EA, Sutton RS, Kehoe EJ, (2008) Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Comput 20: 3034-3054.
66. Daw ND, Kakade S, Dayan P, (2002) Opponent interactions between serotonin and dopamine. Neural Networks 15: 603-616.
67. Reynolds SM, Berridge KC, (2001) Fear and feeding in the nucleus accumbens shell: Rostrocaudal segregation of gaba-elicited defensive behavior versus eating behavior. J Neurosci 21: 3261-3270.
68. Reynolds SM, Berridge KC, (2002) Positive and negative motivation in nucleus accumbens shell: Bivalent rostrocaudal gradients for gaba-elicited eating, taste "liking"/"disliking" reactions, place preference/avoidance, and fear. J Neurosci 22: 7308-7320.
69. Seymour B, Daw N, Dayan P, Singer T, Dolan R, (2007) Differential encoding of losses and gains in the human striatum. J Neurosci 27: 4826-4831.
70. Yacubian J, Gläscher J, Schroeder K, Sommer T, Braus DF, et al. (2006) Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain. J Neurosci 26: 9530-9537.
71. Bowery N, Hudson A, Price G, (1987) Gabaa andgabab receptor site distribution in the rat central nervous system. Neuroscience 20: 365-383.
72. Häusser MA, Yung WH, (1994) Inhibitory synaptic potentials in guinea-pig substantia nigra dopamine neurones in vitro. J Physiol 479: 401-422.
73. Sugita S, Johnson SW, North RA, (1992) Synaptic inputs to gabaa and gabab receptors originate from discrete afferent neurons. Neurosci Lett 134: 207-211.
74. Tepper JM, Martin L, Anderson DR, (1995) Gabaa receptor-mediated inhibition of rat substantia nigra dopaminergic neurons by pars reticulata projection neurons. J Neurosci 15: 3092-3103.
75. Paladini CA, Celada P, Tepper JM, (1999) Striatal, pallidal, and pars reticulata evoked inhibition of nigrostriatal dopaminergic neurons is mediated by gabaa receptors in vivo. Neuroscience 89: 799-812.
76. Brazhnik E, Shah F, Tepper JM, (2008) Gabaergic afferents activate both gabaa and gabab receptors in mouse substantia nigra dopaminergic neurons in vivo. J Neurosci 28: 10386-10398.
77. Suri RE, Bargas J, Arbib MA, (2001) Modeling functions of striatal dopamine modulation in learning and planning. Neuroscience 103: 65-85.
78. Berns GS, Sejnowski TJ, (1998) A computational model of how the basal ganglia produce sequences. J Cogn Neurosci 10: 108-121.
79. Contreras-Vidal JL, Schultz W, (1999) A predictive reinforcement model of dopamine neurons for learning approach behavior. J Comput Neurosci 6: 191-214.
80. Brown J, Bullock D, Grossberg S, (1999) How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J Neurosci 19: 10502-10511.
81. Matsumoto M, Hikosaka O, (2007) Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447: 1111-1117.
82. Hopfield JJ, (1982) Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci USA 79: 2554-2558.
83. Jin DZ, (2009) Generating variable birdsong syllable sequences with branching chain networks in avian premotor nucleus HVC. Phys Rev E 80: 051902.
84. Hanuschkin A, Herrmann JM, Morrison A, Diesmann M, (2010) Compositionality of arm movements can be realized by propagating synchrony. J Comput Neurosci E-pub ahead of print. doi:10.1007/s10827-010-0285-9.
85. Schrader S, Diesmann M, Morrison A, (2010) A compositionality machine realized by a hierarchic architecture of synfire chains. Front Comput Neurosci 4: 154.
86. Seymour B, O'Doherty J, Dayan P, Koltzenburg M, Jones A, et al. (2004) Temporal difference models describe higher-order learning in humans. Nature 429: 664-667.
87. Houk JC, Wise SP, (1995) Distributed modular architectures linking basal ganglia, cerebellum, and cerebral cortex: Their role in planning and controlling action. Cereb Cortex 5: 95-110.
88. Houk JC, (2005) Agents of the mind. Biol Cybern 92: 427-437.
89. Houk JC, (2007) Models of basal ganglia. Scholarpedia 2: 1633.
90. Sethi KD, (2002) Clinical aspects of parkinson disease. Curr Opin Neurol 15: 457-460.
91. Knowlton BJ, Mangels JA, Squire LR, (1996) A neostriatal habit learning system in humans. Science 273: 1399-1420.
92. McDonald RJ, White NM, (1993) A triple dissociation of memory systems: hippocampus, amygdala, and dorsal striatum. Behav Neurosci 107: 3-22.
93. Sutton RS, Barto AG, (1990) Time-derivative models of pavlovian reinforcement. In: Gabriel M, Moore J, editors. Learning and Computational Neuroscience Cambridge (Massachusetts) MIT Press pp. 497-537.
94. Niv Y, Joel D, Meilijson I, Ruppin E, (2002) Evolution of reinforcement learning in uncertain environments: A simple explanation for complex foraging behaviors. Adapt Behav 10: 5-24.
95. Doya K, (2000) Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr Opin Neurol 10: 732-739.
96. La Camera G, Richmond BJ, (2008) Modeling the violation of reward maximization and invariance in reinforcement schedules. PLoS Comput Biol 4: e1000131.
97. Dayan P, (2009) Prospective and retrospective temporal difference learning. Network Comput Neural Syst 20: 32-46.
98. Matsumoto M, Hikosaka O, (2009) Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459: 837-842.
99. Arias-Carrion O, Pöppel E, (2007) Dopamine, learning, and reward-seeking behavior. Acta Neurobiol Exp (Wars) 67: 481-488.
100. Pecina S, Cagniard B, Berridge KC, Aldridge JW, Zhuang X, (2003) Hyperdopaminergic mutant mice have higher "wanting" but not "liking" for sweet rewards. J Neurosci 23: 9395-402.
101. Horvitz JC, (2000) Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96: 651-656.
102. Redgrave P, Prescott TJ, Gurney K, (1999) Is the short-latency dopamine response too short to signal reward error? Trends Neurosci 22: 146-151.
103. Porr B, Wörgötter F, (2007) Learning with relevance: Using a third factor to stabilise hebbian learning. Neural Comput 19: 2694-2719.
104. Gewaltig MO, Diesmann M, (2007) NEST (NEural Simulation Tool). Scholarpedia 2: 1430.
105. Potjans W, Morrison A, Diesmann M, (2010) Enabling functional neural circuit simulations with distributed computing of neuromodulated plasticity. Front Comput Neurosci 4 doi:10.3389/fncom.2010.00141.
106. Tuckwell HC, (1988) Introduction to Theoretical Neurobiology, volume 1 Cambridge Cambridge University Press.
107. Nordlie E, Gewaltig MO, Plesser HE, (2009) Towards reproducible descriptions of neuronal network models. PLoS Comput Biol 5: e1000456.