Functional requirements for reward-modulated spike-timing-dependent plasticity

Journal of Neuroscience

Volumn 30, Issue 40, 2010, Pages 13326-13337

  Frémaux, Nicolas ^a   Sprekeler, Henning ^a   Gerstner, Wulfram ^a  

^a EPFL   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL BEHAVIOR; ANIMAL EXPERIMENT; ARTICLE; CONTROLLED STUDY; LEARNING; NERVE CELL EXCITABILITY; NERVE CELL PLASTICITY; NEUROMODULATION; NONHUMAN; POSTSYNAPTIC POTENTIAL; PRIORITY JOURNAL; REWARD; SIGNAL TRANSDUCTION; STIMULUS RESPONSE; TASK PERFORMANCE;

ACTION POTENTIALS; ANIMALS; BRAIN; COMPUTER SIMULATION; CONDITIONING, OPERANT; LEARNING; LONG-TERM POTENTIATION; MODELS, NEUROLOGICAL; NEURONAL PLASTICITY; NEURONS; REACTION TIME; REWARD; SYNAPSES; SYNAPTIC POTENTIALS; SYNAPTIC TRANSMISSION;

EID: 77957731196     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.6249-09.2010     Document Type: Article

References (51)

1. Arbuthnott GW, Wickens J (2007) Space, time and dopamine. Trends Neurosci 30:62-69.
2. Artola A, Bröcher S, Singer W (1990) Different voltage dependent thresholds for inducing long-term depression and long-term potentiation in slices of rat visual cortex. Nature 347:69-72.
3. Baras D, Meir R (2007) Reinforcement learning, spike-time-dependent plasticity, and the BCM rule. Neural Comput 19:2245-2279.
4. Bi GQ, Poo MM (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J Neurosci 18:10464-10472.
5. Bienenstock EL, Cooper LN, Munroe PW (1982) Theory of the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. J Neurosci 2:32-48. Reprinted in Anderson and Rosenfeld (1990).
6. Bliss TV, Gardner-Medwin AR (1973) Long-lasting potentation of synaptic transmission in the dentate area of unanaesthetized rabbit following stimulation of the perforant path. J Physiol 232:357-374.
7. Brashers-Krug T, Shadmehr R, Bizzi E (1996) Consolidation in human motor memory. Nature 382:252-255.
8. Dayan P, Abbott LF (2001) Theoretical neuroscience: computational and mathematical modeling of neural systems. Cambridge: MIT.
9. Farries MA, Fairhall AL (2007) Reinforcement learning with modulated spike timing dependent synaptic plasticity. J Neurophysiol 98:3648-3665.
10. Florian RV (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput 19:1468-1502.
11. Froemke RC, Merzenich MM, Schreiner CE (2007) A synaptic memory trace for cortical receptive field plasticity. Nature 450:425-429.
12. Georgopoulos AP, Schwartz AB, Kettner RE (1986) Neuronal population coding of movement direction. Science 233:1416-1419.
13. Georgopoulos AP, Kettner RE, Schwartz AB (1988) Primate motor cortex and free arm movements to visual targets in three-dimensional space. II. Coding of the direction of movement by a neuronal population. J Neurosci 8:2928-2937.
14. Gerstner W, Kistler WM (2002) Spiking neuron models. Cambridge, UK: Cambridge UP.
15. Gerstner W, Kempter R, van Hemmen JL, Wagner H (1996) A neuronal learning rule for sub-millisecond temporal coding. Nature 383:76-81.
16. Greensmith E, Bartlett PL, Baxter J (2004) Variance reduction techniques for gradient estimates in reinforcement learning. J Mach Learn Res 5:1471-1530.
17. Gütig R, Aharonov R, Rotter S, Sompolinsky H (2003) Learning input correlations through non-linear temporally asymmetric Hebbian plasticity. J Neurosci 23:3697-3714. (Pubitemid 36966039)
18. Guyonneau R, VanRullen R, Thorpe SJ (2005) Neurons tune to the earliest spikes through STDP. Neural Comput 17:859-879.
19. Hebb DO (1949) The organization of behavior: a neuropsychological theory. New York: Wiley.
20. Izhikevich EM (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17:2443-2452.
21. Jay TM (2003) Dopamine: a potential substrate for synaptic plasticity and memory mechanisms. Prog Neurobiol 69:375-390. (Pubitemid 36859925)
22. Legenstein R, Pecevski D, Maass W (2008) A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput Biol 4:e1000180.
23. Mackintosh NJ (1975) A theory of attention: variations in the associability of stimuli with reinforcement. Psychol Rev 82:276-298.
24. Markram H, Lübke J, Frotscher M, Sakmann B (1997) Regulation of synaptic efficacy by coincidence of postsynaptic AP and EPSP. Science 275:213-215. (Pubitemid 27034762)
25. Oja E (1982) A simplified neuron model as a principal component analyzer. J Math Biol 15:267-273.
26. Pawlak V, Kerr JN (2008) Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J Neurosci 28:2435-2446. (Pubitemid 351348182)
27. Pfister JP, Toyoizumi T, Barber D, Gerstner W (2006) Optimal spike-timing dependent plasticity for precise action potential firing in supervised learning. Neural Comput 18:1318-1348. (Pubitemid 43765446)
28. Potjans W, Morrison A, Diesmann M (2009) A spiking neural network model of an actor-critic learning agent. Neural Comput 21:301-339.
29. Rescorla R, Wagner A (1972) A theory of pavlovian conditioning: variations in the effectiveness of reinforecement and nonreinforcement. In: Classical conditioning II: current theory and research (Prokasy AH, Black WF, eds.) pp 64-99. New York: Appleton Century Crofts.
30. Reynolds JN, Wickens JR (2002) Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw 15:507-521.
31. Reynolds JN, Hyland BI, Wickens JR (2001) A cellular mechanism of reward-related learning. Nature 413:67-70.
32. Schultz W (2007) Behavioral dopamine signals. Trends Neurosci 30:203-210.
33. Schultz W (2010) Dopamine signals for reward value and risk: basic and recent data. Behav Brain Funct 6:24.
34. Schultz W, Dayan P, Montague PR (1997) A neural substrate for prediction and reward. Science 275:1593-1599.
35. Schwartz AB (2004) Cortical neural prosthetics. Annu Rev Neurosci 27:487-507.
36. Seol GH, Ziburkus J, Huang S, Song L, Kim IT, Takamiya K, Huganir RL, Lee HK, Kirkwood A (2007) Neuromodulators control the polarity of spike-timing-dependent synaptic plasticity. Neuron 55:919-929. (Pubitemid 47391189)
37. Seung HS (2003) Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron 40:1063-1073. (Pubitemid 38032784)
38. Shadmehr R, Holcomb HH (1997) Neural correlates of motor memory consolidation. Science 277:821-825. (Pubitemid 27366734)
39. Sjöström PJ, Turrigiano GG, Nelson SB (2001) Rate, timing, and cooperativity jointly determine cortical synaptic plasticity. Neuron 32:1149-1164.
40. Sjöström PJ, Rancz EA, Roth A, Häusser M (2008) Dendritic excitability and synaptic plasticity. Physiol Rev 88:769-840.
41. Song S, Miller KD, Abbott LF (2000) Competitive Hebbian learning through spike-time-dependent synaptic plasticity. Nat Neurosci 3:919-926.
42. Suri RE, Schultz W (1998) Learning of sequential movements with dopamine-like reinforcement signal in neural network model. Exp Brain Res 121:350-354.
43. Sutton R, Barto A (1998) Reinforcement learning. Cambridge: MIT.
44. van Rossum MC, Bi GQ, Turrigiano GG (2000) Stable Hebbian learning from spike timing-dependent plasticity. J Neurosci 20:8812-8821.
45. Vasilaki E, Frémaux N, Urbanczik R, Senn W, Gerstner W (2009) Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail. PLoS Comput Biol 5:e1000586.
46. Victor JD, Purpura K (1997) Metric-space analysis of spike trains: theory, algorithms, and application. Network Comput Neural Syst 8:127-164. (Pubitemid 127719538)
47. Weinberger NM (2003) The nucleus basalis and memory codes: auditory cortical plasticity and the induction of specific, associative behavioral memory. Neurobiol Learn Mem 80:268-284.
48. Wickens JR (2009) Synaptic plasticity in the basal ganglia. Behav Brain Res 199:119-128.
49. Williams R (1992) Simple statistical gradient-following methods for connectionist reinforcement learning. Mach Learn 8:229-256.
50. Xie X, Seung HS (2004) Learning in neural networks by reinforcement of irregular spiking. Phys Rev E 69:041909.
51. Zhang JC, Lau PM, Bi GQ (2009) Gain in sensitivity and loss in temporal contrast of STDP by dopaminergic modulation at hippocampal synapses. Proc Natl Acad Sci U S A 106:13028-13033.