Spatio-Temporal credit assignment in neuronal population learning

PLoS Computational Biology

Volumn 7, Issue 6, 2011, Pages

  Friedrich, Johannes ^a   Urbanczik, Robert ^a   Senn, Walter ^a  

^a UNIVERSITY OF BERN   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

COMBINATORIAL OPTIMIZATION; DECISION MAKING; NEURONS; POPULATION STATISTICS; REINFORCEMENT;

CREDIT ASSIGNMENT; CREDIT ASSIGNMENT PROBLEMS; LEAKY INTEGRATE AND FIRE NEURON; NEURONAL POPULATIONS; OPERANT CONDITIONING; PRESYNAPTIC INPUTS; REINFORCEMENT LEARNINGS; SPATIO-TEMPORAL; SPATIO-TEMPORAL AGGREGATION; TRIAL AND ERROR;

REINFORCEMENT LEARNING;

DOPAMINE;

ARTICLE; CONTROLLED STUDY; DECISION MAKING; LEARNING ALGORITHM; LEARNING DISORDER; MATHEMATICAL MODEL; NERVE CELL NETWORK; NERVE CELL PLASTICITY; NEUROMODULATION; POPULATION SIZE; POSTSYNAPTIC POTENTIAL; PRESYNAPTIC POTENTIAL; REINFORCEMENT; SIGNAL TRANSDUCTION; SPATIAL MEMORY; STIMULUS RESPONSE; SYNAPTIC TRANSMISSION; WORKING MEMORY; ALGORITHM; ANIMAL; BIOLOGICAL MODEL; BIOLOGY; COMPUTER SIMULATION; DOG; LEARNING; MEMORY; NERVE CELL; PHYSIOLOGY; PROBABILITY; REWARD; SYNAPSE; TIME;

ANIMALIA;

ALGORITHMS; ANIMALS; COMPUTATIONAL BIOLOGY; COMPUTER SIMULATION; DECISION MAKING; DOGS; LEARNING; MARKOV CHAINS; MEMORY; MODELS, NEUROLOGICAL; NEURONAL PLASTICITY; NEURONS; REWARD; SIGNAL TRANSDUCTION; SYNAPSES; TIME FACTORS;

EID: 79959853243     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1002092     Document Type: Article

References (42)

1. Sutton RS, Barto AG, (1998) Reinforcement Learning: An Introduction Cambridge, MA MIT Press.
2. Markram H, Lübke J, Frotscher M, Sakmann B, (1997) Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science 275: 213-215.
3. Bi G, Poo M, (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J Neurosci 18: 10464-10472.
4. Song S, Abbott LF, (2001) Cortical development and remapping through spike timing-dependent plasticity. Neuron 32: 339-350.
5. Baras D, Meir R, (2007) Reinforcement learning, spike-time-dependent plasticity, and the BCM rule. Neural Comput 19: 2245-2279.
6. Florian R, (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput 19: 1468-1502.
7. Izhikevich E, (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17: 2443-2452.
8. Legenstein R, Naeger C, Maass W, (2005) What can a neuron learn with spike-timing-dependent plasticity? Neural Comput 17: 2337-2382.
9. Pawlak V, Kerr JND, (2008) Dopamine receptor activation is required for corticostriatal spiketiming-dependent plasticity. J Neurosci 28: 2435-2446.
10. Zhang J, Lau P, Bi G, (2009) Gain in sensitivity and loss in temporal contrast of STDP by dopaminergic modulation of hippocampal synapses. Proc Natl Acad Sci USA 106: 13028-13033.
11. Gavornik JP, Shuler MGH, Loewenstein Y, Bear MF, Shouval HZ, (2009) Learning reward timing in cortex through reward dependent expression of synaptic plasticity. Proc Natl Acad Sci USA 106: 6826-6831.
12. Schultz W, Dayan P, Montague PR, (1997) A neural substrate of prediction and reward. Science 275: 1593-1599.
13. Dayan P, Niv Y, (2008) Reinforcement learning: the good, the bad and the ugly. Curr Opin Neurobiol 18: 185-196.
14. Pfister J, Toyoizumi T, Barber D, Gerstner W, (2006) Optimal spike-timing-dependent plasticity for precise action potential firing in supervised learning. Neural Comput 18: 1318-1348.
15. Castro D, Volkinshtein S, Meir R, (2009) Temporal difference based actor critic learning - convergence and neural implementation. Advances in neural information processing systems 21 Cambridge, MA MIT Press pp. 385-392.
16. Urbanczik R, Senn W, (2009) Reinforcement learning in populations of spiking neurons. Nat Neurosci 12: 250-252.
17. Vasilaki E, Frémaux N, Urbanczik R, Senn W, Gerstner W, (2009) Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail. PLoS Comput Biol 5: e1000586.
18. Clopath C, Büsing L, Vasilaki E, Gerstner W, (2010) Connectivity reflects coding: a model of voltage-based STDP with homeostasis. Nat Neurosci 13: 344-352.
19. Wang X, (2002) Probabilistic decision making by slow reverberation in cortical circuits. Neuron 36: 955-968.
20. Foehring R, Lorenzon N, (1999) Neuromodulation, development and synaptic plasticity. Can J Exp Psychol 53: 45-61.
21. Matsuda Y, Marzo A, Otani S, (2006) The presence of background dopamine signal converts longterm synaptic depression to potentiation in rat prefrontal cortex. J Neurosci 26: 4803-4810.
22. Seol G, Ziburkus J, Huang S, Song L, Kim I, et al. (2007) Neuromodulators control the polarity of spike-timing-dependent synaptic plasticity. Neuron 55: 919-929.
23. Daw ND, Doya K, (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16: 199-204.
24. Mazur J, (2002) Learning and Behavior Upper Saddle River, NJ Prentice Hall.
25. Baum W, Aparicio C, (1999) Optimality And Concurrent Variable-interval Variable-ratio Schedules. J Exp Anal Behav 71: 75-89.
26. Song S, Miller K, Abbott L, (2000) Competitive Hebbian learning through spike-timing dependent synaptic plasticity. Nat Neurosci 3: 919-926.
27. Lynch MA, (2004) Long-term potentiation and memory. Physiol Rev 84: 87-136.
28. Nevian T, Sakmann B, (2006) Spine Ca2+ signaling in spike-timing-dependent plasticity. J Neurosci 26: 11001-11013.
29. Abraham W, (2008) Metaplasticity: tuning synapses and networks for plasticity. Nat Neurosci 9: 387-399.
30. Volterra A, Meldolesi J, (2005) Astrocytes, from brain glue to communication elements: the revolution continues. Nat Rev Neurosci 6: 626-640.
31. Henneberger C, Papouin T, Oliet SH, Rusakov DA, (2010) Long-term potentiation depends on release of D-serine from astrocytes. Nature 463: 232-236.
32. Frey S, Frey JU, (2008) 'Synaptic tagging' and 'cross-tagging' and related associative reinforcement processes of functional plasticity as the cellular basis for memory formation. Prog Brain Res 169: 117-143.
33. Almaguer-Melian W, Bergado JA, Lopez-Rojas J, Frey S, Frey JU, (2010) Differential effects of electrical stimulation patterns, motivational-behavioral stimuli and their order of application on functional plasticity processes within one input in the dentate gyrus of freely moving rats in vivo. Neuroscience 165: 1546-1558.
34. Friedrich J, Urbanczik R, Senn W, (2010) Learning spike-based population codes by reward and population feedback. Neural Comput 22: 1698-1717.
35. Williams R, (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8: 229-256.
36. Baxter J, Bartlett P, (2001) Infinite-horizon policy-gradient estimation. J Artif Intell Res 15: 319-350.
37. Baxter J, Bartlett P, Weaver L, (2001) Experiments with infinite-horizon, policy-gradient estimation. J Artif Intell Res 15: 351-381.
38. Legenstein R, Pecevski D, Maass W, (2008) A learning theory for reward-modulated spike-timingdependent plasticity with application to biofeedback. PLoS Comput Biol 4: e1000180.
39. Fetz EE, Baker MA, (1973) Operantly conditioned patterns on precentral unit activity and correlated responses in adjacent cells and contralateral muscles. J Neurophysiol 36: 179-204.
40. Pouget A, Dayan P, Zemel R, (2000) Information processing with population codes. Nat Rev Neurosci 1: 125-132.
41. Averbeck B, Latham P, Pouget A, (2006) Neural correlations, population coding and computation. Nat Rev Neurosci 7: 358-366.
42. Sutton R, McAllester D, Singh S, Mansour Y, (2002) Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems 12 Cambridge, MA MIT Press pp. 1057-1063.