A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback

PLoS Computational Biology

Volumn 4, Issue 10, 2008, Pages

  Legenstein, Robert ^a   Pecevski, Dejan ^a   Maass, Wolfgang ^a  

^a GRAZ UNIVERSITY OF TECHNOLOGY   (Austria)

Author keywords

[No Author keywords available]

Indexed keywords

BIOFEEDBACK; COMBINATORIAL OPTIMIZATION; COMPLEX NETWORKS; COMPUTATION THEORY; RECURRENT NEURAL NETWORKS; TIMING CIRCUITS;

CONDITION; CREDIT ASSIGNMENT PROBLEMS; FIRING PATTERNS; LEARN+; LEARNING RULES; LEARNING THEORY; SELF-ORGANISING; SPIKE TIMING DEPENDENT PLASTICITIES; SPIKING NEURONES; SYNAPTIC PLASTICITY;

NEURONS;

ARTICLE; BRAIN CORTEX; COMPUTER SIMULATION; FEEDBACK SYSTEM; LEARNING THEORY; NERVE CELL NETWORK; NERVE CELL PLASTICITY; REWARD; SPIKE WAVE; SYNAPSE; WEIGHT CHANGE; ACTION POTENTIAL; ANIMAL; ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; BIOLOGY; HAPLORHINI; LEARNING; NEUROSCIENCE; PATTERN RECOGNITION; PHYSIOLOGY; PSYCHOPHYSIOLOGY; STATISTICS;

ACTION POTENTIALS; ANIMALS; BIOFEEDBACK (PSYCHOLOGY); COMPUTATIONAL BIOLOGY; COMPUTER SIMULATION; HAPLORHINI; LEARNING; MODELS, NEUROLOGICAL; NERVE NET; NEURAL NETWORKS (COMPUTER); NEURONAL PLASTICITY; NEUROSCIENCES; PATTERN RECOGNITION, PHYSIOLOGICAL; REWARD;

EID: 55449121121     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1000180     Document Type: Article

References (50)

1. Abbott LF, Nelson SB (2000) Synaptic plasticity: taming the beast. Nat Neurosci 3: 1178-1183.
2. Jacob V, Brasier D, Erchova I, Feldman D, Shulz DE (2007) Spike timing-dependent synaptic depression in the in vivo barrel cortex of the rat. J Neuroscience 27: 1271-1284.
3. Bailey CH, Giustetto M, Huang YY, Hawkins RD, Kandel ER (2000) Is heterosynaptic modulation essential for stabilizing Hebbian plasticity and memory? Nat Rev Neurosci 1: 11-20.
4. Gu Q (2002) Neuromodulatory transmitter systems in the cortex and their role in cortical plasticity. Neuroscience 111: 815-835.
5. Schultz W (2007) Behavioral dopamine signals. Trends Neurosci 30: 203-210.
6. Reynolds JN, Hyland BI, Wickens JR (2001) A cellular mechanism of reward-related learning. Nature 413: 67-70.
7. Reynolds JN, Wickens JR (2002) Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw 15: 507-521.
8. Bao S, Chan VT, Merzenich MM (2001) Cortical remodelling induced by activity of ventral tegmental dopamine neurons. Nature 412: 79-83.
9. Shulz DE, Sosnik R, Ego V, Haidarliu S, Ahissar E (2000) A neuronal analogue of state-dependent learning. Nature 403: 549-553.
10. Thiel CM, Friston KJ, Dolan RJ (2002) Cholinergic modulation of experience-dependent plasticity in human auditory cortex. Neuron 35: 567-574.
11. Shulz DE, Ego-Stengel V, Ahissar E (2003) Acetylcholine-dependent potentiation of temporal frequency representation in the barrel cortex does not depend on response magnitude during conditioning. J Physiol Paris 97: 431-439.
12. Izhikevich EM (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17: 2443-2452.
13. Florian RV (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput 6: 1468-1502.
14. Baxter J, Bartlett PL (1999) Direct gradient-based reinforcement learning: I. Gradient estimation algorithms. Technical report. Research School of Information Sciences and Engineering, Australian National University.
15. Baras D, Meir R (2007) Reinforcement learning, spike-time-dependent plasticity, and the bcm rule. Neural Comput 19: 2245-2279.
16. Pfister JP, Toyoizumi T, Barber D, Gerstner W (2006) Optimal spike-timing-dependent plasticity for precise action potential firing in supervised learning. Neural Comput 18: 1318-1348.
17. Fetz EE, Baker MA (1973) Operantly conditioned patterns of precentral unit activity and correlated responses in adjacent cells and contralateral muscles. J Neurophysiol 36: 179-204.
18. Fetz EE (1969) Operant conditioning of cortical unit activity. Science 163: 955-958.
19. Fetz EE (2007) Volitional control of neural activity: implications for brain-computer interfaces. J Physiol 579: 571-579.
20. Brunel N (2000) Dynamics of networks of randomly connected excitatory and inhibitory spiking neurons. J Physiol Paris 94: 445-463.
21. Gerstner W, Kistler WM (2002) Spiking Neuron Models. Cambridge, UK: Cambridge University Press.
22. Morrison A, Aertsen A, Diesmann M (2007) Spike-timing-dependent plasticity in balanced random networks. Neural Comput 19: 1437-1467.
23. Bi G, Poo M (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J Neurosci 18: 10464-10472.
24. Song S, Miller KD, Abbott LF (2000) Competitive hebbian learning through spiketiming dependent synaptic plasticity. Nat Neurosci 3: 919-926.
25. Kempter R, Gerstner W, van Hemmen JL (2001) Intrinsic stabilization of output rates by spike-based hebbian learning. Neural Comput 13: 2709-2741.
26. Destexhe A, Rudolph M, Fellous JM, Sejnowski TJ (2001) Fluctuating synaptic conductances recreate in vivo-like activity in neocortical neurons. Neuroscience 107: 13-24.
27. Farries MA, Fairhall AL (2007) Reinforcement learning with modulated spike timing-dependent synaptic plasticity. J Neurophysiol 98: 3648-3665.
28. Stevens CF, Zador AM (1998) Input synchrony and the irregular firing of cortical neurons. Nat Neurosci 1: 210-217.
29. Mainen Z, Sejnowski T (1995) Reliability of spike timing in neocortical neurons. Science 268: 1503-1505.
30. Silberberg G, Bethge M, Markram H, Pawelzik K, Tsodyks M (2004) Dynamics of population rate codes in ensembles of neocortical neurons. J Neurophysiol 91: 704-709.
31. Maass W, Natschläger T, Markram H (2002) Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput 14: 2531-2560.
32. Maass W, Natschläger T, Markram H (2004) Fading memory and kernel properties of generic cortical microcircuit models. J Physiol Paris 98: 315-330.
33. Destexhe A, Marder E (2004) Plasticity in single neuron and circuit computations. Nature 431: 789-795.
34. Maass W, Joshi P, Sontag ED (2007) Computational aspects of feedback in neural circuits. PLoS Comput Biol 3: e165. doi:10.1371/journal.pcbi.0020165.
35. Nikolić D, Haeusler S, Singer W, Maass W (2007) Temporal dynamics of information content carried by neurons in the primary visual cortex. In: Proceedings of NIPS 2006, Advances in Neural Information Processing Systems. Volume 19. Cambridge (Massachusetts): MIT Press. pp 1041-1048.
36. Kempter R, Gerstner W, van Hemmen JL (1999) Hebbian learning and spiking neurons. Phys Rev E 59: 4498-4514.
37. Markram H, Wang Y, Tsodyks M (1998) Differential signaling via the same axon of neocortical pyramidal neurons. Proc Natl Acad Sci U S A 95: 5323-5328.
38. Maass W, Markram H (2002) Synapses as dynamic memory buffers. Neural Networks 15: 155-161.
39. Gupta A, Wang Y, Markram H (2000) Organizing principles for a diversity of GABAergic interneurons and synapses in the neocortex. Science 287: 273-278.
40. Borg-Graham LJ, Monier C, Frégnac Y (1998) Visual input evokes transient and strong shunting inhibition in visual cortical neurons. Nature 393: 369-373.
41. Hirsch JA, Alonso JM, Reid RC, Martinez LM (1998) Synaptic integration in striate cortical simple cells. J Neurosci 18: 9517-9528.
42. Anderson J, Lampl I, Reichova I, Carandini M, Ferster D (2000) Stimulus dependence of two-state fluctuations of membrane potential in cat visual cortex. Nature Neurosci 3: 617-621.
43. Lyon R (1982) A computational model of filtering, detection, and compression in the cochlea. In: Proceedings of IEEE International Conference on ICASSP. pp 1282-1285.
44. Schrauwen B, Campenhout JV (2003) BSA, a fast and accurate spike train encoding scheme. In: Proceedings of the International Joint Conference on Neural Networks. Volume 4. pp 2825-2830.
45. Verstraeten D, Schrauwen B, Stroobandt D, Campenhout JV (2005) Isolated word recognition with the liquid state machine: a case study. Inf Process Lett 95: 521-528.
46. Fetz EE, Finocchio DV (1975) Correlations between activity of motor cortex cells and arm muscles during operantly conditioned response patterns. Exp Brain Res 23: 217-240.
47. Häusler S, Maass W (2007) A statistical analysis of information processing properties of lamina-specific cortical microcircuit models. Cereb Cortex 17: 149-162.
48. Hopfield JJ, Brody CD (2001) What is a moment? Transient synchrony as a collective mechanism for spatio-temporal integration. Proc Natl Acad Sci U S A 98: 1282-1287.
49. Fiete IR, Seung HS (2006) Gradient learning in spiking neural networks by dynamic perturbation of conductances. Phys Rev Lett 97: 048104.
50. Gütig R, Sompolinsky H (2006) The tempotron: a neuron that learns spike timingbased decisions. Nat Neurosci 9: 420-428.