Reinforcement Learning Using a Continuous Time Actor-Critic Framework with Spiking Neurons

PLoS Computational Biology

Volumn 9, Issue 4, 2013, Pages

  Frémaux, Nicolas ^a   Sprekeler, Henning ^a,b   Gerstner, Wulfram ^a  

^a EPFL   (Switzerland)

^b HUMBOLDT UNIVERSITY   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

AMINES; ANIMALS; COMPUTATION THEORY; CONTINUOUS TIME SYSTEMS; FORECASTING; REINFORCEMENT LEARNING;

ACTOR CRITIC; CONTINOUS TIME; DOPAMINE; EXPERIMENTAL EVIDENCE; NEUROMODULATORS; REINFORCEMENT LEARNINGS; REWARD-PREDICTION ERROR; SPIKE TIMING DEPENDENT PLASTICITIES; SPIKING NEURONES; SYNAPTIC PLASTICITY;

NEURONS;

ACTION POTENTIAL; ARTICLE; CONTINUOUS TIME ACTOR CRITIC FRAMEWORK; MATHEMATICAL COMPUTING; MATHEMATICAL MODEL; MAZE TEST; MOTOR CONTROL; NERVE CELL; NERVE CELL PLASTICITY; PREDICTION; REINFORCEMENT; RESPONSE TIME; REWARD; SPIKE WAVE; SPIKING NEURON; TASK PERFORMANCE;

ANIMALIA;

EID: 84876888983     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1003024     Document Type: Article

References (62)

1. Sutton R, Barto A (1998) Reinforcement learning. Cambridge: MIT Press.
2. Sutton RS, (1988) Learning to predict by the methods of temporal differences. Machine Learning 3: 9-44.
3. Williams R, (1992) Simple statistical gradient-following methods for connectionist reinforcement learning. Machine Learning 8: 229-256.
4. Xie X, Seung H, (2004) Learning in neural networks by reinforcement of irregular spiking. Physical Review E 69: 41909.
5. Baxter J, Bartlett P, (2001) Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research 15: 319-350.
6. Florian RV, (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Computation 19: 1468-1502.
7. Schultz W, Dayan P, Montague PR, (1997) A neural substrate of prediction and reward. Science 275: 1593-1599.
8. Wickens JR, Begg AJ, Arbuthnott GW, (1996) Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience 70: 1-5.
9. Reynolds JNJ, Wickens JR, (2000) Substantia nigra dopamine regulates synaptic plasticity and membrane potential uctuations in the rat neostriatum, in vivo. Neuroscience 99: 199-203.
10. Reynolds JNJ, Hyland BI, Wickens JR, (2001) A cellular mechanism of reward-related learning. Nature 413: 67-70.
11. Reynolds JNJ, Wickens JR, (2002) Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw 15: 507-521.
12. Pawlak V, Kerr JND, (2008) Dopamine receptor activation is required for corticostriatal spiketiming-dependent plasticity. J Neurosci 28: 2435-2446.
13. Zhang JC, Lau PM, Bi GQ, (2009) Gain in sensitivity and loss in temporal contrast of STDP by dopaminergic modulation at hippocampal synapses. PNAS 106: 13028-13033.
14. Pawlak V, Wickens J, Kirkwood A, Kerr J, (2010) Timing is not everything: neuromodulation opens the STDP gate. Frontiers in Synaptic Neuroscience 2: 1-14.
15. Potjans W, Morrison A, Diesmann M, (2009) A spiking neural network model of an actor-critic learning agent. Neural Computation 21: 301-339.
16. Vasilaki E, Frémaux N, Urbanczik R, Senn W, Gerstner W, (2009) Spike-based reinforcement learning in continuous state and action space: When policy gradient methods fail. PLoS Comput Biol 5: e1000586.
17. Gold JI, Shadlen MN, (2007) The neural basis of decision making. Annual Review of Neuroscience 30: 535-574.
18. Barto A, Sutton R, Anderson C, (1983) Neuronlike adaptive elements that can solve difficult learning and control problems. IEEE transactions on systems, man, and cybernetics 13: 835-846.
19. Doya K, (2000) Reinforcement learning in continuous time and space. Neural Computation 12: 219-245.
20. Arleo A, GerstnerW (2000) Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity. Biological Cybernetics 83: 287-299.
21. Foster D, Morris R, Dayan P, (2000) Models of hippocampally dependent navigation using the temporal difference learning rule. Hippocampus 10: 1-16.
22. O'Keefe J, Nadal L, (1971) The hippocampus as a spatial map: preliminary evidence from unit activity in the freely-moving rat. Brain Res 34: 171-175.
23. Watkins C, Dayan P, (1992) Q-Learning. Machine Learning 8: 279-292.
24. Sutton RS (1996) Generalization in reinforcement learning: Successful examples using sparse coarse coding. In: Advances in Neural Information Processing Systems 8. MIT Press, pp. 1038-1044.
25. Baird LC (1995) Residual algorithms: Reinforcement learning with function approximation. In: Prieditis A, Russell S, editors, Proceedings of the Twelfth International Conference on Machine Learning. San Francisco, CA.: Morgan Kaufmann., pp. 30-37.
26. Harmon ME, Baird LC, Klopf AH, (1995) Reinforcement learning applied to a differential game. Adaptive Behavior 4: 3-28.
27. Sutton RS (1984) Temporal credit assignment in reinforcement learning. Ph.D. thesis, UMass Amherst.
28. Dayan P, (1992) The convergence of TD(λ) for general λ. Machine learning 8: 341-362.
29. Tsitsiklis JN, Van Roy B, (1997) An analysis of temporal-difference learning with function approximation. Automatic Control, IEEE Transactions on 42: 674-690.
30. Izhikevich E, (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cerebral Cortex 17: 2443-2452.
31. Legenstein R, Pecevski D, Maass W, (2008) A learning theory for reward-modulated spike-timingdependent plasticity with application to biofeedback. PLOS Comput Biol 4: e1000180.
32. Frémaux N, Sprekeler H, Gerstner W, (2010) Functional requirements for reward-modulated spiketiming-dependent plasticity. The Journal of Neuroscience 30: 13326-13337.
33. Gerstner W, Kempter R, van Hemmen J, Wagner H, (1996) A neuronal learning rule for submillisecond temporal coding. Nature 383: 76-78.
34. Markram H, Lübke J, Frotscher M, Sakmann B, (1997) Regulation of synaptic efficacy by coincidence of postysnaptic AP and EPSP. Science 275: 213-215.
35. Bi G, Poo M, (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J Neurosci 18: 10464-10472.
36. Song S, Miller K, Abbott L, (2000) Competitive Hebbian learning through spike-time-dependent synaptic plasticity. Nature Neuroscience 3: 919-926.
37. Georgopoulos A, Kettner R, Schwartz A, (1988) Primate motor cortex and free arm movements to visual targets in three- dimensional space. II. Coding of the direction of movement by a neuronal population. J Neurosci 8: 2928-2937.
38. Pfister JP, Toyoizumi T, Barber D, Gerstner W, (2006) Optimal spike-timing-dependent plasticity for precise action potential firing in supervised learning. Neural Comp 18: 1318-1348.
39. Legenstein R, Chase SM, Schwartz AB, MaassW (2010) A reward-modulated hebbian learning rule can explain experimentally observed network reorganization in a brain control task. The Journal of Neuroscience 30: 8400-8410.
40. Boone G (1997) Minimum-time control of the acrobot. In: Robotics and Automation, Proceedings, 1997 IEEE International Conference on. Volume 4, pp. 3281-3287. doi: 10.1109/ROBOT.1997.606789.
41. Michie D, Chambers R (1968) Boxes: An experiment in adaptive control. In: Dale E, Michie D, editors, Machine Intelligence 2. Edinburgh: Oliver and Boyd. pp. 137-152.
42. Houk J, Adams J, Barto A (1995) A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Houk JC, Davis JL, Beiser DG, editors, Models on Information Processing in the Basal Ganglia, Cambridge: MIT Press. pp. 249-270.
43. Joel D, Niv Y, Ruppin E, (2002) Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Networks 15: 535-547.
44. van der Meer MAA, Redish AD, (2011) Theta phase precession in rat ventral striatum links place and reward information. The Journal of Neuroscience 31: 2843-2854.
45. Hollerman J, Schultz W, (1998) Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neuroscience 1: 304-309.
46. Potjans W, Diesmann M, Morrison A, (2011) An imperfect dopaminergic error signal can drive temporal-difference learning. PLoS Comput Biol 7: e1001133.
47. Robbins T, Roberts A, (2007) Differential regulation of fronto-executive function by the monoamines and acetylcholine. Cerebral Cortex 17: i151-i160.
48. Nakamura K, Matsumoto M, Hikosaka O, (2008) Reward-dependent modulation of neuronal activity in the primate dorsal raphe nucleus. The Journal of Neuroscience 28: 5331-5343.
49. Miyazaki K, Miyazaki KW, Doya K, (2011) Activation of dorsal raphe serotonin neurons underlies waiting for delayed rewards. The Journal of Neuroscience 31: 469-479.
50. Cohen JY, Haesler S, Vong L, Lowell BB, Uchida N, (2012) Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 482: 85-88.
51. Stroesslin T, Sheynikhovich D, Chavarriaga R, Gerstner W, (2005) Robust self-localisation and navigation based on hippocampal place cells. Neural Networks 18: 1125-1140.
52. Sheynikhovich D, Chavarriaga R, Stroesslin T, Arleo A, Gerstner W, (2009) Is there a geometric module for spatial orientation? Insights from a rodent navigation model. Psychological Review 116: 540-66.
53. Loewenstein Y, (2008) Robustness of learning that is based on covariance-driven synaptic plasticity. PLoS Comput Biol 4: e1000007.
54. Seol GH, Ziburkus J, Huang S, Song L, Kim IT, et al. (2007) Neuromodulators control the polarity of spike-timing-dependent synaptic plasticity. Neuron 55: 919-929.
55. Klopf A (1982) The hedonistic neuron: a theory of memory, learning, and intelligence. Washington: Hemisphere.
56. Sutton JP, Beis JS, Trainor LEH, (1988) Hierarchical model of memory and memory loss. J Phys A 21: 4443-4454.
57. Frey U, Morris R, (1997) Synaptic tagging and long-term potentiation. Nature 385: 533-536.
58. Clopath C, Ziegler L, Vasilaki E, Buesing L, Gerstner W, (2008) Tag-trigger-consolidation: A model of early and late long-term-potentiation and depression. PLoS Comput Biol 4: e1000248.
59. Wickens JR, Kotter R (1995) Cellular models of reinforcement. In: Houk J, Davis J, Beiser DG, editors, Models of information processing in basal ganglia, Cambridge: MIT-Press. pp. 187-214.
60. Gerstner W, Kistler WK (2002) Spiking Neuron Models. Cambridge UK: Cambridge University Press.
61. Jolivet R, Rauch A, Lüscher HR, Gerstner W, (2006) Predicting spike timing of neocortical pyramidal neurons by simple threshold models. J Computational Neuroscience 21: 35-49.
62. Florian RV (2007) Correct equations for the dynamics of the cart-pole system. Technical report, Center for Cognitive and Neural Studies (Coneural), Romania.