A spiking neural network model of an actor-critic learning agent

Neural Computation

Volumn 21, Issue 2, 2009, Pages 301-339

  Potjans, Wiebke ^a   Morrison, Abigail ^a   Diesmann, Markus ^a,b  

^a RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

^b UNIVERSITY OF FREIBURG   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ACTION POTENTIAL; ALGORITHM; ANIMAL; ARTICLE; ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; COMPUTER SIMULATION; HUMAN; LEARNING; NERVE CELL; PHYSIOLOGY; REINFORCEMENT; TIME;

ACTION POTENTIALS; ALGORITHMS; ANIMALS; COMPUTER SIMULATION; HUMANS; LEARNING; MODELS, NEUROLOGICAL; NEURAL NETWORKS (COMPUTER); NEURONS; REINFORCEMENT (PSYCHOLOGY); TIME FACTORS;

EID: 67650298948     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2008.08-07-593     Document Type: Article

References (76)

1. Amit, D. J. (1989). Modeling brain function. Cambridge: Cambridge University Press.
2. Antonov, I., Antonova, I., Kandel, E. R., & Hawkins, R. D. (2003). Activity-dependent presynaptic facilitation and Hebbian LTP are both required and interact during classical conditioning in Aplysia. Neuron, 37(1), 135-147.
3. Attalah, H. E., Lopez-Paniagua, D., Rudy, J. W., & O'Reilly, R. C. (2007). Separate neural substrates for skill-learning and performance in the ventral and dorsal striatum. Nat. Neurosci., 10(1), 126-131.
4. Baras, D., & Meir, R. (2007). Reinforcement learning, spike-time-dependent plasticity, and the BCM rule. Neural Comput., 19, 2245-2279.
5. Barto, A. G. (1995). Adaptive critic and the basal ganglia. In J. C. Houk, J. Davis, & D. Beisser (Eds.), Models of information processing in the basal ganglia (pp. 215-232). Cambridge, MA: MIT Press.
6. Barto, A., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Sys. M. Cybern., 13, 834-846.
7. Bertsekas, D. P., & Tsitsiklis, J. N. (1996). Neuro-dynamic programming. Belmont, MA: Athena Scientific.
8. Bi, G.-q., & Poo, M.-m. (1998). Synaptic modifications in cultured hippocampal neurons: Dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci., 18, 10464-10472.
9. Bonhoeffer, T., Staiger, V., & Aertsen, A. (1989). Synaptic plasticity in rat hippocampal slice cultures: local "Hebbian" conjunction of pre- and postsynaptic stimulation leads to distributed synaptic enhancement. Proc. Natl. Acad. Sci. USA, 86(20), 8113-8117.
10. Cash, S., Zucker, R., & Poo, M.-m. (1996). Spread of synaptic depression mediated by presynaptic cytoplasmic signaling. Science, 272(5264), 998-1001.
11. Dayan, P. (1992). The convergence of TD(λ) for general λ. Machine Learning, 8, 341- 362.
12. Dayan, P., & Sejnowski, T. (1994). TD(λ) converges with probability 1. Machine Learning, 14, 295-301.
13. Doya, K. (2000a). Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr. Opin. Neurobiol., 10, 732-739.
14. Doya, K. (2000b). Reinforcement learning in continuous time and space. Neural Comput., 12(1), 219-245.
15. Doya, K. (2002). Metalearning and neuromodulation. Neural Networks, 15, 495-506.
16. Farries,M. A., & Fairhall, A. L. (2007). Reinforcement learning with modulated spike timing-dependent synaptic plasticity. J. Neurophysiol., 98, 3648-3665.
17. Fitzsimonds, R., Song, H.-j., & Poo, M.-m. (1997). Propagation of activity-dependent synaptic depression in simple neural networks. Nature, 388, 439-448.
18. Florian, R. V. (2007). Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput., 19, 1468-1502.
19. Foster, D. J., Morris, R. G. M., & Dayan, P. (2000). A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus, 10, 1-16.
20. Froemke, R. C., & Dan, Y. (2002). Spike-timing-dependent synaptic modification induced by natural spike trains. Nature, 416(6879), 433-438.
21. Gazeres, N., Borg-Graham, L., & Frégnac, Y. (1998). Phenomenological model of visually evoked spike trains in cat geniculate nonlagged X-cells. Vis. Neurosci., 15, 1157-1174.
22. Georgopoulos, A., Kalaska, J. F., Caminiti, R., & Massey, J. T. (1982). On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex. J. Neurosci., 11(2), 1527-1537.
23. Gewaltig, M.-O., & Diesmann, M. (2007). NEST (neural simulation tool). Scholarpedia, 2(4), 1430.
24. Guerrero-Rivera, R., Morrison, A., Diesmann, M., & Pearce, T. C. (2006). Programmable logic construction kits for hyper real-time neuronal modeling. Neural Comput., 18, 2651-2679.
25. Hayon, G., Abeles, M., & Lehmann, D. (2004). Modeling compositionality by dynamic binding of synfire chains. J. Comput. Neurosci., 17, 179-201.
26. Houk, J. C., Adams, J. L.,&Barto, A. G. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. Cambridge, MA:MIT Press.
27. Izhikevich, E.M. (2007). Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb. Cortex, 17(10), 2443-2452.
28. Joel, D., Niv, J., & Ruppin, E. (2002). Actor-critic models of the basal ganglia: New anatomical and computational perspectives. Neural Networks, 15, 535-547.
29. Kempter, R., Leibold, C., Wagner, H., & van Hemmen, J. (2001). Formation of temporal-feature maps by axonal propagation of synaptic learning. Proc. Natl. Acad. Sci. USA, 7(98), 4166-4171.
30. Klopf, A. (1986). A drive-reinforcementmodel of single neuron function. In J. Denker (Ed.), Neural networks for computing: AIP Conference Proceedings (Vol. 151, pp. 265-270). New York: American Institute of Physics.
31. Klopf, A. (1988). A neuronal model of classical conditioning. Psychobiology, 16, 85-125.
32. Konda, V., & Tsitsiklis, J. (2003). On actor-critic algorithms. SIAM Journal on Control and Optimization, 42(4), 1143-1166.
33. Kosko, B. (1986). DifferentialHebbian learning. In J. Denker (Ed.), Neural networks for Computing: AIP Conference Proceedings (Vol. 151, pp. 277-288). New York: American Institute of Physics.
34. Kossel, A., Bonhoeffer, T., & Boltz, J. (1990). Non-Hebbian synapses in rat visual cortex. NeuroReport, 1(2), 115-118.
35. Leibold, C., Kempter, R., & van Hemmen, J. (2001). Temporal map formation in the barn owl's brain. Phys. Rev. Lett., 87(24), 248101.
36. Leibold, C., & van Hemmen, J. (2002). Mapping time. Biol. Cybern., 87, 428-439.
37. Lerchner, A., La Camera, G., & Richmond, B. (2007). Knowing without doing. Nat. Neurosci., 10(1), 15-17.
38. Markram, H., Lübke, J., Frotscher, M., & Sakmann, B. (1997). Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science, 275, 213-215.
39. Montague, P., Dayan, P., Person, C., & Sejnowski, T. (1995). Bee foraging in uncertain environments using predictive Hebbian learning. Nature, 377, 725-728.
40. Montague, P. R., Dayan, P., & Sejowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci., 16(5), 1936-1947.
41. Morimoto, J., & Doya, K. (2001). Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning. Robotics and Autonomous Systems, 36, 37-51.
42. Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006). Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci., 9(8), 1057-1063.
43. Morrison, A., Diesmann, M., & Gerstner, W. (2008). Phenomenological models of synaptic plasticity based on spike timing. Biol. Cybern., 98, 459-478.
44. Muller, E., Buesing, L., Schemmel, J., & Meier, K. (2007). Spike-frequency adapting neural assemblies: Beyond mean adaptation and renewal theories. Neural Comput., 19, 2958-3010.
45. Munos, R. (2006). Policy gradient in continuous time. Journal of Machine Learning Research, 7, 771-791.
46. Niv, Y., Joel,D., Meilijson, I., & Ruppin, E. (2002). Evolution of reinforcement learning in uncertain environments: A simple explanation for complex foraging behaviors. Adaptive Behavior, 10(1), 5-24.
47. O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron, 28, 329-337.
48. O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304, 452-454.
49. Pessiglione, M., Seymour, B., Flandin, G., Dolan, R., & Frith, C. (2006). Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature, 442, 1042-1045.
50. Pfister, J.-P., & Gerstner, W. (2006). Triplets of spikes in a model of spike timing-dependent plasticity. J. Neurosci., 26, 9673-9682.
51. Philipp, S., Grübl, A., Meier, K., & Schemmel, J. (2007). Interconnecting VLSI spiking neural networks using isochronous connections. In Proceedings of IWANN2007 (pp. 471-478). Berlin: Springer.
52. Porr, B., & Wörgötter, F. (2003). Isotropic sequence order learning. Neural Comput., 15, 831-864.
53. Porr, B., & Wörgötter, F. (2007). Learning with relevance: Using a third factor to stabilize Hebbian learning. Neural Comput., 19(10), 2694-2719.
54. Potjans, W., Morrison, A., & Diesmann, M. (2007a). Reinforcement learning in an actor-critic spiking network model. Neuroforum, 8(1).
55. Potjans, W., Morrison, A., & Diesmann, M. (2007b). A spiking neural networkmodel for the actor-critic temporal-difference learning algorithm. In Proceedings of the 37th SFN Meeting. San Diego, CA: Society for Neuroscience.
56. Rao, R. P. N., & Sejnowski, T. J. (2001). Spike-timing-dependent Hebbian plasticity as temporal difference learning. Neural Comput., 13, 2221-2237.
57. Reynolds, J. N., & Wickens, J. R. (2002). Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks, 15, 507-521.
58. Roberts, P. D. (1999). Computational consequences of temporally asymmetric learning rules: I. Differential Hebbian learning. J. Comput. Neurosci., 7, 235-246.
59. Schultz, W. (2002). Getting formal with dopamine and reward. Neuron, 36, 241-263.
60. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599.
61. Schuman, E., & Madison, D. (1994). Locally distributed synaptic potentiation on the hippocampus. Science, 263, 532-536.
62. Seung, H. S. (2003). Learning spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron, 40, 1063-1073.
63. Seymour, B., O'Doherty, J., Dayan, P., Koltzenburg, M., Jones, A., Dolan, R., et al. (2004). Temporal difference models describe higher-order learning in humans. Nature, 429, 664-667.
64. Suri, R., & Schultz, W. (1999). A neural network model with dopamine-like reinforcement signal that learns a spatial delayed reponse task. Neuroscience, 91(3), 871-890.
65. Suri, R. E., & Schultz, W. (2001). Temporal difference model reproduces anticipatory neural activity. Neural Comput., 13, 841-862.
66. Sutton, R. (1988). Learning to predict by methods of temporal difference. Machine Learning, 3, 9-44.
67. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
68. Tao, H.-z. W., Zhang, L. I., Bi, G.-q., & Poo, M.-m. (2000). Selective presynaptic propagation of long-term potentiation in defined neural networks. J. Neurosci., 20(9), 3233-3243.
69. Tesauro, G. (1994). TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Comput., 6(2), 215-219.
70. Tricomi, E. M., Delgado, M. R., & Fiez, J. A. (2004). Modulation of caudate activity by action contingency. Neuron, 41, 281-292.
71. VanRullen, R., Guyonneau, R., & Thorpe, S. J. (2005). Spike times make sense. TINS, 28(1), 1-4.
72. Williams, R. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229-256.
73. Witten, I. H. (1977). An adaptive optimal controller for discrete-time Markov environments. Information and Control, 34, 286-295.
74. Wörgötter, F., & Porr, B. (2005). Temporal sequence learning, prediction, and control: A review of different models and their relation to biological mechanisms. Neural Comput., 17, 245-319.
75. Xie, X., & Seung, H. S. (2004). Learning in neural networks by reinforcement of irregular spiking. Phys. Rev. E, 69, 41909.
76. Zhang, L. I., Tao, H.W., Holt, C. E., Harris, W. A., & Poo, M.-m. (1998). A critical window for cooperation and competition among developing retinotectal synapses. Nature, 395, 37-44.