On the asymptotic equivalence between differential Hebbian and temporal difference learning

Neural Computation

Volumn 21, Issue 4, 2009, Pages 1173-1202

  Kolodziejski, Christoph ^a   Porr, Bernd ^b   Wörgötter, Florentin ^a  

^a UNIVERSITY OF GÖTTINGEN   (Germany)

^b UNIVERSITY OF GLASGOW   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL; ARTICLE; BIOLOGICAL MODEL; BIOPHYSICS; COMPUTER SIMULATION; LEARNING; NERVE CELL; PHYSIOLOGY; REINFORCEMENT; TIME;

ANIMALS; BIOPHYSICS; COMPUTER SIMULATION; LEARNING; MODELS, NEUROLOGICAL; NEURONS; REINFORCEMENT (PSYCHOLOGY); TIME FACTORS;

EID: 65549116541     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2008.04-08-750     Document Type: Letter

References (42)

1. Baird, L. (1993). Advantage updating (Tech. Rep. WL-TR-93-1146). Ohio: Wright Laboratory, Wright-Patterson Air Force Base.
2. Baxter, J., Bartlett, P. L., & Weaver, L. (2001). Experiments with infinite-horizon, policy-gradient estimation. Journal of Artificial Intelligence Research, 15, 351-381.
3. Dayan, P. (2002). Matters temporal. Trends in Cognitive Sciences, 6(3), 105-106.
4. Dayan, P., & Sejnowski, T. (1994). TD(λ) converges with probability 1. Mach. Learn., 14(3), 295-301.
5. Doya, K. (1996). Temporal difference learning in continuous time and space. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 1073-1079). Cambridge, MA: MIT Press.
6. Doya, K. (2000). Complementary roles of basal ganglia and cerebellum in learning and motor control. Current Opinion in Neurobiology, 10(6), 732-739.
7. Dudek, S., & Bear, M. (1992). Homosynaptic long-term depression in area CA1 of hippocampus and effects of N-methyl-D-aspartate receptor blockade. Proceedings of the National Academy of Sciences, 89(10), 4363-4367.
8. Florian, R. V. (2007). Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Computation, 19, 1468-1502.
9. Gerstner, W., Kempter, R., van Hemmen, L., & Wagner, H. (1996). A neuronal learning rule for sub-millisecond temporal coding. Nature, 383, 76-78.
10. Graybiel, A. (1998). The basal ganglia and chunking of action repertoires. Neurobiol. Learn. Mem., 70(1-2), 119-136.
11. Hassani, O. K., Cromwell, H. C., & Schultz, W. (2001). Influence of expectation of different rewards on behavior-related neuronal activity in the striatum. J. Neurophysiol., 85(6), 2477-2489.
12. Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79, 2554-2558.
13. Izhikevich, E. (2007). Solving the distal reward problem through linkage of STDP and dopamine signaling. Cerebral Cortex, 17, 2443-2452.
14. Klopf, A. H. (1988). A neuronal model of classical conditioning. Psychobiol., 16(2), 85-123.
15. Kolodziejski, C., Porr, B., & Wörgötter, F. (2008). Mathematical properties of neuronal TD-rules and differential Hebbian learning: A comparison. Biological Cybernetics, 98(3), 259-272.
16. Kosco, B. (1986). Differential Hebbian learning. In J. S. Denker (Ed.), Neural networks for computing: AIP Conference Proc. proceedings (Vol. 151). New York: American Institute of Physics.
17. Kushner, H. K., & Clark, D. S. (1978). Stochastic approximation for constrained and unconstrained systems. Berlin: Springer-Verlag.
18. Linsker, R. (1988). Self-organisation in a perceptual network. Computer, 21(3), 105-117.
19. Markram, H., Lübke, J., Frotscher, M., & Sakmann, B. (1997). Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science, 275, 213-215.
20. Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience, 76(5), 1936-1947.
21. Morris, G., Arkadir, D., Nevet, A., Vaadia, E., & Bergman, H. (2004). Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron, 43(1), 133-143.
22. Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006). Midbrain dopamine neurons encode decisions for future action. Nature Neuroscience, 9(8), 1057-1063.
23. Oja, E. (1982). A simplified neuron model as a principal component analyzer. J. Math. Biol., 15(3), 267-273.
24. Pawlak, V., & Kerr, J. N. D. (2008). Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J. Neurosci., 28(10), 2435-2446.
25. Porr, B., & Wörgötter, F. (2003). Isotropic-sequence-order learning in a closed-loop behavioural system. Phil. Trans. R. Soc. Lond. A, 361, 2225-2244.
26. Porr, B., & Wörgötter, F. (2007). Learning with "relevance": Using a third factor to stabilise Hebbian learning. Neural Comp., 19, 2694-2719.
27. Potjans, W., Morrison, A., & Diesmann, M. (2009). A spiking neural network model of an actor-critic learning agent. Neural Computation, 21(2), 301-339.
28. Rao, R., & Sejnowski, T. (2001). Spike-timing-dependent Hebbian plasticity as temporal difference learning. Neural Computation, 13, 2221-2237.
29. Redgrave, P., & Gurney, K. (2006). The short-latency dopamine signal: A role in discovering novel actions? Nature Reviews Neuroscience, 7, 967-975.
30. Roberts, P. (1999). Computational consequences of temporally asymmetric learning rules: I. Differential Hebbian learning. J. Comput. Neurosci., 7(3), 235-246.
31. Roberts, P., Santiago, R., & Lafferriere, G. (2008). An implementation of reinforcement learning based on spike-timing dependent plasticity. Biological Cybernetics, 99(6), 517-523.
32. Schultz, W. (1998). Predictive reward signal of dopamine neurons. J. Neurophysiol., 80, 1-27.
33. Schultz, W., Apicella, P., Scarnati, E., & Ljungberg, T. (1992). Neuronal activity in monkey ventral striatum related to the expectation of reward. J. Neurosci., 12(12), 4595-4610.
34. Singh, S. P., Jaakkola, T., Littman, M. L., & Szepesvári, C. (2000). Convergence results for single-step on-policy reinforcement-learning algorithms. Machine Learning, 38(3), 287-308.
35. Sutton, R. S. (1988). Learning to predict by the method of temporal differences. Machine Learning, 3, 9-44.
36. Sutton, R., & Barto, A. (1981). Towards a modern theory of adaptive networks: Expectation and prediction. Psychol. Review, 88, 135-170.
37. Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction.Cambridge, MA: MIT Press.
38. Tamosiunaite, M., Ainge, J., Kulvicius, T., Porr, B., Dudchenko, P., & Wörgötter, F. (2008). Path-finding in real and simulated rats: On the usefulness of forgetting and frustration for navigation learning. J. Comp. Neuroscience, 25, 562-582.
39. Tsitsiklis, J. N., & Van Roy, B. (1997). An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42(5), 674-690.
40. Watkins, C., & Dayan, P. (1992). Technical note: Q-learning. Mach. Learn., 8, 279-292.
41. Widrow, B., & Hoff, M. E. (1960). Adaptive switching circuits. In IRE WESCON Convention Record (pp. 96-104). New York: Institute of Radio Engineers.
42. Wiering, M. (2004). Convergence and divergence in standard averaging reinforcement learning. In J. Boulicaut, F. Esposito, F. Giannotti, & D. Pedreschi (Eds.), Proceedings of the 15th European Conference on Machine learning ECML'04 (pp. 477-488). Berlin: Springer-Verlag.