Mathematical properties of neuronal TD-rules and differential Hebbian learning: A comparison

Biological Cybernetics

Volumn 98, Issue 3, 2008, Pages 259-272

  Kolodziejski, Christoph ^a   Porr, Bernd ^b   Wörgötter, Florentin ^a  

^a UNIVERSITY OF GÖTTINGEN   (Germany)

^b UNIVERSITY OF GLASGOW   (United Kingdom)

Author keywords

Correlation based learning; Differential Hebbian learning; Reinforcement learning; Temporal difference learning

Indexed keywords

CONVERGENCE OF NUMERICAL METHODS; CORRELATION METHODS; MATHEMATICAL MORPHOLOGY; NEURAL NETWORKS; TEMPORAL LOGIC;

CORRELATION-BASED LEARNING; DIFFERENTIAL HEBBIAN LEARNING; MATHEMATICAL THEORY; TEMPORAL DIFFERENCE LEARNING;

REINFORCEMENT LEARNING;

CORDEAUXIA EDULIS;

ACTION POTENTIAL; ANIMAL; ARTICLE; ARTIFICIAL NEURAL NETWORK; HUMAN; LEARNING; MATHEMATICS; NERVE CELL; NERVE CELL PLASTICITY; PHYSIOLOGY; TIME;

ACTION POTENTIALS; ANIMALS; HUMANS; LEARNING; MATHEMATICS; NEURAL NETWORKS (COMPUTER); NEURONAL PLASTICITY; NEURONS; TIME FACTORS;

EID: 40149107540     PISSN: 03401200     EISSN: 14320770     Source Type: Journal    
DOI: 10.1007/s00422-007-0209-6     Document Type: Article

References (56)

1. Arleo A, Gerstner W (2000) Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity. Biol Cybern 83(3): 287-299
2. Barto A (1995) Adaptive critics and the basal ganglia. In: Houk JC, Davis JL, Beiser DG(eds) Models of information processing in the basal ganglia. MIT Press, Cambridge, pp 215-232
3. Bi GQ, Poo M (2001) Synaptic modification by correlated activity: Hebb's postulate revisited. Annu Rev Neurosci 24: 139-166
4. Boykina TB (2003) Derivatives of the dirac delta function by explicit construction of sequences. Am J Phys 71(5): 462-468
5. Dayan P (1992) The convergence of TD(λ). Mach Learn 8(3/4): 341-362
6. Dayan P, Abbott L (2003) Theoretical Neuroscience, Computational and mathematical modeling of neural systems. MIT Press, Cambridge
7. Dayan P, Seynowski T (1994) TD(λ) converges with probability 1. Mach Learn 14: 295-301
8. Florian RV (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput 19: 1468-1502
9. Foster DJ, Morris RGM, Dayan P (2000) A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus 10: 1-16
10. Gerstner W, Kempter R, van Hemmen JL, Wagner H (1996) A neuronal learning rule for sub-millisecond temporal coding. Nature 383: 76-78
11. Hull CL (1939) The problem of stimulus equivalence in behavior theory. Psychol Rev 46: 9-30
12. Hull CL (1943) Principles of behavior. Appleton Century Crofts, New York
13. Humeau Y, Shaban H, Bissiere S, Luthi A (2003) Presynaptic induction of heterosynaptic associative plasticity in the mammalian brain. Nature 426(6968): 841-845
14. Izhikevich E (2007) Solving the distal reward problem through linkage of stdp and dopamine signaling. Cerebral Cortex 101093/cercor/bhl152
15. Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: a survey. J Artif Intell Res 4: 237-285
16. Klopf AH (1972) Brain function and adaptive systems-a heterostatic theory. Technical report, Air Force Cambridge Research Laboratories Special Report No. 133, Defense Technical Information Center, Cameron Station, Alexandria, VA 22304
17. Klopf AH (1982) The hedonistic neuron: a theory of memory, learning, and intelligence. Hemisphere, Washington DC
18. Klopf AH (1986) A drive-reinforcement model of single neuron function. In: Denker JS (ed) Neural networks for computing: AIP Conference Proceedings. American Institute of Physics, New York, vol 151
19. Klopf AH (1988) A neuronal model of classical conditioning. Psychobiology 16(2): 85-123
20. Kolodziejski C, Porr B, Wörgötter F (2006) Fast, flexible and adaptive motor control achieved by pairing neuronal learning with recruitment. In: Proceedings of the fifteenth annual computational neuroscience meeting CNS, Edinburgh
21. Kolodziejski C, Porr B, Wörgötter F (2007) Anticipative adaptive muscle control: Forward modeling with self-induced disturbances and recruitment. In: Proceedings of the fifteenth annual computational neuroscience meeting CNS, Toronto
22. Kosco B (1986) Differential Hebbian learning. In: Denker JS (eds) Neural networks for computing: AIP Conference proceedings, American Institute of Physics, New York, vol 151
23. Krichmar JL, Seth AK, Nitz DA, Fleischer JG, Edelman GM (2005) Spatial navigation and causal analysis in a brain-based device modeling cortical-hippocampal interactions. Neuroinformatics 3: 197-222
24. T Kulvicius B Porr F Wörgötter 2007 Chained learning architectures in a simple closed-loop behavioural context Biol Cybern 10.1007/s00422-007-0176-y Kulvicius T, Porr B, Wörgötter F (2007) Chained learning architectures in a simple closed-loop behavioural context. Biol Cybern. doi: 10.1007/s00422-007-0176-y
25. Magee JC, Johnston D (1997) A synaptically controlled, associative signal for Hebbian plasticity in hippocampal neurons. Science 275: 209-213
26. Manoonpong P, Geng T, Kulvicius T, Porr B, Wörgötter F (2007) Adaptive, fast walking in a biped robot under neuronal control and learning. PLoS Comput Biol 3(7): e134
27. Markram H, Lübke J, Frotscher M, Sakmann B (1997) Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science 275: 213-215
28. Miller JD, Sanghera MK, German DC (1981) Mesencephalic dopaminergic unit activity in the behaviorally conditioned rat. Life Sci 29: 1255-1263
29. Montague PR, Dayan P, Person C, Sejnowski TJ (1995) Bee foraging in uncertain environments using predictive hebbian learning. Nature 377: 725-728
30. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive hebbian learning. J Neurosci 16(5): 1936-1947
31. Pfister JP, Toyoizumi T, Barber D, Gerstner W (2006) Optimal spike-timing dependent plasticity for precise action potential firing in supervised learning. Neural Comput 18: 1309-1339
32. Porr B, Wörgötter F (2003) Isotropic sequence order learning. Neural Comput 15: 831-864
33. Porr B, Wörgötter F (2006) Strongly improved stability and faster convergence of temporal sequence learning by utilising input correlations only. Neural Comput 18: 1380-1412
34. Porr B, Wörgötter F (2007) Learning with "relevance": Using a third factor to stabilise hebbian learning. Neural Comput (in press)
35. Porr B, von Ferber C, Wörgötter F (2003) ISO-learning approximates a solution to the inverse-controller problem in an unsupervised behavioral paradigm. Neural Comput 15: 865-884
36. Roberts P (1999) Computational consequences of temporally asymmetric learning rules: I. differential hebbian learning. J Comput Neurosci 7(3): 235-46
37. Santiago RA, Roberts PD, Lafferriere G (2007) Spike timing dependent plasticity implements reinforcement learning. In: Proceedings of the fifteenth annual computational neuroscience meeting CNS, Toronto
38. Saudargiene A, Porr B, Wörgötter F (2004) How the shape of pre- and postsynaptic signals can influence STDP: a biophysical model. Neural Comp 16: 595-626
39. Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80: 1-27
40. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275: 1593-1599
41. Singh SP, Sutton RS (1996) Reinforcement learning with replacing eligibility traces. Mach Learn 22: 123-158
42. Strösslin T, Sheynikhovich D, Chavarriaga R, Gerstner W (2005) Robust self-localisation and navigation based on hippocampal place cells. Neural Netw 18(9): 1125-1140
43. Suri RE (2002) TD models of reward predictive responses in Dopamine neurons. Neural Netw 15(4-6): 523-533
44. Suri RE, Schultz W (1998) Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp Brain Res 121: 350-354
45. Suri RE, Schultz W (1999) A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neurosci 91(3): 871-890
46. Suri RE, Schultz W (2001) Temporal difference model reproduces anticipatory neural activity. Neural Comp 13(4): 841-62
47. Suri RE, Bargas J, Arbib MA (2001) Modeling functions of striatal dopamine modulation in learning and planning. Neurosci 103(1): 65-85
48. Sutton R, Barto A (1981) Towards a modern theory of adaptive networks: Expectation and prediction. Psychol Rev 88: 135-170
49. Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3: 9-44
50. Sutton RS, Barto AG (1990) Time-derivative models of Pavlovian reinforcement. In: Gabriel M, Moore J(eds) Learning and computational neuroscience: foundation of adaptive networks. MIT Press, Cambridge
51. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction, 2002nd edn. Bradford Books, MIT Press, Cambridge
52. Tsukamoto M, Yasui T, Yamada MK, Nishiyama N, Matsuki N, Ikegaya Y (2003) Mossy fibre synaptic NMDA receptors trigger non-Hebbian long-term potentiation at entorhino-CA3 synapses in the rat. J Physiol 546(3): 665-675
53. Watkins CJCH (1989) Learning from delayed rewards. PhD thesis, University of Cambridge, Cambridge
54. Watkins CJCH, Dayan P (1992) Technical note: Q-Learning. Mach Learn 8: 279-292
55. Witten IH (1977) An adaptive optimal controller for discrete-time Markov environments. Inf Control 34: 86-295
56. Wörgötter F, Porr B (2005) Temporal sequence learning for prediction and control - a review of different models and their relation to biological mechanisms. Neural Comput 17: 245-319