Short-term plasticity as cause–effect hypothesis testing in distal reward learning

Biological Cybernetics

Volumn 109, Issue 1, 2015, Pages 75-94

  Soltoggio, Andrea ^a  

^a LOUGHBOROUGH UNIVERSITY   (United Kingdom)

Author keywords

Distal reward; Memory consolidation; Operant learning; Plasticity versus stability; Short term plasticity; Transient weights

Indexed keywords

PLASTICITY; TOPOLOGY;

DISTAL REWARD; HYPOTHESIS TESTING; INFORMATION FLOWS; LONG TERM CHANGE; MEMORY CONSOLIDATION; NETWORK TOPOLOGY; OPERANT LEARNING; SHORT TERM PLASTICITY;

PLASTICITY TESTING;

ANIMAL; BIOLOGICAL MODEL; HUMAN; LEARNING; MEMORY; NERVE CELL PLASTICITY; PHYSIOLOGY; REWARD; SYNAPSE; TIME;

ANIMALS; HUMANS; LEARNING; MEMORY; MODELS, NEUROLOGICAL; NEURONAL PLASTICITY; REWARD; SYNAPSES; TIME FACTORS;

EID: 84922260944     PISSN: 03401200     EISSN: 14320770     Source Type: Journal    
DOI: 10.1007/s00422-014-0628-0     Document Type: Article

References (97)

1. Abbott LF, Regehr WG (2004) Synaptic computation. Nature 431:796–803
2. Abraham WC (2008) Metaplasticity: tuning synapses and networks for plasticity. Nat Rev Neurosci 9:387–399
3. Abraham WC, Bear MF (1996) Metaplasticity: the plasticity of synaptic plasticity. Trends Neurosci 19:126–130
4. Abraham WC, Robins A (2005) Memory retention—the synaptic stability versus plasticity dilemma. Trends Neurosci 28:73–78
5. Alexander WH, Sporns O (2002) An embodied model of learning, plasticity, and reward. Adapt Behav 10(3–4):143–159
6. Asada M, Hosoda K, Kuniyoshi Y, Ishiguro H, Inui T, Yoshikawa Y, Ogino M, Yoshida C (2009) Cognitive developmental robotics: a survey. IEEE Trans Auton Mental Dev 1(1):12–34
7. Bailey CH, Giustetto M, Huang YY, Hawkins RD, Kandel ER (2000) Is heterosynaptic modulation essential for stabilizing Hebbian plasticity and memory? Nat Rev Neurosci 1(1):11–20
8. Baras D, Meir R (2007) Reinforcement learning, spike-time-dependent plasticity, and the BCM rule. Neural Comput 19(8):2245–2279
9. Ben-Gal I (2007) Bayesian networks. In: Encyclopedia of statistics in quality and reliability. Wiley, London
10. Berridge KC (2007) The debate over dopamine’s role in reward: the case for incentive salience. Psychopharmacology 191:391–431
11. Bosman R, van Leeuwen W, Wemmenhove B (2004) Combining Hebbian and reinforcement learning in a minibrain model. Neural Netw 17:29–36
12. Bouton ME (1994) Conditioning, remembering, and forgetting. J Exp Psychol Anim Behav Process 20(3):219
13. Bouton ME (2000) A learning theory perspective on lapse, relapse, and the maintenance of behavior change. Health Psychol 19(1S):57
14. Bouton ME (2004) Context and behavioral processes in extinction. Learn Mem 11(5):485–494
15. Bouton ME, Moody EW (2004) Memory processes in classical conditioning. Neurosci Biobehav Rev 28(7):663–674
16. Brembs B (2003) Operant conditioning in invertebrates. Curr Opin Neurobiol 13(6):710–717
17. Brembs B, Lorenzetti FD, Reyes FD, Baxter DA, Byrne JH (2002) Operant reward learning in Aplysia: neuronal correlates and mechanisms. Science 296(5573):1706–1709
18. Clopath C, Ziegler L, Vasilaki E, Büsing L, Gerstner W (2008) Tag-trigger-consolidation: a model of early and late long-term-potentiation and depression. PLoS Comput Biol 4(12):335–347
19. Cox RB, Krichmar JL (2009) Neuromodulation as a robot controller: a brain inspired strategy for controlling autonomous robots. IEEE Robot Autom Mag 16(3):72–80
20. Deco G, Rolls ET (2005) Synaptic and spiking dynamics underlying reward reversal in the orbitofrontal cortex. Cereb Cortex 15:15–30
21. Dudai Y (2004) The neurobiology of consolidations, or, how stable is the engram? Annu Rev Psychol 55:51–86
22. Farries MA, Fairhall AL (2007) Reinforcement learning with modulated spike timing-dependent synaptic plasticity. J Neurophysiol 98:3648–3665
23. Fisher SA, Fischer TM, Carew TJ (1997) Multiple overlapping processes underlying short-term synaptic enhancement. Trends Neurosci 20(4):170–177
24. Florian RV (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput 19:1468–1502
25. Frémaux N, Sprekeler H, Gerstner W (2010) Functional requirements for reward-modulated spike-timing-dependent plasticity. J Neurosci 30(40):13,326–13,337
26. Frey U, Morris RGM (1997) Synaptic tagging and long-term potentiation. Nature 385:533–536
27. Friedrich J, Urbanczik R, Senn W (2010) Learning spike-based population codes by reward and population feedback. Neural Comput 22:1698–1717
28. Friedrich J, Urbanczik R, Senn W (2011) Spatio-temporal credit assignment in neuronal population learning. PLoS Comput Biol 7(6):1–13
29. Fusi S, Senn W (2006) Eluding oblivion with smart stochastic selection of synaptic updates. Chaos An Interdiscip J Nonlinear Sci 16(2):026,112
30. Fusi S, Drew PJ, Abbott L (2005) Cascade models of synaptically stored memories. Neuron 45(4):599–611
31. Fusi S, Asaad WF, Miller EK, Wang XJ (2007) A neural circuit model of flexible sensorimotor mapping: learning and forgetting on multiple timescales. Neuron 54(2):319–333
32. Garris P, Ciolkowski E, Pastore P, Wighmann R (1994) Efflux of dopamine from the synaptic cleft in the nucleus accumbens of the rat brain. J Neurosci 14(10):6084–6093
33. Gerstner W (2010) From Hebb rules to spike-timing-dependent plasticity: a personal account. Front Synaptic Neurosci 2:1–3
34. Gil M, DeMarco RJ, Menzel R (2007) Learning reward expectations in honeybees. Learn Mem 14:291–496
35. Goelet P, Castellucci VF, Schacher S, Kandel ER (1986) The long and the short of long-term memory—a molecular framework. Nature 322(6078):419–422
36. Grossberg S (1971) On the dynamics of operant conditioning. J Theor Biol 33(2):225–255
37. Grossberg S (1988) Nonlinear neural networks: principles, mechanisms, and architectures. Neural Netw 1:17–61
38. Hamilton RH, Pascual-Leone A (1998) Cortical plasticity associated with braille learning. Trends Cogn Sci 2(5):168–174
39. Hammer M, Menzel R (1995) Learning and memory in the honeybee. J Neurosci 15(3):1617–1630
40. Heckerman D, Geiger D, Chickering DM (1995) Learning bayesian networks: the combination of knowledge and statistical data. Mach Learn 20:197–243
41. Howson C, Urbach P (1989) Scientific reasoning: the Bayesian approach. Open Court Publishing Co, Chicago, USA
42. Hull CL (1943) Principles of behavior. Appleton Century, New York
43. Izhikevich EM (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17:2443–2452
44. Jay MT (2003) Dopamine: a potential substrate for synaptic plasticity and memory mechanisms. Prog Neurobiol 69(6):375–390
45. Jonides J, Lewis RL, Nee DE, Lustig CA, Berman MG, Moore KS (2008) The mind and brain of short-term memory. Annu Rev Psychol 59:193
46. Kempter R, Gerstner W, Van Hemmen JL (1999) Hebbian learning and spiking neurons. Phys Rev E 59(4):4498–4514
47. Krichmar JL, Roehrbein F (2013) Value and reward based learning in neurorobots. Front Neurorobot 7(13):1–2
48. Lamprecht R, LeDoux J (2004) Structural plasticity and memory. Nat Rev Neurosci 5(1):45–54
49. Legenstein R, Chase SM, Schwartz A, Maass W (2010) A reward-modulated Hebbian learning rule can explain experimentally observed network reorganization in a brain control task. J Neurosci 30(25):8400–8401
50. Leibold C, Kempter R (2008) Sparseness constrains the prolongation of memory lifetime via synaptic metaplasticity. Cereb Cortex 18(1):67–77
51. Lin LJ (1993) Reinforcement learning for robots using neural networks. Ph.D. thesis, School of Computer Science. Carnegie Mellon University
52. Lungarella M, Metta G, Pfeifer R, Sandini G (2003) Developmental robotics: a survey. Connect Sci 15(4):151–190
53. Lynch MA (2004) Long-term potentiation and memory. Physiol Rev 84(1):87–136
54. Mayford M, Siegelbaum SA, Kandel ER (2012) Synapses and memory storage. Cold Spring Harbor Perspect Biol 4(6):a005,751
55. McGaugh JL (2000) Memory—a century of consolidation. Science 287:248–251
56. Menzel R, Müller U (1996) Learning and memory in honeybees: from behavior to natural substrates. Annu Rev Neurosci 19:179–404
57. Montague PR, Dayan P, Person C, Sejnowski TJ (1995) Bee foraging in uncertain environments using predictive Hebbian learning. Nature 377:725–728
58. Nguyen PV, Abel T, Kandel ER (1994) Requirement of a critical period of transcription for induction of a late phase of ltp. Science 265(5175):1104–1107
59. Nitz DA, Kargo WJ, Fleisher J (2007) Dopamine signaling and the distal reward problem. Learn Mem 18(17):1833–1836
60. O’Brien MJ, Srinivasan N (2013) A spiking neural model for stable reinforcement of synapses based on multiple distal rewards. Neural Comput 25(1):123–156
61. O’Doherty JP, Kringelbach ML, Rolls ET, Andrews C (2001) Abstract reward and punishment representations in the human orbitofrontal cortex. Nat Neurosci 4(1):95–102
62. Ono K (1987) Superstitious behavior in humans. J Exp Anal Behav 47(3):261–271
63. Päpper M, Kempter R, Leibold C (2011) Synaptic tagging, evaluation of memories, and the distal reward problem. Learn Mem 18:58–70
64. Pennartz CMA (1996) The ascending neuromodulatory systems in learning by reinforcement: comparing computational conjectures with experimental findings. Brain Res Rev 21:219–245
65. Pennartz CMA (1997) Reinforcement learning by hebbian synapses with adaptive threshold. Neuroscience 81(2):303–319
66. Redgrave P, Gurney K, Reynolds J (2008) What is reinforced by phasic dopamine signals? Brain Res Rev 58:322–339
67. Robins A (1995) Catastrophic forgetting, rehearsal, and pseudorehearsal. Connect Sci J Neural Comput Artif Intell Cogn Res 7:123–146
68. Sandberg A, Tegnér J, Lansner A (2003) A working memory model based on fast hebbian learning. Netw Comput Neural Syst 14(4):789–802
69. Sarkisov DV, Wang SSH (2008) Order-dependent coincidence detection in cerebellar Purkinje neurons at the inositol trisphosphate receptor. J Neurosci 28(1):133–142
70. Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80:1–27
71. Schultz W, Apicella P, Ljungberg T (1993) Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J Neurosci 13:900–913
72. Schultz W, Dayan P, Montague PR (1997) A neural substrate for prediction and reward. Science 275:1593–1598
73. Senn W, Fusi S (2005) Learning only when necessary: better memories of correlated patterns in networks with bounded synapses. Neural Comput 17(10):2106–2138
74. Skinner BF (1948) “Superstition” in the pigeon. J Exp Psychol 38:168–172
75. Skinner BF (1953) Science and human behavior. MacMillan, New York
76. Soltoggio A, Stanley KO (2012) From modulated Hebbian plasticity to simple behavior learning through noise and weight saturation. Neural Netw 34:28–41
77. Soltoggio A, Steil JJ (2013) Solving the distal reward problem with rare correlations. Neural Comput 25(4):940–978
78. Soltoggio A, Bullinaria JA, Mattiussi C, Dürr P, Floreano D (2008) Evolutionary advantages of neuromodulated plasticity in dynamic, reward-based scenarios. In: Artificial life XI: proceedings of the eleventh international conference on the simulation and synthesis of living systems. MIT Press, Cambridge
79. Soltoggio A, Lemme A, Reinhart FR, Steil JJ (2013a) Rare neural correlations implement robotic conditioning with reward delays and disturbances. Front Neurorobot 7:1–16 (Research Topic: Value and Reward Based Learning in Neurobots)
80. Soltoggio A, Reinhart FR, Lemme A, Steil JJ (2013b) Learning the rules of a game: neural conditioning in human–robot interaction with delayed rewards. In: Proceedings of the third joint IEEE international conference on development and learning and on epigenetic robotics, Osaka, Japan
81. Sporns O, Alexander WH (2002) Neuromodulation and plasticity in an autonomous robot. Neural Netw 15:761–774
82. Sporns O, Alexander WH (2003) Neuromodulation in a learning robot: interactions between neural plasticity and behavior. Proc Int Joint Conf Neural Netw 4:2789–2794
83. Staubli U, Fraser D, Faraday R, Lynch G (1987) Olfaction and the “data” memory system in rats. Behav Neurosci 101(6):757–765
84. Sutton RS (1984) Temporal credit assignment in reinforcement learning. Ph.D. thesis, Department of Computer Science, University of Massachusetts, Amherst, MA 01003
85. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge, MA, USA
86. Swartzentruber D (1995) Modulatory mechanisms in pavlovian conditioning. Anim Learn Behav 23(2):123–143
87. Thorndike EL (1911) Animal intelligence. Macmillan, New York
88. Urbanczik R, Senn W (2009) Reinforcement learning in populations of spiking neurons. Nat Neurosci 12:250–252
89. Van Hemmen J (1997) Hebbian learning, its correlation catastrophe, and unlearning. Netw Comput Neural Syst 8(3):V1–V17
90. Wang SSH, Denk W, Häusser M (2000) Coincidence detection in single dendritic spines mediated by calcium release. Nat Neurosci 3(12):1266–1273
91. Weng J, McClelland J, Pentland A, Sporns O, Stockman I, Sur M, Thelen E (2001) Autonomous mental development by robots and animals. Science 291(5504):599–600
92. Wighmann R, Zimmerman J (1990) Control of dopamine extracellular concentration in rat striatum by impulse flow and uptake. Brain Res Brain Res Rev 15(2):135–144
93. Wise RA, Rompre PP (1989) Brain dopamine and reward. Annu Rev Psychol 40:191–225
94. Xie X, Seung HS (2004) Learning in neural networks by reinforcement of irregular spiking. Phys Rev E 69:1–10
95. Ziemke T, Thieme M (2002) Neuromodulation of reactive sensorimotor mappings as short-term memory mechanism in delayed response tasks. Adapt Behav 10:185–199
96. Zucker RS (1989) Short-term synaptic plasticity. Annu Rev Neurosci 12(1):13–31
97. Zucker RS, Regehr WG (2002) Short-term synaptic plasticity. Annu Rev Physiol 64(1):355–405