From modulated Hebbian plasticity to simple behavior learning through noise and weight saturation

Neural Networks

Volumn 34, Issue , 2012, Pages 28-41

  Soltoggio, Andrea ^a   Stanley, Kenneth O ^b  

^a BIELEFELD UNIVERSITY   (Germany)

^b UNIVERSITY OF CENTRAL FLORIDA   (United States)

Author keywords

Adaptive behavior; Computational models; Learning; Neural plasticity; Neuromodulation

Indexed keywords

ADAPTIVE BEHAVIOR; COMPUTATIONAL MODEL; LEARNING; NEURAL PLASTICITY; NEUROMODULATION;

ARTIFICIAL INTELLIGENCE; COGNITIVE SYSTEMS;

MODULATION;

AGGRESSION; ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; BEHAVIOR LEARNING; FEAR; HEBBIAN PLASTICITY; LEARNING; MATHEMATICAL ANALYSIS; MATHEMATICAL MODEL; MATHEMATICAL PARAMETERS; MEMORY; NERVE CELL PLASTICITY; NEURAL NOISE; NEUROMODULATION; PRIORITY JOURNAL; REWARD; WEIGHT SATURATION;

BEHAVIOR; ELECTRICITY; LEARNING; MODELS, NEUROLOGICAL; NEURONAL PLASTICITY;

EID: 84865440918     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2012.06.005     Document Type: Article

References (101)

1. Abbott L.F. Modulation of function and gated learning in a network memory. Proceedings of the National Academy of Science of the United States of America 1990, 87(23):9241-9245.
2. Alger B.E., Nicoll R.A. Feed-forward dendritic inhibition in rat Hippocampal pyramidal cells studied in vitro. Journal Physiology 1982, 328:105-123.
3. Arkin R.C. Behavior-based robotics 1998, MIT Press.
4. Aston-Jones G., Cohen J.D. Adaptive gain and the role of the locus Coeruleus-Norepinephine system in optimal performance. The Journal of Comparative Neurology 2005, 493:99-110.
5. Bailey C.H., Giustetto M., Huang Y.-Y., Hawkins R.D., Kandel E.R. Is heterosynaptic modulation essential for stabilizing Hebbian plasticity and memory?. Nature Reviews Neuroscience 2000, 1(1):11-20.
6. Baxter D.A., Canavier C.C., Clark J.W., Byrne J.H. Computational model of the serotonergic modulation of sensory neurons in aplysia. Journal of Neurophysiology 1999, 82:1914-2935.
7. Bear M.F., Connors B.W., Paradiso M.A. Neuroscience: exploring the brain 2005, Williams & Wilkins, Baltimore, MD.; London.
8. Berridge K.C., Robinson T.E. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience?. Brain Research Reviews 1998, 28:309-369.
9. Bi G.-q., Poo M.-m. Synaptic modification by correlated activity: Hebb's postulate revisited. Annual Review of Neuroscience 2001, 24:139-166.
10. Bienenstock L.E., Cooper L.N., Munro P.W. Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. The Journal of Neuroscience 1982, 2(1):32-48.
11. Braitenberg V. Vehicles: experiments in synthetic psychology 1984, The MIT Press.
12. Brembs B., Lorenzetti F.D., Reyes F.D., Baxter D.A., Byrne J.H. Operant reward learning in aplysia: neuronal correlates and mechanisms. Science 2002, 296(5573):1706-1709.
13. Brown T.H., Chapman P.F., Kairiss E.W., Keenan C.L. Long-term synaptic potentiation. Science 1988, 242:724-728.
14. Brown T.H., Kairiss E.W., Keenan C.L. Hebbian synapse: biophysical mechanisms and algorithms. Annual Review of Neuroscience 1990, 13:475-511.
15. Buzaki G. Rhythms of the brain 2006, Oxford University Press.
16. Carew T.J., Walters E.T., Kandel E.R. Classical conditioning in a simple withdrawal reflex in Aplysia californica. The Journal of Neuroscience 1981, 1(12):1426-1437.
17. Clark G.A. International encyclopedia of the social & behavioural sciences: volume 26 2001, Elsevier, Ch. Synaptic efficacy, regulation of, pp. 15371-15378.
18. Clark G.A., Kandel E.R. Branch-specific heterosynaptic facilitation in Aplysia siphon sensory cells. Proceedings of the National Academy of Sciences 1984, 81(8):2577-2581.
19. Cohen M.X. Neurocomputational mechanisms of reinforcement-guided learning in humans: a review. Cognitive, Affective and Behavioral Neuroscience 2008, 8(2):113-125.
20. Cooper S.J. Donald O. Hebb's synapse and learning rule: a history and commentary. Neuroscience and Biobehavioral Reviews 2005, 28(8):851-874.
21. Cox R.B., Krichmar J.L. Neuromodulation as a robot controller: a brain inspired strategy for controlling autonomous robots. IEEE Robotics & Automation Magazine 2009.
22. Dayan P., Abbott L.F. Theoretical neuroscience 2001, MIT Press, Cambridge, MA, USA.
23. Dickinson P.S. Neuromodulation of central pattern generators in invertebrates and vertebrates. Current Opinion in Neurobiology 2006, 16:604-614.
24. Doya K. Metalearning and neuromodulation. Neural Networks 2002, 15(4-6):495-506.
25. Doya K., Uchibe E. The cyber rodent project: exploration and adaptive mechanisms for self-preservation and self-reproduction. Adaptive Behavior 2005, 13(2):149-160.
26. Faisal A.A., Selen L.P.J., Wolpert D.M. Noise in the nervous system. Nature Reviews Neuroscience 2008.
27. Farries M.A., Fairhall A.L. Reinforcement learning with modulated spike timing-dependent synaptic plasticity. Journal of Neurophysiology 2007, 98:3648-3665.
28. Fellous J.-M., Linster C. Computational models of neuromodulation. Neural Computation 1998, 10:771-805.
29. Finch D.M., Tan A.M., Isokawa-Akesson M. Feedforward inhibition of the rat entorhinal cortex and subicular complex. The Journal of Neuroscience 1988, 8(7):2213-2226.
30. Floreano D., Mattiussi C. Bio-inspired artificial intelligence: theories, methods, and technologies. Intelligent robotics and autonomous agents series 2008, The MIT Press.
31. Floreano D., Nolfi S. Evolutionary robotics 2004, The MIT Press.
32. Florian R.V. Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Computation 2007, 19:1468-1502.
33. Frémaux N., Sprekeler H., Gerstner W. Functional requirements for reward-modulated spike-timing-dependent plasticity. The Journal of Neuroscience 2010, 30(40):13326-13337.
34. Fusi S., Annunziato M., Badoni D., Salamon A., Amit D.J. Spike-driven synaptic plasticity: theory, simulation, vlsi implementation. Neural Computation 2000, 12(10):2227-2258.
35. Gerstner W., Kistler M.W. Mathematical formulations of Hebbian learning. Biological Cybernetics 2002, 87:404-415.
36. Gerstner W., Kistler M.W. Spiking neuron models: single neurons, populations, plasticity 2002, Cambridge University Press, Cambridge, UK.
37. Gil M., DeMarco R.J., Menzel R. Learning reward expectations in honeybees. Learning and Memory 2007, 14:291-496.
38. Giocomo L.M., Hasselmo M.E. Neuromodulation by glutamate and acetylcholine can change circuit dynamics by regulating the relative influence of afferent input and excitatory feedback. Molecular Neurobiology 2007, 36:184-200.
39. Grossberg S. Adaptive pattern classification and universal recoding: I. parallel development and coding of neuronal feature detectors. Biological Cybernetics 1976, 23:121-134.
40. Gustafsson B., Wigstroem H., Abraham W.C., Huang Y.-Y. Long-term potentiation in the Hippocampus using depolarizing current pulses as the conditioning stimulus to a single volley synaptic potentials. The Journal of Neuroscience 1987, 7(3):774-780.
41. Hammer M. An identified neuron mediates the unconditioned stimulus in associative olfactory learning in honeybees. Nature 1993, 366:59-63.
42. Harris-Warrick R.M., Marder E. Modulation of neural networks for behavior. Annual Review of Neuroscience 1991, 14:39-57.
43. Hasselmo M.E. Runaway synaptic modification in models of cortex: implications for Alzheimer's disease. Neural Networks 1994, 7(1):13-40.
44. Hasselmo M.E. Neuromodulation and cortical function: modeling the physiological basis of behavior. Behavioural Brain Research 1995, 67:1-27.
45. Hasselmo M.E. The role of acetylcholine in learning and memory. Current Opinion in Neurobiology 2006, 16:710-715.
46. Hasselmo M.E., Bodelon M.E., Wyble B.P. A proposed function for Hippocampal theta rhythm: separate phases of encoding and retrieval enhance reversal of prior learning. Neural Computation 2002, 14(4):793-817.
47. Hasselmo M.E., Schnell E. Laminar selectivity of the cholinergic suppression of synaptic transmission in rat Hippocampal region CA1: computational modeling and brain slice physiology. Journal of Neuroscience 1994, 14(6):3898-3914.
48. Hebb D.O. The organization of behavior: a neuropsychological theory 1949, Wiley, New York.
49. Izhikevich E.M. Solving the distal reward problem through linkage of STDP and dopamine signaling. Cerebral Cortex 2007, 17:2443-2452.
50. Kandel E.R., Tauc L. Heterosynaptic facilitation in neurones of the abdominal ganglion of Aplysia depilans. The Journal of Physiology 1965, 181:1-27.
51. Kelso S.R., Ganong A.H., Brown T.H. Hebbian synapses in hippocampus. Proceedings of the National Academy of Sciences 1986, 83(14):5326-5330.
52. Krichmar J.L. The neuromodulatory system: a framework for survival and adaptive behavior in a challenging world. Adaptive Behavior 2008, 16:385-399.
53. Langton, C. 1990. Computation at the edge of chaos: phase-transitions and emergent computation. Ph.D. Thesis, University of Michigan.
54. Legenstein R., Chase S.M., Schwartz A., Maass W. A reward-modulated hebbian learning rule can explain experimentally observed network reorganization in a brain control task. The Journal of Neuroscience 2010, 30(25):8400-8401.
55. Levy W.B., Steward O. Synapses as associative memory elements in the Hippocampal formation. Brain Research 1979, 175(2):233-245.
56. Levy W.B., Steward O. Temporal contiguity requirements for long-term associative potentiation/depression in the Hippocampus. Neuroscience 1983, 8(4):791-797.
57. Lisman J. A mechanism for the Hebb and the anti-Hebb processes underlying learning and memory. Proceedings of the National Academy of Sciences 1989, 86:9574-9578.
58. Ludvig E.A., Sutton R.S., Kehoe E.J. Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Computation 2008, 20:3034-3054.
59. Maass W., Bishop C.M. Pulsed neural networks 1999, MIT Press, London; Cambridge, Mass.
60. Maass W., Markram H. On the computational power of recurrent circuits of spiking neurons. Journal of Computer and System Sciences 2004, 69(4):593-616.
61. Marder E. Neural modulation: following your own rhythm. Current Biology 1996, 6(2):119-121.
62. Markram H., Lübke J., Frotscher M., Sakmann B. Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science 1997, 275:213-215.
63. Marr D. A theory of cerebellar cortex. Journal Physiology 1969, 202:437-470.
64. McNaughton B.L., Barnes C.A., Rao G., Rasmussen M. Long-term enhancement of Hippocampal synaptic transmission and the acquisition of spatial information. The Journal of Neuroscience 1986, 6(2):563-571.
65. Miller K.D., Mackay D.J.C. The role of constraints in Hebbian learning. Neural Computation 1994, 6:100-126.
66. Moldakarimov S.B., Sejnowski T.J. Concise learning and memory: the editor's selection 2008, Elsevier, Ch. Neural computation theories of learning.
67. Montague P.R., Dayan P., Person C., Sejnowski T.J. Bee foraging in uncertain environments using predictive Hebbian learning. Nature 1995, 377:725-728.
68. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. The Journal of Neuroscience 1996, 16(5):1936-1947.
69. Montague P.R., Hyman S.E., Cohen J.D. Computational roles for dopamine in behavioural control. Nature 2004, 4:2-9.
70. Niv Y., Joel D., Meilijson I., Ruppin E. Evolution of reinforcement learning in uncertain environments: a simple explanation for complex foraging behaviours. Adaptive Behavior 2002, 10(1):5-24.
71. Oja E. Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology 1982, 15(3):267-273.
72. Pennartz C.M.A. The ascending neuromodulatory systems in learning by reinforcement: comparing computational conjectures with experimental findings. Brain Research Reviews 1996, 21:219-245.
73. Pennartz C.M.A. Reinforcement learning by hebbian synapses with adaptive threshold. Neuroscience 1997, 81(2):303-319.
74. Pennartz C.M.A., Kitai S.T. Hippocampal inputs to identified neurons in an in vitro slice preparation of the rat nucleus accumbens: evidence for feed-forward inhibition. The Journal of Neuroscience 1991, 11(9):2838-2847.
75. Pfeiffer M., Nessler B., Douglas R.J., Maass W. Reward-modulated Hebbian learning of decision making. Neural Computation 2010, 22:1-46.
76. Porr B., Wörgötter F. Strongly improved stability and faster convergence of temporal sequence learning by using input correlation only. Neural Computation 2006, 18:1380-1412.
77. Porr B., Wörgötter F. Learning with relevance: using a third factor to stabilize Hebbian learning. Neural Computation 2007, 19(10):2694-2719.
78. Rauschecker J.P., Singer W. The effects of early visual experience on the cat's visual cortex and their possible explanation by Hebb synapses. Journal Physiology 1981, 310:215-239.
79. Redgrave P., Gurney K., Reynolds J. What is reinforced by phasic dopamine signals?. Brain Research Reviews 2008, 58:322-339.
80. Russell S., Norvig P. Artificial intelligence: a modern approach 2003, Prentice Hall. 2nd ed.
81. Schultz W. Behavioural theories and the neurophysiology of reward. Annual Review of Psychology 2006, 57:87-115.
82. Schultz W., Apicella P., Ljungberg T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. The Journal of Neuroscience 1993, 13:900-913.
83. Schultz W., Dayan P., Montague P.R. A neural substrate for prediction and reward. Science 1997, 275:1593-1598.
84. Smith T., Husbands P., Philippides A., O'Shea M. Neuronal plasticity and temporal adaptivity: gasnet robot control networks. Adaptive Behavior 2002, 10:161-183.
85. Soltoggio A., Bullinaria J.A., Mattiussi C., Dürr P., Floreano D. Evolutionary advantages of neuromodulated plasticity in dynamic, reward-based scenarios. Artificial life XI: proceedings of the eleventh international conference on the simulation and synthesis of living systems 2008, MIT Press.
86. Soltoggio, A., Dürr, P., Mattiussi, C., & Floreano, D. 2007. Evolving neuromodulatory topologies for reinforcement learning-like problems. In: Proceedings of the IEEE congress on evolutionary computation, CEC 2007.
87. Soltoggio, A., & Steil, J.J. 2012. Solving the distal reward problem with rare Correlations. Neural Computation (submitted for publication).
88. Soula, H., Alwan, A., & Beslon, G. 2005. Learning at the edge of chaos: temporal coupling of spiking neurons controller for autonomous robotic. In: Proceedings of the AAAI spring symposia on developmental robotics.
89. Sporns O., Alexander W.H. Neuromodulation and plasticity in an autonomous robot. Neural Networks 2002, 15:761-774.
90. Staddon J.E.R. Adaptive behaviour and learning 1983, Cambridge University Press.
91. Stanley K.O., D'Ambrosio D.B., Gauci J.J. A hypercude-based encoding for evolving large-scale neural networks. Artificial Life 2009.
92. Stanley K.O., Miikkulainen R. Evolving neural networks through augmenting topologies. Evolutionary Computation 2002, 10(2):99-127.
93. Stent G.S. A physiological mechanism for Hebb's postulate of learning. Proceedings of the National Academy of Sciences 1973, 70(4):997-1001.
94. Sutton R.S., Barto A.G. Reinforcement learning: an introduction 1998, MIT Press, Cambridge, MA, USA.
95. Thorndike E.L. Animal intelligence 1911, Macmillan.
96. Turrigiano G.G. The self-tuning neuron: synaptic scaling of excitatory synapses. Cell 2008, 135:422-435.
97. van Rossum M.C.W., Bi G.Q., Turrigiano G.G. Stable Hebbian learning from spike timing-dependent plasticity. The Journal of Neuroscience 2000, 20(23):8812-8821.
98. Walters E.T., Byrne J.H. Associative conditioning of single sensory neurons suggests a cellular mechanism for learning. Science 1983, 219:405-408.
99. Werbos, P.J. 1974. Beyond regression: New tools for prediction and analysis in the behavioral sciences. Ph.D. Thesis, Harvard University.
100. Wilson H.R. Spikes, decisions, and actions: the dynamical foundations of neuroscience 1999, Oxford University Press, Oxford.
101. Wise R.A., Rompre P.P. Brain dopamine and reward. Annual Review of Psychology 1989, 40:191-225.