A spiking network model of decision making employing rewarded STDP

PLoS ONE

Volumn 9, Issue 3, 2014, Pages

  Skorheim, Steven ^a   Lonjers, Peter ^a   Bazhenov, Maxim ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; DECISION MAKING; FORAGING BEHAVIOR; LEARNING; MATHEMATICAL ANALYSIS; MATHEMATICAL MODEL; NERVE CELL; NERVE CELL NETWORK; NERVE CELL PLASTICITY; REWARD; SPIKE TIMING DEPENDENT PLASTICITY; SPIKE WAVE; SYNAPSE; BIOLOGICAL MODEL; HUMAN; PHYSIOLOGY;

DECISION MAKING; HUMANS; LEARNING; MODELS, NEUROLOGICAL; NEURONAL PLASTICITY; NEURONS; SYNAPSES;

EID: 84898432369     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0090821     Document Type: Article

References (85)

1. Izhikevich EM (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17: 2443-2452.
2. Farries MA, Fairhall AL (2007) Reinforcement learning with modulated spike timing dependent synaptic plasticity. J Neurophysiol 98: 3648-3665.
3. Florian RV (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput 19: 1468-1502.
4. Legenstein R, Pecevski D, Maass W (2008) A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput Biol 4: e1000180.
5. Hull CL (1943) Principles of Behavior. New York: Appelton-century.
6. Frey U, Morris RG (1997) Synaptic tagging and long-term potentiation. Nature 385: 533-536. (Pubitemid 27074671)
7. Morris RG (2006) Elements of a neurobiological theory of hippocampal function: the role of synaptic plasticity, synaptic tagging and schemas. The European journal of neuroscience 23: 2829-2846.
8. Seamans JK, Yang CR (2004) The principal features and mechanisms of dopamine modulation in the prefrontal cortex. Prog Neurobiol 74: 1-58.
9. Nitz DA, Kargo WJ, Fleischer J (2007) Dopamine signaling and the distal reward problem. Neuroreport 18: 1833-1836. (Pubitemid 350303761)
10. Zhang JC, Lau PM, Bi GQ (2009) Gain in sensitivity and loss in temporal contrast of STDP by dopaminergic modulation at hippocampal synapses. Proc Natl Acad Sci U S A 106: 13028-13033.
11. Cassenaer S, Laurent G (2012) Conditional modulation of spike-timing-dependent plasticity for olfactory learning. Nature 482: 47-52.
12. Fremaux N, Sprekeler H, Gerstner W (2010) Functional requirements for reward-modulated spike-timing-dependent plasticity. J Neurosci 30: 13326-13337.
13. Turing AM (1948) Intelligent Machinery. National Physical Laboratory.
14. Cheng B, Titterington D (1994) Neural Networks: A Review from a Statistical Perspective. Statistical Science: 2-54.
15. Ciresan D, Meier U, Gambardella LM, Schmidhuber J (2010) Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition. Neural Computation 22: 3207-3220.
16. Zhang GP, Hu MY, Patuwo BE, Indro DC (1999) Artificial neural networks in bankruptcy prediction: General framework and cross-validation analysis. European Journal of Operational Research 116: 16-32.
17. Basheer IA, Hajmeer M (2000) Artificial neural networks: fundamentals, computing, design, and application. J Microbiol Methods 43: 3-31.
18. Hebb DO (1961) Distinctive features of learning in the higher animal. In: JF d, editor.Brain mechanisms and learning.Lodon: Oxford University press. pp. 37-46.
19. Rao RPN, Sejnowski TJ (2001) Spike-Timing-Dependent Hebbian Plasticity as Temporal Difference Learning. Neural Computation 13: 2221-2237. (Pubitemid 33676882)
20. Bienenstock EL, Cooper LN, Munro PW (1982) Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. J Neurosci 2: 32-48. (Pubitemid 12146056)
21. Lisman J (1989) A mechanism for the Hebb and the anti-Hebb processes underlying learning and memory. Proc Natl Acad Sci U S A 86: 9574-9578.
22. Hansel C, Artola A, Singer W (1997) Relation between dendritic Ca2 + levels and the polarity of synaptic long-term modifications in rat visual cortex neurons. Eur J Neurosci 9: 2309-2322.
23. Ismailov I, Kalikulov D, Inoue T, Friedlander MJ (2004) The kinetic profile of intracellular calcium predicts long-term potentiation and long-term depression. J Neurosci 24: 9847-9861. (Pubitemid 39463565)
24. Malenka RC, Kauer JA, Zucker R, Nicoll RA (1988) Postsynaptic calcium is sufficient for potentiation of the hippocampal synaptic transmission. Science: 81-83.
25. Bliss TV, Collingridge GL (1993) A synaptic model of memory: long-term potentiation in the hippocampus. Nature 361: 31-39. (Pubitemid 23020772)
26. Kawato M, Kuroda S, Schweighofer N (2011) Cerebellar supervised learning revisited: biophysical modeling and degrees-of-freedom control. Curr Opin Neurobiol 21: 791-800.
27. Jain AK, Mao J, Mohiuddin KM (1996) Artificial Neural Networks: A tutorial. Computer 23: 31-44. (Pubitemid 126515068)
28. White H (1989) Learning in Artificial Neural Networks: A Statistical Perspective. Neural Computation 1: 425-464.
29. Huerta R, Nowotny T (2009) Fast and robust learning by reinforcement signals: explorations in the insect brain. Neural Comput 21: 2123-2151.
30. Huerta R, Nowotny T, Garcia-Sanchez M, Abarbanel HD, Rabinovich MI (2004) Learning classification in the olfactory system of insects. Neural Comput 16: 1601-1640. (Pubitemid 38835135)
31. Lotfi A, Benyettou A (2011) Using Probabilistic Neural Networks for Handwritten Digit Recognition. Journal of Artificial Intelligence: 288-294.
32. Potjans W, Morrison A, Diesmann M (2009) A spiking neural network model of an actor-critic learning agent. Neural Comput 21: 301-339.
33. Chadderdon GL, Neymotin SA, Kerr CC, Lytton WW (2012) Reinforcement learning of targeted movement in a spiking neuronal model of motor cortex. PLoS One 7: e47251.
34. Miller P, Katz DB (2010) Stochastic transitions between neural states in taste processing and decision-making. J Neurosci 30: 2559-2570.
35. Lee K, Kwon DS (2008) Synaptic plasticity model of a spiking neural network for reinforcement learning. Neurocomputing 71: 3037-3043.
36. van Rossum MC, Bi GQ, Turrigiano GG (2000) Stable Hebbian learning from spike timing-dependent plasticity. J Neurosci 20: 8812-8821.
37. Wu Z, Yamaguchi Y (2006) Conserving total synaptic weight ensures one-trial sequence learning of place fields in the hippocampus. Neural Netw 19: 547-563.
38. Elliott T, Shadbolt NR (2002) Multiplicative synaptic normalization and a nonlinear Hebb rule underlie a neurotrophic model of competitive synaptic plasticity. Neural Comput 14: 1311-1322.
39. Finelli LA, Haney S, Bazhenov M, Stopfer M, Sejnowski TJ (2008) Synaptic learning rules and sparse coding in a model sensory system. PLoS Comput Biol 4: e1000062.
40. Bazhenov M, Huerta R, Smith BH (2013) A computational framework for understanding decision making through integration of basic learning rules. J Neurosci 33: 5686-5697.
41. Bi GQ, Poo MM (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J Neurosci 18: 10464-10472.
42. Hardingham NR, Hardingham GE, Fox KD, Jack JJ (2007) Presynaptic efficacy directs normalization of synaptic strength in layer 2/3 rat neocortex after paired activity. J Neurophysiol 97: 2965-2975. (Pubitemid 46557726)
43. Morrison A, Aertsen A, Diesmann M (2007) Spike-timing-dependent plasticity in balanced random networks. Neural Comput 19: 1437-1467.
44. Babadi B, Abbott LF (2010) Intrinsic stability of temporally shifted spike-timing dependent plasticity. PLoS Comput Biol 6: e1000961.
45. Gilson M, Fukai T (2010) Stability versus neuronal specialization for STDP: long-tail weight distributions solve the dilemma. PLoS One 6: e25339.
46. Delgado JY, Gomez-Gonzalez JF, Desai NS (2010) Pyramidal neuron conductance state gates spike-timing-dependent plasticity. J Neurosci 30: 15713-15725.
47. Abbott LF, Nelson SB (2000) Synaptic plasticity: taming the beast. Nat Neurosci 3 Suppl: 1178-1183.
48. Kempter R, Gerstner W, van Hemmen JL (2001) Intrinsic stabilization of output rates by spike-based Hebbian learning. Neural Comput 13: 2709-2741. (Pubitemid 33597994)
49. Gutig R, Aharonov R, Rotter S, Sompolinsky H (2003) Learning input correlations through nonlinear temporally asymmetric Hebbian plasticity. J Neurosci 23: 3697-3714. (Pubitemid 36966039)
50. Nishiyama M, Hong K, Mikoshiba K, Poo MM, Kato K (2000) Calcium stores regulate the polarity and input specificity of synaptic modification. Nature 408: 584-588.
51. Zhou YD, Acker CD, Netoff TI, Sen K, White JA (2005) Increasing Ca2+ transients by broadening postsynaptic action potentials enhances timing-dependent synaptic depression. Proc Natl Acad Sci U S A 102: 19121-19125. (Pubitemid 43049576)
52. Haas JS, Nowotny T, Abarbanel HD (2006) Spike-timing-dependent plasticity of inhibitory synapses in the entorhinal cortex. J Neurophysiol 96: 3305-3313. (Pubitemid 44837874)
53. Sjostrom PJ, Turrigiano GG, Nelson SB (2001) Rate, timing, and cooperativity jointly determine cortical synaptic plasticity. Neuron 32: 1149-1164. (Pubitemid 34075403)
54. Feldman DE (2009) Synaptic mechanisms for plasticity in neocortex. Annu Rev Neurosci 32: 33-55.
55. Royer S, Pare D (2003) Conservation of total synaptic weight through balanced synaptic depression and potentiation. Nature 422: 518-522. (Pubitemid 36433646)
56. Chistiakova M, Volgushev M (2009) Heterosynaptic plasticity in the neocortex. Exp Brain Res 199: 377-390.
57. Turrigiano GG, Leslie KR, Desai NS, Rutherford LC, Nelson SB (1998) Activity-dependent scaling of quantal amplitude in neocortical neurons. Nature 391: 892-896. (Pubitemid 28157666)
58. Jay TM (2003) Dopamine: a potential substrate for synaptic plasticity and memory mechanisms. Prog Neurobiol 69: 375-390.
59. Pawlak V, Kerr JN (2008) Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J Neurosci 28: 2435-2446. (Pubitemid 351348182)
60. Schultz W (1999) The Reward Signal of Midbrain Dopamine Neurons. News Physiol Sci 14: 249-255.
61. Yuste R, Denk W (1995) Dendritic spines as basic functional units of neuronal integration. Nature 375: 682-684.
62. Schiller J, Schiller Y, Clapham DE (1998) NMDA receptors amplify calcium influx into dendritic spines during associative pre- and postsynaptic activation. Nat Neurosci 1: 114-118.
63. Lynch GS, Dunwiddie T, Gribkoff V (1977) Heterosynaptic depression: a postsynaptic correlate of long-term potentiation. Nature 266: 737-739. (Pubitemid 8073808)
64. Bonhoeffer T, Staiger V, Aertsen A (1989) Synaptic plasticity in rat hippocampal slice cultures: local "Hebbian" conjunction of pre- and postsynaptic stimulation leads to distributed synaptic enhancement. Proc Natl Acad Sci U S A 86: 8113-8117. (Pubitemid 19265330)
65. Kossel A, Bonhoeffer T, Bolz J (1990) Non-Hebbian synapses in rat visual cortex. Neuroreport 1: 115-118.
66. Engert F, Bonhoeffer T (1997) Synapse specificity of long-term potentiation breaks down at short distances. Nature 388: 279-284. (Pubitemid 27317972)
67. Schuman EM, Madison DV (1994) Locally distributed synaptic potentiation in the hippocampus. Science 263: 532-536. (Pubitemid 24082755)
68. Chen JY, Lonjers P, Lee C, Chistiakova M, Volgushev M, et al. (2013) Heterosynaptic Plasticity Prevents Runaway Synaptic Dynamics. J Neurosci 33: 15915-15929.
69. Skinner BF (1948) Superstition in the pigeon. J Exp Psychol 38: 168-172.
70. Segal DS, Mandell AJ (1974) Long-term administration of d-amphetamine: progressive augmentation of motor activity and stereotypy. Pharmacol Biochem Behav 2: 249-255.
71. Baker DA, Specio SE, Tran-Nguyen LT, Neisewander JL (1998) Amphetamine infused into the ventrolateral striatum produces oral stereotypies and conditioned place preference. Pharmacol Biochem Behav 61: 107-111. (Pubitemid 28477489)
72. Ermentrout GB, Galan RF, Urban NN (2008) Reliability, synchrony and noise. Trends Neurosci 31: 428-434.
73. Anderson JS, Lampl I, Gillespie DC, Ferster D (2000) The contribution of noise to contrast invariance of orientation tuning in cat visual cortex. Science 290: 1968-1972.
74. Mainen ZF, Sejnowski TJ (1995) Reliability of spike timing in neocortical neurons. Science 268: 1503-1506.
75. Rulkov NF, Timofeev I, Bazhenov M (2004) Oscillations in large-scale cortical networks: map-based model. J Comput Neurosci 17: 203-223. (Pubitemid 39157973)
76. Rulkov NF, Bazhenov M (2008) Oscillations and synchrony in large-scale cortical network models. J Biol Phys 34: 279-299.
77. Bazhenov M, Stopfer M (2010) Forward and back: motifs of inhibition in olfactory processing. Neuron 67: 357-358.
78. Timofeev I, Bazhenov M (2005) Mechanisms and biological role of thalamocortical oscillations. Trends in Chronobiology Research: 1-47.
79. Pouille F, Scanziani M (2001) Enforcement of temporal fidelity in pyramidal cells by somatic feed-forward inhibition. Science 293: 1159-1163. (Pubitemid 32758100)
80. Assisi C, Stopfer M, Laurent G, Bazhenov M (2007) Adaptive regulation of sparseness by feedforward inhibition. Nat Neurosci 10: 1176-1184. (Pubitemid 47339947)
81. Stokes CC, Isaacson JS (2011) From dendrite to soma: dynamic routing of inhibition by complementary interneuron microcircuits in olfactory cortex. Neuron 67: 452-465.
82. Mittmann W, Koch U, Hausser M (2005) Feed-forward inhibition shapes the spike output of cerebellar Purkinje cells. J Physiol 563: 369-378. (Pubitemid 40360898)
83. Markram H, Lubke J, Frotscher M, Sakmann B (1997) Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science 275: 213-215. (Pubitemid 27034762)
84. Rulkov NF (2002) Modeling of spiking-bursting neural behavior using two-dimensional map. Phys Rev E Stat Nonlin Soft Matter Phys 65: 041922.
85. Bazhenov M, Rulkov NF, Fellous JM, Timofeev I (2005) Role of network dynamics in shaping spike timing reliability. Phys Rev E Stat Nonlin Soft Matter Phys 72: 041903.