Covariance-based synaptic plasticity in an attractor network model accounts for fast adaptation in free operant learning

Journal of Neuroscience

Volumn 33, Issue 4, 2013, Pages 1521-1534

  Neiman, Tal ^a   Loewenstein, Yonatan ^a  

^a HEBREW UNIVERSITY OF JERUSALEM   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE BEHAVIOR; ANIMAL EXPERIMENT; ARTICLE; AUDITORY STIMULATION; COVARIANCE; DECISION MAKING; INSTRUMENTAL CONDITIONING; LEARNING; NERVE CELL NETWORK; NERVE CELL PLASTICITY; NERVE STIMULATION; NONHUMAN; PRIORITY JOURNAL; RAT; REWARD; SIMULATION;

ADAPTATION, PHYSIOLOGICAL; ANIMALS; BEHAVIOR, ANIMAL; CONDITIONING, OPERANT; MALE; MODELS, NEUROLOGICAL; NEURAL NETWORKS (COMPUTER); NEURONAL PLASTICITY; NEURONS; RATS; RATS, SPRAGUE-DAWLEY;

EID: 84872735075     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.2068-12.2013     Document Type: Article

References (58)

1. Amari S., Arbib MA. (1977). Competition and cooperation in neural nets. Systems Neurosci 2:72-120.
2. Baron J. (2000). Thinking and deciding. Cambridge, United Kingdom: Cambridge University.
3. Corrado GS., Sugrue LP, Seung HS, NewsomeWT. (2005). Linear-nonlinear-Poisson models of primate choice dynamics. J Exp Anal Behav 84:581-617. CrossRef Medline
4. Davison M., McCarthy D. (1988). The matching law: a research review. Hillsdale, NJ: Lawrence Erlbaum.
5. Daw ND., Doya K. (2006). The computational neurobiology of learning and reward. Curr Opin Neurobiol 16:199-204. CrossRef Medline
6. Daw ND., O'Doherty JP, Dayan P, Seymour B, Dolan RJ. (2006). Cortical substrates for exploratory decisions in humans. Nature 441:876-879. CrossRef Medline
7. Dayan P., Abbott LF, Abbott L. (2001). Theoretical neuroscience: computational and mathematical modeling of neural systems. Cambridge, MA: MIT Press.
8. Dorris MC., GlimcherPW. (2004). Activity in posterior parietal cortex is correlated with the relative subjective desirability of action. Neuron 44:365-378. CrossRef Medline
9. Doya K. (2008). Modulators of decision making. Nat Neurosci 11:410-416. CrossRef Medline
10. Elsmore TF., McBride SA. (1994). An eight-alternative concurrent schedule: foraging in a radial maze. J Exp Anal Behav 61:331-348. CrossRef Medline
11. Farries MA., Fairhall AL. (2007). Reinforcement learning with modul spike timing-dependent synaptic plasticity. J Neurophysiol 98:3648-3665. CrossRef Medline
12. Frémaux N, Sprekeler H, Gerstner W. (2010). Functional requirements for reward-modulated spike-timing-dependent plasticity. J Neurosci 30: 13326-13337. CrossRef Medline
13. Fusi S., Asaad WF, Miller EK, Wang XJ. (2007). A neural circuit model of flexible sensorimotor mapping: learning and forgetting on multiple time scales. Neuron 54:319-333. CrossRef Medline
14. Gallistel CR., Mark TA, King AP, Latham PE. (2001). The rat approximates an ideal detector of changes in rates of reward: implications for the law of effect. J Exp Psychol 27:354-372. CrossRef Medline
15. Gallistel CR., King AP, Gottlieb D, Balci F, Papachristos EB, Szalecki M, Carbone KS. (2007). Is matching innate? J Exp Anal Behav 87:161-199. CrossRef Medline
16. Gibbon J. (1995). Dynamics of time matching: arousal makes better seem worse. Psychonom Bull Rev 2:208-215. CrossRef
17. Herrnstein RJ. (1961). Relative and absolute strength of response as a function of frequency of reinforcement. J Exp Anal Behav 4:267-272. CrossRef Medline
18. Herrnstein RJ., Rachlin H, Laibson DI. (2000). The matching law: papers in psychology and economics. Cambridge, MA: Harvard University.
19. Heskes TM., Kappen B. (1993). On-line learning processes in artificial neural networks: math foundations of neural networks, pp 199-233. Amsterdam: Elsevier.
20. HeymanGM. (1982). Is time allocation unconditioned behavior. Quant Anal Behav 2:459-490.
21. Hunter IW., Davison MC. (1978). Response rate and changeover performance on concurrent variable-interval schedules. J Exp Anal Behav 29: 535-556. CrossRef Medline
22. Izhikevich EM. (2007). Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17:2443-2452. CrossRef Medline
23. Jay TM. (2003). Dopamine: a potential substrate for synaptic plasticity and memory mechanisms. Prog Neurobiol 69:375-390. CrossRef Medline
24. Kable JW., Glimcher PW. (2007). The neural correlates of subjective value during intertemporal choice. Nat Neurosci 10:1625-1633. CrossRef Medline
25. Kempter R., Gerstner W, Van Hemmen JL. (1999). Hebbian learning and spiking neurons. Phys Rev 59:4498.
26. Lau B., Glimcher PW. (2008). Value representations in the primate striatum during matching behavior. Neuron 58:451-463. CrossRef Medline
27. Law CT., Gold JI. (2009). Reinforcement learning can account for associative and perceptual learning on a visual-decision task. Nat Neurosci 12:655-663. CrossRef Medline
28. Legenstein R., Chase SM, Schwartz AB, Maass W. (2010). A rewardmodulated hebbian learning rule can explain experimentally observed network reorganization in a brain control task. J Neurosci 30:8400-8410. CrossRef Medline
29. Levelt WJM. (1968). On binocular rivalry. Paris: Mouton.
30. Loewenstein Y. (2008). Robustness of learning that is based on covariancedriven synaptic plasticity. PLoS Comp Biol 4:e1000007. CrossRef Medline
31. Loewenstein Y. (2010). Synaptic theory of replicator-like melioration. Front Comput Neurosci 4.
32. Loewenstein Y., Seung HS. (2006). Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proc Natl Acad Sci U S A 103:15224-15229. CrossRef Medline
33. Mark TA., Gallistel CR. (1994). Kinetics of matching. J Exp Psychol Anim Behav Process 20:79-95. CrossRef Medline
34. Miller HL., Loveland DH. (1974). Matching when the number of response alternatives is large. Learn Behav 2:106-110. CrossRef
35. Moreno-Bote R, Rinzel J, Rubin N. (2007). Noise-induced alternations in an attractor network model of perceptual bistability. J Neurophysiol 98: 1125-1139. CrossRef Medline
36. Moreno-Bote R, Shpiro A, Rinzel J, Rubin N. (2010). Alternation rate in perceptual bistability is maximal at and symmetric around equidominance. J Vis 10(11):1. CrossRef
37. Moreno-Bote R, Knill DC, Pouget A. (2011). Bayesian sampling in visual perception. Proc Natl Acad Sci U S A 108:12491-12496. CrossRef Medline
38. Morris G., Arkadir D, Nevet A, Vaadia E, BergmanH. (2004). Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 43:133-143. CrossRef Medline
39. Myerson J., Miezin FM. (1980). The kinetics of choice: an operant systems analysis. Psychol Rev 87:160. CrossRef
40. Niv Y., Edlund JA, Dayan P, O'Doherty JP. (2012). Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain. J Neurosci 32:551-562. CrossRef Medline
41. Padoa-Schioppa C, Assad JA. (2006). Neurons in the orbitofrontal cortex encode economic value. Nature 441:223-226. CrossRef Medline
42. Pawlak V., Kerr JN. (2008). Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J Neurosci 28:2435-2446. CrossRef Medline
43. Pessiglione M., Seymour B, Flandin G, Dolan RJ, Frith CD. (2006). Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442:1042-1045. CrossRef Medline
44. Riani M., Simonotto E. (1994). Stochastic resonance in the perceptual interpretation of ambiguous figures: a neural network model. Phys Rev Lett 72:3120-3123. CrossRef Medline
45. Sakai Y., Fukai T. (2008). The actor-critic learning is behind the matching law: matching versus optimal behaviors. Neural Comput 20:227-251. CrossRef Medline
46. Salinas E. (2003). Background synaptic activity as a switch between dynamical states in a network. Neural Comput 15:1439-1475. CrossRef Medline
47. SchultzW. (1997). Dopamine neurons and their role in reward mechanisms. Curr Opin Neurobiol 7:191-197. CrossRef Medline
48. Seung HS. (2003). Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron 40:1063-1073. CrossRef Medline
49. Soltani A., Wang XJ. (2006). A biophysically based neural model of matching law behavior: melioration by stochastic synapses. J Neurosci 26: 3731-3744. CrossRef Medline
50. Sugrue LP., Corrado GS, Newsome WT. (2004). Matching behavior and the representation of value in the parietal cortex. Science 304:1782-1787. CrossRef Medline
51. Sutton RS., Barto AG. (1998). Reinforcement learning: an introduction. Cambridge, United Kingdom: Cambridge University.
52. Tong F., Nakayama K, Vaughan JT, Kanwisher N. (1998). Binocular rivalry and visual awareness in human extrastriate cortex. Neuron 21:753-759. CrossRef Medline
53. van KampenNG. (2007). Stochastic processes in physics and chemistry. Amsterdam: North Holland.
54. Wang XJ. (2002). Probabilistic decision making by slow reverberation in cortical circuits. Neuron 36:955-968. CrossRef Medline
55. Wickens JR. (2009). Synaptic plasticity in the basal ganglia. Behav Brain Res 199:119-128. CrossRef Medline
56. Williams RJ. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8:229-256. CrossRef
57. Xie X., Seung HS. (2004). Learning in neural networks by reinforcement of irregular spiking. Phys Rev 69:041909. CrossRef Medline
58. Zhang JC., Lau PM, Bi GQ. (2009). Gain in sensitivity and loss in temporal contrast of STDP by dopaminergic modulation at hippocampal synapses. Proc Natl Acad Sci U S A 106:13028-13033. CrossRef Medline