Synaptic theory of Replicator-like melioration

Frontiers in Computational Neuroscience

Volumn 4, Issue , 2010, Pages

  Loewenstein, Yonatan ^a  

^a HEBREW UNIVERSITY OF JERUSALEM   (Israel)

Author keywords

Operant conditioning; Reinforcement learning; Synaptic plasticity

Indexed keywords

CELLULAR PHYSIOLOGY; LEARNING BEHAVIOR; LEARNING RATES; NEURAL ACTIVITY; OPERANT CONDITIONING; SYNAPTIC PLASTICITY; SYNAPTIC PLASTICITY RULES;

EXPERIMENTS; NEURONS; REINFORCEMENT LEARNING; BEHAVIORAL RESEARCH;

PLASTICITY; EXPERIMENTS;

ANALYSIS OF COVARIANCE; ARTICLE; CELL ACTIVITY; CELL FUNCTION; DECISION MAKING; INSTRUMENTAL CONDITIONING; LEARNING; MATHEMATICAL COMPUTING; MATHEMATICAL MODEL; NERVE CELL; NERVE CELL NETWORK; NERVE CELL PLASTICITY;

EID: 84866930679     PISSN: 16625188     EISSN: None     Source Type: Journal    
DOI: 10.3389/fncom.2010.00017     Document Type: Article

References (43)

1. Baras, D., and Meir, R. (2007). Reinforcement learning, spike-time-dependent plasticity, and the BCM rule. Neural. Comput. 19, 2245-2279.
2. Barraclough, D. J., Conroy, M. L., and Lee, D. (2004). Prefrontal cortex and decision making in a mixed-strategy game. Nat. Neurosci. 7, 404-410.
3. Bogacz, R., Brown, E., Moehlis, J., Holmes, P., and Cohen, J. D. (2006). The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychol. Rev. 113, 700-765.
4. Borgers, T., and Sarin, R. (1997). Learning through reinforcement and replicator dynamics. J. Econ. Theory 77, 1-14.
5. Cross, J. G. (1973). A stochastic learning model of economic behavior. Q. J Econ. 87, 239-266.
6. Davison, M., and McCarthy, D. (1988). The Matching Law: A Research Review. Hillsdale, NJ: Lawrence Erlbaum Assoc Inc.
7. Dayan, P., and Abbott, L. F. (2001). Theoretical Neuroscience. Cambridge MA: MIT.
8. Erev, I., and Roth, A. E. (1998). Predicting how people play games: reinforcement learning in experimental games with unique, mixed strategy equilibria. Am. Econ. Rev. 88, 848-881.
9. Farries, M. A., and Fairhall, A. L. (2007). Reinforcement learning with modulated spike timing dependent synaptic plasticity. J. Neurophysiol. 98, 3648-3665.
10. Fiete, I. R., and Seung, H. S. (2006). Gradient learning in spiking neural networks by dynamic perturbation of conductances. Phys. Rev. Lett. 97, 048104.
11. Florian, R. V. (2007). Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural. Comput. 19, 1468-1502.
12. Fudenberg, D., and Levine, D. K. (1998). The Theory of Learning in Games. Cambridge MA: MIT.
13. Gallistel, C. R., Mark, T. A., King, A. P., and Latham, P. E. (2001). The rat approximates an ideal detector of changes in rates of reward: implications for the law of effect. J. Exp. Psychol. Anim. Behav. Process. 27, 354-372.
14. Glimcher, P. W. (2005). Indeterminacy in brain and behavior. Annu. Rev. Psychol. 56, 25-56.
15. Herfst, L. J., and Brecht, M. (2008). Whisker movements evoked by stimulation of single motor neurons in the facial nucleus of the rat. J. Neurophysiol. 99, 2821-2832.
16. Herrnstein, R. J. (1961). Relative and absolute strength of response as a function of frequency of reinforcement. J. Exp. Anal. Behav. 4, 267-272.
17. Herrnstein, R. J. (1997). The Matching Law: Papers in Psychology and Economics. Cambridge: Harvard University Press.
18. Herrnstein, R. J., and Prelec, D. (1991). Melioration, a theory of distributed choice. J. Econ. Perspect. 5, 137-156.
19. Heskes, T., and Kappen, B. (1993). "On-line learning processes in artificial neural networks," in Mathematical Approaches to Neural Networks, Vol.51 ed. J. G. Taylor (Amsterdam: Elsevier), 199-233.
20. Hofbauer, J., and Sigmund, K. (1998). Evolutionary Games and Population Dynamics. Cambridge: Cambridge University Press.
21. Izhikevich, E. M. (2007). Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb. Cortex 17, 2443-2452.
22. Kempter, R., Gerstner, W., and van Hemmen, J. L. (1999). Hebbian learning and spiking neurons. Phys. Rev. E 59, 4498-4514.
23. Law, C. T., and Gold, J. I. (2009). Reinforcement learning can account for associative and perceptual learning on a visual-decision task. Nat. Neurosci. 12, 655-663.
24. Legenstein, R., Chase, S. M., Schwartz, A. B., and Maass, W. (2009). "Functional network organization in motor cortex can be explained by reward-modulated Hebbian learning," in Advances in Neural Information Processing Systems, Vol. 22, eds Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, 1105-1113.
25. Legenstein, R., Pecevski, D., and Maass, W. (2008). A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput. Biol. 4, e1000180. doi: 10.1371/journal.pcbi.1000180.
26. Loewenstein, Y. (2008a). Robustness of learning that is based on covariance-driven synaptic plasticity. PLoS Comput. Biol. 4, e1000007. doi: 10.1371/journal.pcbi.1000007.
27. Loewenstein, Y. (2008b). Covariance-based synaptic plasticity: a model for operant conditioning. Neuroscience Meeting Planner, Washington, DC: Society for Neuroscience Abs. SFN meeting.
28. Loewenstein, Y., Prelec, D., and Seung, H. S. (2009). Operant matching as a Nash equilibrium of an intertemporal game. Neural. Comput. 21, 2755-2773.
29. Loewenstein, Y., and Seung, H. S. (2006). Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proc. Natl. Acad. Sci. U.S.A. 103, 15224-15229.
30. Mazzoni, P., Andersen, R. A., and Jordan, M. I. (1991). A more biologically plausible learning rule for neural networks. Proc. Natl. Acad. Sci. U.S.A. 88, 4433-4437.
31. Narendra, K. S., and Thathachar, M. A. L. (1989). Learning Automata: An Introduction. Englewood Cliffs, NJ: Prentice-Hall.
32. Neiman, T., and Loewenstein, Y. (2007). A dynamic model for matching behavior that is based on the covariance of reward and action. Neural Plast. 2007, 79.
33. Neiman, T., and Loewenstein, Y. (2008). Adaptation to matching behavior: theory and experiments. Neuroscience Meeting Planner, Washington, DC: Society for Neuroscience Abs. SFN meeting.
34. Shamir, M. (2009). The temporal winner-take-all readout. PLoS Comput. Biol. 5, e1000286. doi: 10.1371/journal.pcbi.1000286.
35. Shanks, D. R., Tunney, R. J., and McCarthy, J. D. (2002). A re-examination of probability matching and rational choice. J. Behav. Decis. Mak. 15, 233-250.
36. Soltani, A., and Wang, X. J. (2006). A biophysically based neural model of matching law behavior: melioration by stochastic synapses. J. Neurosci. 26, 3731-3744.
37. Sugrue, L. P., Corrado, G. S., and Newsome, W. T. (2004). Matching behavior and the representation of value in the parietal cortex. Science 304, 1782-1787.
38. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning. Cambridge, MA: MIT Press.
39. Thorndike, E. L. (1911). Animal Intelligence: Experimental Studies. New York: Macmillan.
40. Vulkan, N. (2000). An economist's perspective on probability matching. J. Econ. Surv. 14, 101-118.
41. Wang, X. J. (2002). Probabilistic decision making by slow reverberation in cortical circuits. Neuron 36, 955-968.
42. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229-256.
43. Xie, X., and Seung, H. S. (2004). Learning in neural networks by reinforcement of irregular spiking. Phys. Rev. E Stat. Nonlin. Soft Matter. Phys. 69, 041909.