Spike-based Decision Learning of Nash Equilibria in Two-Player Games

PLoS Computational Biology

Volumn 8, Issue 9, 2012, Pages

  Friedrich, Johannes ^a   Senn, Walter ^a  

^a UNIVERSITY OF BERN   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

BEHAVIORAL RESEARCH; COMPUTATION THEORY; DECISION MAKING; GAME THEORY; MULTI AGENT SYSTEMS; NEURONS; POPULATION STATISTICS; STOCHASTIC SYSTEMS;

ADAPTIVE DECISION MAKING; CO-ADAPTATION; COMPUTATIONAL ALGORITHM; DECISION TASK; MULTI-AGENT ENVIRONMENT; NASH EQUILIBRIA; POPULATION CODING; REINFORCEMENT LEARNINGS; STOCHASTICS; TWO-PLAYER GAMES;

REINFORCEMENT LEARNING;

ARTICLE; COVARIANCE LEARNING; DECISION MAKING; GAME; HUMAN; LEARNING; NERVE CELL; PLAY; REINFORCEMENT; REINFORCEMENT LEARNING; REWARD; SPIKE WAVE; STOCHASTIC MODEL; TASK PERFORMANCE; TEMPORAL DIFFERENCE LEARNING;

ANIMALIA;

EID: 84866941777     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1002691     Document Type: Article

References (51)

1. Dayan P, Daw ND, (2008) Decision theory, reinforcement learning, and the brain. Cogn Affect Behav Ne 8: 429-453.
2. Sutton RS, Barto AG (1998) Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
3. Seymour B, O'Doherty JP, Dayan P, Koltzenburg M, Jones AK, et al. (2004) Temporal difference models describe higher-order learning in humans. Nature 429: 664-7.
4. Potjans W, Morrison A, Diesmann M, (2009) A spiking neural network model of an actor-aritic learning agent. Neural Comput 21: 301-339.
5. Howard RA (1960) Dynamic programming and Markov processes. Cambridge, MA: MIT Press.
6. Smallwood RD, Sondik EJ, (1973) The optimal control of partially observable markov processes over a finite horizon. Oper Res 21: 1071-1088.
7. Friedrich J, Urbanczik R, Senn W, (2011) Spatio-temporal credit assignment in neuronal population learning. PLoS Comput Biol 7: e1002092.
8. Fudenberg D, Levine DK (1998) Theory of Learning in Games. Cambridge, MA: MIT Press.
9. Von Neumann J, Morgenstern O (1944) Theory of games and economic behavior. Princeton: Princeton University Press.
10. Hewig J, Trippe R, Hecht H, Coles GH, Holroyd CB, et al. (2007) Decision-making in blackjack: An electrophysiological analysis. Cereb Cortex 17: 865-877.
11. Dorris MC, Glimcher PW, (2004) Activity in posterior parietal cortex is correlated with the relative subjective desirability of action. Neuron 44: 365-378.
12. Thorndike EL, (1898) Animal Intelligence: An Experimental Study of the Associative Processes in Animals. Psychol Monogr 2: 321-330.
13. Rescorla R, Wagner A (1972) A theory of Pavlovian conditioning: variations in the effectiveness of reinforecement and nonreinforcement. In: Black AH, Prokasy WF, editors. Classical Conditioning II: current research and theory. New York: Appleton Century Crofts. pp. 64-99.
14. Dayan P, Long T (1998) Statistical models of conditioning. Adv Neural Inf Process Syst 10. pp. 117-123.
15. Fiete IR, Seung HS, (2006) Gradient learning in spiking neural networks by dynamic perturbation of conductances. Phys Rev Lett 97: 048104.
16. Frémaux N, Sprekeler H, Gerstner W, (2010) Functional requirements for reward-modulated spike- timing-dependent plasticity. J Neurosci 30: 13326-13337.
17. Erev I, Roth AE, (1998) Predicting how people play games: reinforcement learning in experimental games with unique, mixed strategy equilibria. Amer Econ Rev 88: 848-881.
18. Anderson JR, Bothell D, Byrne MD, Douglass S, Lebiere C, et al. (2004) An integrated theory of the mind. Psychol Rev 111: 1036-1060.
19. Gerstner W, Kistler WM (2002) Spiking Neuron Models. Cambridge, UK: Cambridge University Press.
20. Schultz W, Dayan P, Montague PR, (1997) A neural substrate of prediction and reward. Science 275: 1593-1599.
21. Wang XJ, (2008) Decision making in recurrent neuronal circuits. Neuron 60: 215-234.
22. Urbanczik R, Senn W, (2009) Reinforcement learning in populations of spiking neurons. Nat Neurosci 12: 250-252.
23. Izhikevich EM, (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17: 2443-2452.
24. Averbeck B, Latham PE, Pouget A, (2006) Neural correlations, population coding and computation. Nat Rev Neurosci 7: 358-3666.
25. Von Neumann J, (1928) Zur Theorie der Gesellschaftsspiele. Math Ann 100: 295-320.
26. Hewig J, Trippe R, Hecht H, Coles GH, Holroyd CB, et al. (2008) An electrophysiological analysis of coaching in blackjack. Cortex 44: 1197-1205.
27. Avenhaus R, Von Stengel B, Zamir S (2002) Inspection games. In: Aumann RJ, Hart S, editors. Handbook of Game Theory with Economic Applications. pp. 1947-1987.
28. Glimcher PW, Camerer C, Fehr E, Poldrack R, editors (2008) Neuroeconomics: decision making and the brain. Amsterdam: Elsevier.
29. Pfister J, Toyoizumi T, Barber D, Gerstner W, (2006) Optimal spike-timing-dependent plasticity for precise action potential firing in supervised learning. Neural Comput 18: 1318-1348.
30. Florian RV, (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput 19: 1468-1502.
31. Friedrich J, Urbanczik R, Senn W, (2010) Learning spike-based population codes by reward and population feedback. Neural Comput 22: 1698-1717.
32. Seung HS, (2003) Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron 40: 1063-1073.
33. Werfel J, Xie X, Seung HS, (2005) Learning curves for stochastic gradient descent in linear feedfor-ward networks. Neural Comput 17: 2699-2718.
34. Crawford VP, (1985) Learning behavior and mixed-strategy Nash equilibria. J Econ Behav Organ 6: 69-78.
35. Stahl DO, (1988) On the instability of mixed-strategy Nash equilibria. J Econ Behav Organ 9: 59-69.
36. Shanks DR, Tunney RJ, McCarthy JD, (2002) A re-examination of probability matching and ra-tional choice. J Behav Decis Making 15: 233-250.
37. Loewenstein Y, Seung HS, (2006) Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proc Natl Acad Sci U S A 103: 15224-15229.
38. Loewenstein Y, Prelec D, Seung HS, (2009) Operant matching as a Nash equilibrium of an in-tertemporal game. Neural Comput 21: 2755-2773.
39. Sakai Y, Fukai T, (2008) The actor-critic learning is behind the matching law: matching versus optimal behaviors. Neural Comput 20: 227-251.
40. Herrnstein RJ, (1961) Relative and absolute strength of response as a function of frequency of reinforcement. J Exp Anal Behav 4: 267-272.
41. Loewenstein Y, (2010) Synaptic theory of replicator-like melioration. Front Comput Neurosci 4: 17.
42. Soltani A, Wang XJ, (2006) A biophysically based neural model of matching law behavior: melio-ration by stochastic synapses. J Neurosci 26: 3731-3744.
43. Luce R (1959) Individual Choice Behavior: A Theoretical Analysis. New York: Wiley.
44. Shanks DR, Tunney RJ, McCarthy JD, (1998) Do people play Nash equilibrium? Lessons from evolutionary game theory. J Econ Lit 36: 1-28.
45. Littman ML (1994) Markov games as a framework for multi-agent reinforcement learning. In: Proceedings of the International Conference on Machine Learning. San Francisco, CA. pp. 157-163.
46. Hu J, Wellman MP, (2003) Nash Q-learning for general-sum stochastic games. J Mach Learn Res 4: 1039-1069.
47. Holt CA, Roth AE, (2004) The Nash equilibrium: a perspective. Proc Natl Acad Sci U S A 101: 3999-4002.
48. Yoshida W, Dolan RJ, Friston KJ, (2008) Game theory of mind. PLoS Comput Biol 4: e1000254.
49. Schopenhauer A (1890) The wisdom of life. London: S. Sonnenschein. 147 pp.
50. Seung HS, (1996) How the brain keeps the eyes still. Proc Natl Acad Sci U S A 93: 13339-13344.
51. Williams RJ, (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8: 229-256.