On learning dynamics underlying the evolution of learning rules

Theoretical Population Biology

Volumn 91, Issue , 2014, Pages 20-36

  Dridi, Slimane ^a   Lehmann, Laurent ^a  

^a UNIVERSITY OF LAUSANNE   (Switzerland)

Author keywords

Evolutionary game theory; Fictitious play; Fluctuating environments; Producer scrounger game; Reinforcement learning; Stochastic approximation

Indexed keywords

ANIMAL; COMPUTER SIMULATION; EVOLUTIONARY BIOLOGY; GAME THEORY; LEARNING; NATURAL SELECTION; NUMERICAL MODEL; STOCHASTICITY;

ANIMALIA;

ARTICLE; BAYES THEOREM; EVOLUTION; EVOLUTIONARY GAME THEORY; FICTITIOUS PLAY; FLUCTUATING ENVIRONMENTS; HUMAN; LEARNING; PROBABILITY; PRODUCER-SCROUNGER GAME; REINFORCEMENT LEARNING; STATISTICS; STOCHASTIC APPROXIMATION;

EVOLUTIONARY GAME THEORY; FICTITIOUS PLAY; FLUCTUATING ENVIRONMENTS; PRODUCER-SCROUNGER GAME; REINFORCEMENT LEARNING; STOCHASTIC APPROXIMATION;

BAYES THEOREM; BIOLOGICAL EVOLUTION; HUMANS; LEARNING; PROBABILITY; STOCHASTIC PROCESSES;

EID: 84891158799     PISSN: 00405809     EISSN: 10960325     Source Type: Journal    
DOI: 10.1016/j.tpb.2013.09.003     Document Type: Article

References (100)

1. Achbany Y., Fouss F., Yen L., Pirotte A., Saerens M. Optimal tuning of continual online exploration in reinforcement learning. Lecture Notes in Computer Science 2006, vol. 4131:790-800. Springer, Berlin / Heidelberg. S. Kollias, A. Stafylopatis, W. Duch, E. Oja (Eds.).
2. Anderson S.P., de Palma A., Thisse J.-F. Discrete Choice Theory of Product Differentiation 1992, The MIT Press, Cambridge, MA.
3. André J.-B. The evolution of reciprocity: social types or social incentives?. The American Naturalist 2010, 175:197-210.
4. Arbilly M., Motro U., Feldman M.W., Lotem A. Co-evolution of learning complexity and social foraging strategies. Journal of Theoretical Biology 2010, 267:573-581.
5. Arbilly M., Motro U., Feldman M.W., Lotem A. Evolution of social learning when high expected payoffs are associated with high risk of failure. Journal of The Royal Society Interface 2011, 8:1604-1615.
6. Arbilly M., Motro U., Feldman M.W., Lotem A. Recombination and the evolution of coordinated phenotypic expression in a frequency-dependent game. Theoretical Population Biology 2011, 80:244-255.
7. Axelrod R. Effective choice in the prisoner's dilemma. The Journal of Conflict Resolution 1980, 24:3-25.
8. Axelrod R., Hamilton W.D. The evolution of cooperation. Science 1981, 211:1390-1396.
9. Benaïm M. Dynamics of stochastic approximation algorithms. Séminaire de Probabilités XXXIII, vol. 1709 1999, 1-68. Springer, Berlin. J. Azéma (Ed.).
10. Benaïm, M., El Karoui, N., 2005. Promenade Aléatoire: Chaînes de Markov et Simulations; Martingales et Stratégies. Editions de l'Ecole Polytechnique, Palaiseau, France.
11. Benaïm M., Hirsch M.W. Mixed equilibria and dynamical systems arising from fictitious play in perturbed games. Games and Economic Behavior 1999, 29:36-72.
12. Benaïm M., Hirsch M.W. Stochastic approximation algorithms with constant step size whose average is cooperative. The Annals of Applied Probability 1999, 9:216-241.
13. Benveniste A., Metivier M., Priouret P. Adaptive Algorithms and Stochastic Approximations 1991, Springer-Verlag, New York (NY).
14. Bernstein C., Kacelnik A., Krebs J.R. Individual decisions and the distribution of predators in a patchy environment. Journal of Animal Ecology 1988, 57:1007-1026.
15. Binmore K.G., Samuelson L. Evolutionary stability in repeated games played by finite automata. Journal of Economic Theory 1992, 57:278-305.
16. Börgers T., Sarin R. Learning through reinforcement and replicator dynamics. Journal of Economic Theory 1997, 77:1-14.
17. Borkar V.S. Stochastic Approximation: A Dynamical Systems Viewpoint 2008, Cambridge University Press, Cambridge, MA.
18. Boyd R., Richerson P.J. Culture and the Evolutionary Process 1985, University of Chicago Press.
19. Brown G.W. Iterative solution of games by fictitious play. Activity Analysis of Production and Allocation 1951, 374-376. Wiley, New York.
20. Bush R.R., Mostelller F. A mathematical model for simple learning. Psychological Review 1951, 58:313-323.
21. Camerer C.F. Behavioral Game Theory: Experiments in Strategic Interaction 2003, Princeton University Press, Princeton, NJ.
22. Camerer C., Ho T.H. Experienced-weighted attraction learning in normal form games. Econometrica 1999, 67:827-874.
23. Cavalli-Sforza L.L., Feldman M.W. Cultural versus genetic adaptation. Proceedings of the National Academy of Sciences of the United States of America 1983, 80:4993-4996.
24. Chalmeau R. Do chimpanzees cooperate in a learning task?. Primates 1994, 35:385-392.
25. Chasparis, G., Shamma, J., Arapostathis, A., 2010. Aspiration learning in coordination games. In 49th IEEE Conference on Decision and Control (CDC), pages 5756-5761.
26. Cho I.-K., Matsui A. Learning aspiration in repeated games. Journal of Economic Theory 2005, 124:171-201.
27. Cournot A.A. Recherches sur les Principes Mathématiques de la Théorie des Richesses 1838, Hachette, L., Paris, France.
28. Dijker A. Physical constraints on the evolution of cooperation. Evolutionary Biology 2011, 38:124-143.
29. Dugatkin L.A. Principles of Animal Behavior 2010, WW Norton & Co, New York (NY). second ed.
30. Dunlap A.S., Stephens D.W. Components of change in the evolution of learning and unlearned preference. Proceedings of the Royal Society B: Biological Sciences 2009, 276:3201-3208.
31. Emery N.J., Clayton N.S. The mentality of crows: Convergent evolution of intelligence in corvids and apes. Science 2004, 306:1903-1907.
32. Enquist M.E., Ghirlanda S. Neural Networks and Animal Behavior 2005, Princeton University Press, Princeton, NJ.
33. Erev I., Roth A.E. Predicting how people play games: Reinforcement learning in experimental games with unique, mixed strategy equilibria. American Economic Review 1998, 88:848-881.
34. Fawcett T.W., Hamblin S., Giraldeau L.-A. Exposing the behavioral gambit: the evolution of learning and decision rules. Behavioral Ecology 2013, 24:2-11.
35. Feldman M., Aoki K., Kumm J. Individual versus social learning: evolutionary analysis in a fluctuating environment. Anthropological Science 1996, 104:209-232.
36. Foster D.P., Young H. Learning, hypothesis testing, and nash equilibrium. Games and Economic Behavior 2003, 45:73-96.
37. Fudenberg D., Levine D.K. The Theory of Learning in Games 1998, MIT Press, Cambridge, MA.
38. Fudenberg D., Takahashi S. Heterogeneous beliefs and local information in stochastic fictitious play. Games and Economic Behavior 2011, 71:100-120.
39. Gale J., Binmore K.G., Samuelson L. Learning to be imperfect: the ultimatum game. Games and Economic Behavior 1995, 8:56-90.
40. Geisler W.S., Diehl R.L. Bayesian natural selection and the evolution of perceptual systems. Philosophical Transactions: Biological Sciences 2002, 357:419-448.
41. Giraldeau L.-A., Caraco T. Social Foraging Theory 2000, Princeton University Press, Princeton, NJ.
42. Grimmett G.R., Stirzaker D.R. Probability and Random Processes 2001, Oxford University Press, Oxford. third ed.
43. Groß R., Houston A.I., Collins E.J., McNamara J.M., Dechaume-Moncharmont F.-X., Franks N.R. Simple learning rules to cope with changing environments. Journal of The Royal Society Interface 2008, 5:1193-1202.
44. Hamblin S., Giraldeau L.-A. Finding the evolutionarily stable learning rule for frequency-dependent foraging. Animal Behaviour 2009, 78:1343-1350.
45. Evolution and the Mechanisms of Decision Making 2012, MIT Press, Cambridge, MA. P. Hammerstein, J.R. Stevens (Eds.).
46. Harley C.B. Learning the evolutionarily stable strategy. Journal of Theoretical Biology 1981, 89:611-633.
47. Herrnstein R.J. On the law of effect. Journal of the Experimental Analysis of Behavior 1970, 13:243-266.
48. Hirsch M.W., Smale S., Devaney R.L. Differential Equations, Dynamical Systems, and an Introduction to Chaos 2004, Academic Press, San Diego, CA.
49. Ho T.H., Camerer C.F., Chong J.-K. Self-tuning experience weighted attraction learning in games. Journal of Economic Theory 2007, 133:177-198.
50. Hofbauer J., Sandholm W.H. On the global convergence of stochastic fictitious play. Econometrica 2002, 70:2265-2294.
51. Hofbauer J., Sigmund K. Evolutionary Games and Population Dynamics 1998, Cambridge University Press, Cambridge, MA.
52. Hofbauer J., Sigmund K. Evolutionary game dynamics. Bulletin of the American Mathematical Society 2003, 40:479-519.
53. Hollis K.L., Dumas M.J., Singh P., Fackelman P. Pavlovian conditioning of aggressive behavior in blue gourami fish (Trichogaster trichopterus): winners become winners and losers stay losers. Journal of Comparative Psychology 1995, 109:123-133.
54. Hopkins E. Two competing models of how people learn in games. Econometrica 2002, 70:2141-2166.
55. Houston A.I. Comments on "Learning the evolutionarily stable strategy". Journal of Theoretical Biology 1983, 105:175-178.
56. Houston A.I., Sumida B.H. Learning rules, matching and frequency dependence. Journal of Theoretical Biology 1987, 126:289-308.
57. Izquierdo L.R., Izquierdo S.S., Gotts N.M., Polhill J.G. Transient and asymptotic dynamics of reinforcement learning in games. Games and Economic Behavior 2007, 61:259-276.
58. Jordan J.S. Bayesian learning in normal form games. Games and Economic Behavior 1991, 3:60-81.
59. Josephson J. A numerical analysis of the evolutionary stability of learning rules. Journal of Economic Dynamics and Control 2008, 32:1569-1599.
60. Karlin S., Taylor H.E. A First Course in Stochastic Processes 1975, Academic Press, San Diego, CA.
61. Katsnelson E., Motro U., Feldman M.W., Lotem A. Evolution of learned strategy choice in a frequency-dependent game. Proceedings of the Royal Society B: Biological Sciences 2011.
62. Kushner H.J., Yin G.G. Stochastic Approximation and Recursive Algorithms and Applications 2003, Springer, New York (NY). second ed.
63. Leimar O., Hammerstein P. Evolution of cooperation through indirect reciprocity. Proceedings of the Royal Society B: Biological Sciences 2001, 268:745-753.
64. Leslie D.S., Collins E.J. Individual q-learning in normal form games. SIAM Journal on Control and Optimization 2005, 44:495-514.
65. Ljung L. Analysis of recursive stochastic algorithms. IEEE Transactions on Automatic Control 1977, 22:551-575.
66. Lotem A. Learning to avoid the behavioral gambit. Behavioral Ecology 2013, 24. 13-13.
67. Luce R.D. Individual Choice Behavior 1959, Wiley, New York.
68. Macy M.W., Flache A. Learning dynamics in social dilemmas. Proceedings of the National Academy of Sciences 2002, 99:7229-7236.
69. Maynard Smith J. Evolution and the Theory of Games 1982, Cambridge University Press, Cambridge.
70. Maynard Smith J., Price G.R. The logic of animal conflict. Nature 1973, 246:15-18.
71. McElreath, R., 2010. Baryplot 1.0.
72. McElreath R., Boyd R. Mathematical Models of Social Evolution: A Guide for the Perplexed 2007, University Of Chicago Press, Chicago.
73. McKelvey R.D., Palfrey T.R. Quantal response equilibria for normal form games. Games and Economic Behavior 1995, 10:6-38.
74. McNamara J.M., Houston A.I. Memory and the efficient use of information. Journal of Theoretical Biology 1987, 125:385-395.
75. McNamara J.M., Houston A.I. Integrating function and mechanism. Trends in Ecology & Evolution 2009, 24:670-675.
76. Niv Y. Reinforcement learning in the brain. Journal of Mathematical Psychology 2009, 53:139-154.
77. Norman M.F. Some convergence theorems for stochastic learning models with distance diminishing operators. Journal of Mathematical Psychology 1968, 5:61-101.
78. Plotnik J.M., Lair R., Suphachoksahakun W., De Waal F.B.M. Elephants know when they need a helping trunk in a cooperative task. Proceedings of the National Academy of Sciences 2011, 108:5116-5121.
79. R Development Core Team 2011. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.
80. Rapoport A., Chammah A.M. Prisoner's Dilemma 1965, University of Michigan Press, Ann Arbor.
81. Rescorla R.A., Wagner A.R. A theory of pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. Classical Conditioning II: Current Research and Theory 1972, 64-99. Appleton-Century-Crofts, New York (NY). A.H. Black, W.F. Prokasy (Eds.).
82. Rodriguez-Gironés M.A., Vásquez R.A. Density-dependent patch exploitation and acquisition of environmental information. Theoretical Population Biology 1997, 52:32-42.
83. Rogers A.R. Does biology constrain culture. American Anthropologist 1988, 90:819-831.
84. Sandholm W.H. Population Games and Evolutionary Dynamics 2011, MIT Press, Cambridge, MA.
85. Schlag K.H. Why imitate, and if so, how?. Journal of Economic Theory 1998, 78:130-156.
86. Shettleworth S.J. Cognition, Evolution, and Behavior 2009, Oxford University Press, New York (NY).
87. Stephens D.W. Change, regularity, and value in the evolution of animal learning. Behavioral Ecology 1991, 2:77-89.
88. Stephens D.W., Clements K.C. Game theory and learning. Game Theory and Animal Behavior 1998, 239-260. Oxford University Press, New York.
89. Sutton R.S., Barto A.G. Reinforcement Learning 1998, MIT Press, Cambridge, MA.
90. Thorndike E.L. Animal Intelligence 1911, Hafner, Darien, CT.
91. Tracy N.D., Seaman J.W. Properties of evolutionarily stable learning rules. Journal of Theoretical Biology 1995, 177:193-198.
92. Tuyls K., Verbeeck K., Lenaerts T. A selection-mutation model for Q-learning in multi-agent systems. Proceedings of the Second International Joint Conference on Autonomous Agents and Multiagent Systems 2003, 693-700. ACM, Melbourne, Australia.
93. van der Horst W., van Assen M., Snijders C. Analyzing behavior implied by EWA learning: an emphasis on distinguishing reinforcement from belief learning. Journal of Mathematical Psychology 2010, 54:222-229.
94. Villarreal R., Domjan M. Pavlovian conditioning of social affirmative behavior in the mongolian gerbil (Meriones unguiculatus). Journal of Comparative Psychology 1998, 112:26-35.
95. Wakano J.Y., Aoki K., Feldman M.W. Evolution of social learning: a mathematical analysis. Theoretical Population Biology 2004, 66:249-258.
96. Wakano J.Y., Yamamura N. A simple learning strategy that realizes robust cooperation better than pavlov in iterated prisoners' dilemma. Journal of Ethology 2001, 19:1-8.
97. Walsh P.T., Hansell M., Borello W.D., Healy S.D. Individuality in nest building: do southern masked weaver (Ploceus velatus) males vary in their nest-building behaviour?. Behavioural Processes 2011, 88:1-6.
98. Weibull J.W. Evolutionary Game Theory 1997, MIT Press, Cambridge, MA.
99. Wolfram Research, Inc. 2011. Mathematica, Version 8.0.4. Champaign, Illinois.
100. Young H.P. Strategic Learning and its Limits 2004, Oxford University Press, Oxford.