A model for the evolution of reinforcement learning in fluctuating games

Animal Behaviour

Volumn 104, Issue , 2015, Pages 87-114

  Dridi, Slimane ^a   Lehmann, Laurent ^a  

^a UNIVERSITY OF LAUSANNE   (Switzerland)

Author keywords

Evolution of cognition; Evolutionarily stable learning rules; Exploration exploitation trade off; Repeated games; Social interactions; Trial and error learning

Indexed keywords

BEHAVIORAL RESPONSE; COGNITION; ENVIRONMENTAL CHANGE; EVOLUTION; EVOLUTIONARY BIOLOGY; FITNESS; LEARNING; NATURAL SELECTION; POLYMORPHISM;

EID: 84927523336     PISSN: 00033472     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.anbehav.2015.01.037     Document Type: Article

References (98)

1. Achbany Y., Fouss F., Yen L., Pirotte A., Saerens M. Optimal tuning of continual online exploration in reinforcement learning. Artificial neural networks - ICANN 2006, volume 4131 of lecture notes in computer science 2006, 790-800. Springer, Berlin, Germany. S. Kollias, A. Stafylopatis, W. Duch, E. Oja (Eds.).
2. Amano T., Ushiyama K., Moriguchi S., Fujita G., Higuchi H. Decision-making in group foragers with incomplete information: test of individual-based model in geese. Ecological Monographs 2006, 76(4):601-616.
3. Anderson S.P., de Palma A., Thisse J.-F. Discrete choice theory of product differentiation 1992, MIT Press, Cambridge, MA.
4. Arbilly M., Motro U., Feldman M.W., Lotem A. Co-evolution of learning complexity and social foraging strategies. Journal of Theoretical Biology 2010, 267(4):573-581.
5. Arbilly M., Motro U., Feldman M.W., Lotem A. Recombination and the evolution of coordinated phenotypic expression in a frequency-dependent game. Theoretical Population Biology 2011, 80(4):244-255.
6. Arnold S.J. The evolution of a special class of modifiable behaviors in relation to environmental pattern. The American Naturalist 1978, 112(984):415-427.
7. Axelrod R. Effective choice in the prisoner's dilemma. The Journal of Conflict Resolution 1980, 24(1):3-25.
8. Axelrod R., Hamilton W.D. The evolution of cooperation. Science 1981, 211:1390-1396.
9. Begon M., Harper J.L., Townsend C.R. Ecology: Individuals, populations and communities 1996, Blackwell Science, Boston, MA. 6th ed.
10. Benaim M. Dynamics of stochastic approximation algorithms. Séminaire de Probabilités XXXIII 1999, vol. 1709:1-68.
11. Benveniste A., Metivier M., Priouret P. Adaptive algorithms and stochastic approximations 1991, Springer-Verlag, Berlin, Germany.
12. Bernstein C., Kacelnik A., Krebs J.R. Individual decisions and the distribution of predators in a patchy environment. Journal of Animal Ecology 1988, 57(3):1007-1026.
13. Binmore K.G., Samuelson L. Evolutionary stability in repeated games played by finite automata. Journal of Economic Theory 1992, 57:278-305.
14. Black J.M. Partnerships in birds: The study of monogamy 1996, Oxford University Press, New York, NY.
15. Borenstein E., Feldman M.W., Aoki K. Evolution of learning in fluctuating environments: when selection favors both social and exploratory individual learning. Evolution 2008, 62(3):586-602.
16. Borkar V.S. Stochastic approximation: A dynamical systems viewpoint 2008, Cambridge University Press, Cambridge, U.K.
17. Boyd R., Richerson P.J. Culture and the evolutionary process 1988, University of Chicago Press, Chicago, IL.
18. Brown G.W. Iterative solution of games by fictitious play. Activity analysis of production and allocation 1951, 374-376. Wiley, New York, NY. T. Koopmans (Ed.).
19. Bush R.R., Mostelller F. Amathematical model for simple learning. Psychological Review 1951, 58(5):313-323.
20. Camerer C.F. Behavioral game theory: Experiments in strategic interaction 2003, Princeton University Press, Princeton, NJ.
21. Camerer C., Ho T.H. Experienced-weighted attraction learning in normal form games. Econometrica 1999, 67(4):827-874.
22. Charnov E.L. Optimal foraging, the marginal value theorem. Theoretical Population Biology 1976, 9(2):129-136.
23. Chmura T., Goerg S.J., Selten R. Learning in experimental games. Games and Economic Behavior 2012, 76(1):44-73.
24. Dayan P., Abbott L.F. Theoretical neuroscience: Computational and mathematical modeling of neural systems 2005, MIT Press, Cambridge, MA.
25. Dickinson A. Contemporary animal learning theory 1980, Cambridge University Press, Cambridge, U.K.
26. Dijker A. Physical constraints on the evolution of cooperation. Evolutionary Biology 2011, 38(2):124-143.
27. Dridi S., Lehmann L. On learning dynamics underlying the evolution of learning rules. Theoretical Population Biology 2014, 91:20-36.
28. Dubois F., Morand-Ferron J., Giraldeau L.-A. Learning in a game context: strategy choice by some keeps learning from evolving in others. Proceedings of the Royal Society B: Biological Sciences 2010, 277(1700):3609-3616.
29. Duersch P., Oechssler J., Schipper B.C. When is tit-for-tat unbeatable?. International Journal of Game Theory 2014, 43(1):25-36.
30. Dugatkin L.A. Principles of animal behavior 2010, WW Norton, New York, NY. 2nd ed.
31. Dugatkin L.A., Reeve H.K. Game theory and animal behavior 2000, Oxford University Press, New York, NY.
32. Dunlap A.S., Stephens D.W. Components of change in the evolution of learning and unlearned preference. Proceedings of the Royal Society B: Biological Sciences 2009, 276(1670):3201-3208.
33. Emery N.J., Clayton N.S. The mentality of crows: convergent evolution of intelligence in corvids and apes. Science 2004, 306(5703):1903-1907.
34. Emery N.J., Clayton N.S. Comparative social cognition. Annual Review of Psychology 2009, 60(1):87-113.
35. Enquist M.E., Ghirlanda S. Neural networks and animal behavior 2005, Princeton University Press, Princeton, NJ.
36. Erev I., Roth A.E. Predicting how people play games: reinforcement learning in experimental games with unique, mixed strategy equilibria. American Economic Review 1998, 88(4):848-881.
37. Fawcett T.W., Hamblin S., Giraldeau L.-A. Exposing the behavioral gambit: the evolution of learning and decision rules. Behavioral Ecology 2013, 24(1):2-11.
38. Feldman M., Aoki K., Kumm J. Individual versus social learning: evolutionary analysis in a fluctuating environment. Anthropological Science 1996, 104:209-232.
39. Feltovich N. Reinforcement-based versus belief-based learning models in experimental asymmetric-information games. Econometrica 2000, 68(3):605-641.
40. Fudenberg D., Levine D.K. The theory of learning in games 1998, MIT Press, Cambridge, MA.
41. van Gils J.A., Schenk I.W., Bos O., Piersma T. Incompletely informed shorebirds that face a digestive constraint maximize net energy gain when exploiting patches. The American Naturalist 2003, 161(5):777-793.
42. Hamblin S., Giraldeau L.-A. Finding the evolutionarily stable learning rule for frequency-dependent foraging. Animal Behaviour 2009, 78(6):1343-1350.
43. Evolution and the mechanisms of decision making 2012, MIT Press, Cambridge, MA. P. Hammerstein, J.R. Stevens (Eds.).
44. Harley C.B. Learning the evolutionarily stable strategy. Journal of Theoretical Biology 1981, 89(4):611-633.
45. Heller D. An evolutionary approach to learning in a changing environment. Journal of Economic Theory 2004, 114(1):31-55.
46. Hirsch M.W., Smale S., Devaney R.L. Differential equations, dynamical systems, and an introduction to chaos 2004, Academic Press, San Diego, CA.
47. Ho T.H., Camerer C.F., Chong J.-K. Self-tuning experience weighted attraction learning in games. Journal of Economic Theory 2007, 133(1):177-198.
48. Hofbauer J., Sandholm W.H. On the global convergence of stochastic fictitious play. Econometrica 2002, 70(6):2265-2294.
49. Hofbauer J., Sigmund K. Evolutionary games and population dynamics 1998, Cambridge University Press, Cambridge, U.K.
50. Hopkins E. Two competing models of how people learn in games. Econometrica 2002, 70(6):2141-2166.
51. Horváth G., Kovářík J., Mengel F. Limited memory can be beneficial for the evolution of cooperation. Journal of Theoretical Biology 2012, 300:193-205.
52. Izquierdo L.R., Izquierdo S.S., Gotts N.M., Polhill J.G. Transient and asymptotic dynamics of reinforcement learning in games. Games and Economic Behavior 2007, 61(2):259-276.
53. Johnston T.D. Selective costs and benefits in the evolution of learning. Advances in the study of behavior 1982, Vol. 12:65-106. Academic Press, San Diego, CA.
54. Josephson J. Anumerical analysis of the evolutionary stability of learning rules. Journal of Economic Dynamics and Control 2008, 32(5):1569-1599.
55. Karlin S., Taylor H.E. Afirst course in stochastic processes 1975, Academic Press, San Diego, CA.
56. Katsnelson E., Motro U., Feldman M.W., Lotem A. Evolution of learned strategy choice in a frequency-dependent game. Proceedings of the Royal Society B: Biological Sciences 2011, rspb20111734.
57. Kempe M., Mesoudi A. Experimental and theoretical models of human cultural evolution. Wiley Interdisciplinary Reviews: Cognitive Science 2014, 5(3):317-326.
58. Kendal J., Giraldeau L.-A., Laland K. The evolution of social learning rules: payoff-biased and frequency-dependent biased transmission. Journal of Theoretical Biology 2009, 260(2):210-219.
59. Kleiman D.G. Monogamy in mammals. The Quarterly Review of Biology 1977, 52(1):39-69.
60. Krebs J.R., Davies N.B., West S.A. An introduction to behavioural ecology 1993, Wiley-Blackwell, Chichester, U.K. 3rd ed.
61. Laland K.N. Social learning strategies. Learning & Behavior 2004, 32(1):4-14.
62. Leslie D.S., Collins E.J. Individual Q-learning in normal form games. SIAM Journal on Control and Optimization 2005, 44(2):495-514.
63. Lima S.L. Downy woodpecker foraging behavior: efficient sampling in simple stochastic environments. Ecology 1984, 65(1):166-174.
64. Luttbeg B., Warner R.R. Reproductive decision-making by female peacock wrasses: flexible versus fixed behavioral rules in variable environments. Behavioral Ecology 1999, 10(6):666-674.
65. Macy M.W., Flache A. Learning dynamics in social dilemmas. Proceedings of the National Academy of Sciences of the United States of America 2002, 99(90003):7229-7236.
66. Maynard Smith J. Evolution and the theory of games 1982, Cambridge University Press, Cambridge, U.K.
67. McElreath R., Boyd R. Mathematical models of social evolution: A guide for the perplexed 2007, University of Chicago Press, Chicago, IL.
68. McKelvey R.D., Palfrey T.R. Quantal response equilibria for normal form games. Games and Economic Behavior 1995, 10(1):6-38.
69. McNamara J.M., Green R.F., Olsson O. Bayes' theorem and its applications in animal behaviour. Oikos 2006, 112(2):243-251.
70. McNamara J.M., Houston A.I. Optimal foraging and learning. Journal of Theoretical Biology 1985, 117(2):231-249.
71. McNamara J.M., Houston A.I. Memory and the efficient use of information. Journal of Theoretical Biology 1987, 125(4):385-395.
72. McNamara J.M., Houston A.I. Integrating function and mechanism. Trends in Ecology & Evolution 2009, 24(12):670-675.
73. Mery F., Kawecki T.J. Experimental evolution of learning ability in fruit flies. Proceedings of the National Academy of Sciences of the United States of America 2002, 99(22):14274-14279.
74. Mohlin E. Evolution of theories of mind. Games and Economic Behavior 2012, 75(1):299-318.
75. Molleman L., van den Berg P., Weissing F.J. Consistent individual differences in human social learning strategies. Nature Communications 2014, 5.
76. Niv Y. Reinforcement learning in the brain. Journal of Mathematical Psychology 2009, 53(3):139-154.
77. Nowak M.A., Sigmund K. Astrategy of win-stay, lose-shift that outperforms tit-for-tat in the prisoner's dilemma game. Nature 1993, 364:56-58.
78. Pemantle R. Nonconvergence to unstable points in urn models and stochastic approximations. The Annals of Probability 1990, 18(2):698-712.
79. Premack D., Woodruff G. Does the chimpanzee have a theory of mind?. Behavioral and Brain Sciences 1978, 1(04):515-526.
80. Rapoport A., Chammah A.M. Prisoner's dilemma 1965, University of Michigan Press, Ann Arbor, MI.
81. Rendell L., Boyd R., Cownden D., Enquist M., Eriksson K., Feldman M.W., et al. Why copy others? insights from the social learning strategies tournament. Science 2010, 328(5975):208-213.
82. Rescorla R.A., Wagner A.R. Atheory of pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. Classical conditioning II: Current research and theory 1972, 64-99. Appleton-Century-Crofts, New York, NY. A.H. Black, W.F. Prokasy (Eds.).
83. Rogers A.R. Does biology constrain culture?. American Anthropologist 1988, 90(4):819-831.
84. Schlag K.H. Why imitate, and if so, how?. Journal of Economic Theory 1998, 78:130-156.
85. Schloegl C., Dierks A., Gajdon G.K., Huber L., Kotrschal K., Bugnyar T. What you see is what you get? exclusion performances in ravens and keas. PLoS One 2009, 4(8):e6368.
86. Schneeberger K., Dietz M., Taborsky M. Reciprocal cooperation between unrelated rats depends on cost to donor and benefit to recipient. BMC Evolutionary Biology 2012, 12(1):41.
87. Shettleworth S.J. Cognition, evolution, and behavior 2009, Oxford University Press, New York, NY.
88. Shettleworth S.J., Krebs J.R., Stephens D.W., Gibbon J. Tracking a fluctuating environment: a study of sampling. Animal Behaviour 1988, 36(1):87-105.
89. Staddon J.E.R., Cerutti D.T. Operant conditioning. Annual Review of Psychology 2003, 54(1):115-144.
90. Stephens D.W. Change, regularity, and value in the evolution of animal learning. Behavioral Ecology 1991, 2(1):77-89.
91. Stephens D.W., Clements K.C. Game theory and learning. Game theory and animal behavior 1998, 239-260. Oxford University Press, New York, NY. L.A. Dugatkin, H.K. Reeve (Eds.).
92. Sutton R.S., Barto A.G. Reinforcement learning: An introduction 1998, MIT Press, Cambridge, MA.
93. Taylor A.H., Miller R., Gray R.D. New Caledonian crows reason about hidden causal agents. Proceedings of the National Academy of Sciences of the United States of America 2012, 109(40):16389-16391.
94. Thorndike E.L. Animal intelligence 1911, Hafner, Darien, CT.
95. Trimmer P.C., Houston A.I., Marshall J.A.R., Mendl M.T., Paul E.S., McNamara J.M. Decision-making under uncertainty: biases and bayesians. Animal Cognition 2011, 14(4):465-476.
96. Valone T.J. Are animals capable of bayesian updating? An empirical review. Oikos 2006, 112(2):252-259.
97. Wakano J.Y., Aoki K., Feldman M.W. Evolution of social learning: a mathematical analysis. Theoretical Population Biology 2004, 66(3):249-258.
98. Weibull J.W. Evolutionary game theory 1997, MIT Press, Cambridge, MA.