Altruistic learning

Frontiers in Behavioral Neuroscience

Volumn 3, Issue SEP, 2009, Pages

  Seymour, Ben ^a   Yoshida, Wako ^a   Dolan, Ray ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Altruism; Evolution; Free rider problem; Neuroeconomics; Reinforcement learning; Strong reciprocity; Theory of mind

Indexed keywords

EID: 77957253043     PISSN: 16625153     EISSN: None     Source Type: Journal    
DOI: 10.3389/neuro.08.023.2009     Document Type: Article

References (74)

1. Abbeel, P., and Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-first International Conference on Machine Learning, Banff, Alberta, Canada. ACM International Conference Proceeding Series. New York, NY, ACM.
2. Adams, C. D., and Dickinson, A. (1981). Instrumental responding following reinforcer devaluation. Q. J. Exp. Psychol. B Comp. Physiol. Psychol. 33, 109-121.
3. Ariely, D., and Norton, M. I. (2007). Psychology and experimental economics: a gap in abstraction. Curr. Dir. Psychol. Sci. 16, 336-339.
4. Axelrod, R. (1984). The Evolution of Cooperation. New York, Basic Books.
5. Balleine, B. W., and Dickinson, A. (1998). Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37, 407-419.
6. Balleine, B. W., Liljeholm, M., and Ostlund, S. B. (2009). The integrative function of the basal ganglia in instrumental conditioning. Behav. Brain Res. 199, 43-52.
7. Barraclough, D. J., Conroy, M. L., and Lee, D. (2004). Prefrontal cortex and decision making in a mixed-strategy game. Nat. Neurosci. 7, 404-410.
8. Bateson, M., Nettle, D., and Roberts, G. (2006). Cues of being watched enhance cooperation in a real-world setting. Biol. Lett. 2, 412-414.
9. Behrens, T. E., Hunt, L. T., and Rushworth, M. F. (2009). The computation of social behavior. Science 324, 1160-1164.
10. Behrens, T. E. J., Hunt, L. T., Woolrich, M. W., and Rushworth, M. F. S. (2008). Associative learning of social value. Nature 456, 245-245.
11. Bellman, R. (1957). Dynamic Programming. Princeton, NJ, Princeton University Press.
12. Berg, J. E., Dickhaut, J., and McCabe, K. (1995). Trust, reciprocity, and social history. Games Econ. Behav. 10, 122-142.
13. Bernhard, H., Fischbacher, U., and Fehr, E. (2006). Parochial altruism in humans. Nature 442, 912-915.
14. Bowles, S., and Gintis, H. (2004). The evolution of strong reciprocity: cooperation in heterogeneous populations. Theor. Popul. Biol. 65, 17-28.
15. Boyd, R., Gintis, H., Bowles, S., and Richerson, P. J. (2003). The evolution of altruistic punishment. Proc. Natl. Acad. Sci. USA 100, 3531-3535.
16. Boyd, R., and Richerson, P. J. (1988). The evolution of reciprocity in sizable groups. J. Theor. Biol. 132, 337-356.
17. Camerer, C. F. (2003). Behavioural Game Theory: Experiments in Strategic Interaction. Princeton, NJ, Princeton University Press.
18. Camerer, C. F., Ho, T. H., and Chong, J. K. (2004a). A cognitive hierarchy model of games. Q. J. Econ. 119, 861-898.
19. Camerer, C. F., Loewenstein, G., and Prelec, D. (2004b). Neuroeconomics: why economics needs brains. Scand. J. Econ. 106, 555-579.
20. Cardinal, R. N., Parkinson, J. A., Hall, J., and Everitt, B. J. (2002). Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci. Biobehav. Rev. 26, 321-352.
21. Chater, N., Vlaev, I., and Grinberg, M. (2008). A new consequence of Simpson's paradox: stable cooperation in one-shot prisoner's dilemma from populations of individualistic learners. J. Exp. Psychol. Gen. 137, 403-421.
22. Choi, J. K., and Bowles, S. (2007). The coevolution of parochial altruism and war. Science 318, 636-640.
23. Claus, C., and Boutilier, C. (1998). The dynamics of reinforcement learning in cooperative multiagent systems. In Proceedings of the Fifteenth National Conference on Artificial Intelligence. Menlo Park, CA, American Association for Artificial Intelligence, pp. 746-752.
24. Corbit, L. H., and Balleine, B. W. (2000). The role of the hippocampus in instrumental conditioning. J. Neurosci. 20, 4233-4239.
25. Corbit, L. H., Muir, J. L., and Balleine, B. W. (2003). Lesions of mediodorsal thalamus and anterior thalamic nuclei produce dissociable effects on instrumental conditioning in rats. Eur. J. Neurosci. 18, 1286-1294.
26. Daw, N. D., Niv, Y., and Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704-1711.
27. Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., and Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature 441, 876-879.
28. Dayan, P. (2008). The role of value systems in decision making. In Better than Conscious? Decision Making, the Human Mind, and Implications for Institutions, C. Engel and W. Singer, eds (Frankfurt, MIT Press), pp. 51-70.
29. Dayan, P., Niv, Y., Seymour, B., and Daw, D. (2006). The misbehavior of value and the discipline of the will. Neural Netw. 19, 1153-1160.
30. de Quervain, D. J. F., Fischbacher, U., Treyer, V., Schelthammer, M., Schnyder, U., Buck, A., and Fehr, E. (2004). The neural basis of altruistic punishment. Science 305, 1254-1258.
31. Dickinson, A., and Balleine, B. (1994). Motivational control of goal-directed action. Anim. Learn. Behav. 22, 1-18.
32. Dickinson, A., and Balleine, B. W. (2002). The role of learning in the operation of motivational systems. In Stevens' Handbook of Experimental Psychology, 3rd Edn., Vol. 3, Learning Motivation*Emotion, H. Pashler R. Gallistel, eds (New York, John Wiley Sons).
33. Erev, I., and Roth, A. E. (1998). Predicting how people play games: reinforcement learning in experimental games with unique, mixed strategy equilibria. Am. Econ. Rev. 88, 848-881.
34. Erev, I., and Roth, A. E. (2007). Multiagent learning and the descriptive value of simple models. Artif. Intell. 171, 423-428.
35. Fehr, E., and Fischbacher, U. (2003). The nature of human altruism. Nature 425, 785-791.
36. Fehr, E., Kirchsteiger, A., and Riedl, A. (1993). Does fairness prevent market clearing? An experimental investigation. Q. J. Econ. 108, 437-459.
37. Hampton, A. N., Bossaerts, P., and O'Doherty, J. P. (2008). Neural correlates of mentalizing-related computations during strategic interactions in humans. Proc. Natl. Acad. Sci. USA 105, 6741-6746.
38. Harbaugh, W. T. (1998). The prestige motive for making charitable transfers. Am. Econ. Rev. 88, 277-282.
39. Henrich, J., Boyd, R., Bowles, S., Camerer, C., Fehr, E., Gintis, H., and McElreath, R. (2001). In search of Homo economicus: behavioral experiments in 15 small-scale societies. Am. Econ. Rev. 91, 73-78.
40. Henrich, J., McElreath, R., Barr, A., Ensminger, J., Barrett, C., Bolyanatz, A., Cardenas, J. C., Gurven, M., Gwako, E., Henrich, N., Lesorogol, C., Marlowe, F., Tracer, D., and Ziker, J. (2006). Costly punishment across human societies. Science 312, 1767-1770.
41. Heyes, C. M., and Dawson, G. R. (1990). A demonstration of observationallearning in rats using a bidirectional control. Q. J. Exp. Psychol. B Comp. Physiol. Psychol. 42, 59-71.
42. Holman, E. W. (1975). Some conditions for dissociation of consummatory and instrumental behavior in rats. Learn. Motiv. 6, 358-366.
43. Hu, J. L., and Wellman, M. P. (2004). Nash Q-learning for general-sum stochastic games. J. Mach. Learn. Res. 4, 1039-1069.
44. Kim, H., Shimojo, S., and O'Doherty, J. P. (2006). Is avoiding an aversive outcome rewarding? Neural substrates of avoidance learning in the human brain. PLoS Biol. 4, e233. doi: 10.1371/ journal.pbio.0040233.
45. King-Casas, B., Tomlin, D., Anen, C., Camerer, C. F., Quartz, S. R., and Montague, P. R. (2005). Getting to know you: reputation and trust in a two-person economic exchange. Science 308, 78-83.
46. Klucharev, V., Hytonen, K., Rijpkema, M., Smidts, A., and Fernandez, G. (2009). Reinforcement learning signal predicts social conformity. Neuron 61, 140-151.
47. Kumaran, D., and Maguire, E. A. (2006). The dynamics of hippocampal activation during encoding of overlapping sequences. Neuron 49, 617-629.
48. Lee, D. (2006). Neural basis of quasirational decision making. Curr. Opin. Neurobiol. 16, 191-198.
49. Lee, D. (2008). Game theory and neural basis of social decision making. Nat. Neurosci. 11, 404-409.
50. Lengyel, M., and Dayan, P. (2007). Hippocampal contributions to control: The third way. Adv. Neural Inf. Process. Syst. 20, 889-896.
51. Littman, M. L. (1994). Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the Eleventh International Conference on Machine Learning. San Francisco, CA, Morgan Kaufmann, pp. 157-163.
52. Mackintosh, N. J. (1983). Conditioning and associative learning. New York, Oxford University Press.
53. Mobbs, D., Yu, R. J., Meyer, M., Passamonti, L., Seymour, B., Calder, A. J., Schweizer, S., Frith, C. D., and Dalgleish, T. (2009). A key role for similarity in vicarious reward. Science 324, 900.
54. Ng, Y. N., and Russell, S. (2000). Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning. San Francisco, CA, Morgan Kaufmann, pp. 663-670.
55. O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., and Dolan, R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron 38, 329-337.
56. O'Doherty, J. P., Hampton, A., and Kim, H. (2007). Model-based fMRI and its application to reward learning and decision making. Ann. NY Acad. Sci. 1104, 35-53.
57. Price, B., and Boutilier, C. (2003). Accelerating reinforcement learning through implicit imitation. J. Artif. Intell. Res. 19, 569-629.
58. Rescorla, R. A., and Wagner, A. R. (1972). A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and non-reinforcement. In Classical Conditioning, Vol. II, A. H. Black and W. F. Prokasy, eds (New York, Appleton-Century-Crofts).
59. Seo, H., Barraclough, D. J., and Lee, D. (2009). Lateral intraparietal cortex and reinforcement learning during a mixed-strategy game. J. Neurosci. 29, 7278-7289.
60. Seo, H., and Lee, D. (2008). Cortical mechanisms for reinforcement learning in competitive games. Philos. Trans. R. Soc. Lond., B, Biol. Sci. 363, 3845-3857.
61. Seymour, B., and Dolan, R. (2008). Emotion, decision making, and the amygdala. Neuron 58, 662-671.
62. Singer, T., Kiebel, S. J., Winston, J. S., Dolan, R. J., and Frith, C. D. (2004). Brain responses to the acquired moral status of faces. Neuron 41, 653-662.
63. Smith, A. (1976). The Theory of Moral Sentiments, D. D. Raphael and A. L. Macfie, eds (Oxford, Oxford University Press).
64. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA, MIT Press.
65. Tomasello, M., Carpenter, M., Call, J., Behne, T., and Moll, H. (2005). Understanding and sharing intentions: the origins of cultural cognition. Behav. Brain Sci. 28, 675-691.
66. Tricomi, E., Balleine, B. W., and O'Doherty, J. P. (2009). A specific role for posterior dorsolateral striatum in human habit learning. Eur. J. Neurosci. 29, 2225-2232.
67. Trivers, R. (1971). The evolution of reciprocal altruism. Q. Rev. Biol. 46, 35-57.
68. Valentin, V. V., Dickinson, A., and O'Doherty, J. P. (2007). Determining the neural substrates of goal-directed learning in the human brain. J. Neurosci. 27, 4019-4026.
69. Vlaev, I., and Chater, N. (2006). Game relativity: how context influences strategic decision making. J. Exp. Psychol. Learn. Mem. Cogn. 32, 131-149.
70. Wittmann, B. C., Daw, N. D., Seymour, B., and Dolan, R. J. (2008). Striatal activity underlies novelty-based choice in humans. Neuron 58, 967-973.
71. Wolpert, D. M., Doya, K., and Kawato, M. (2003). A unifying computational framework for motor control and social interaction. Philos. Trans. R. Soc. Lond., B, Biol. Sci. 358, 593-602.
72. Xiao, E., and Houser, D. (2005). Emotion expression in human punishment behavior. Proc. Natl. Acad. Sci. USA 102, 7398-7401.
73. Yin, H. H., Ostlund, S. B., Knowlton, B. J., and Balleine, B. W. (2005). The role of the dorsomedial striatum in instrumental conditioning. Eur. J. Neurosci. 22, 513-523.
74. Yoshida, W., Dolan, R. J., and Friston, K. J. (2008). Game theory of mind. PLoS Comput. Biol. 4(12), e1000254. doi: 10.1371/journal.pcbi.1000254.