The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments

Journal of Theoretical Biology

Volumn 299, Issue , 2012, Pages 172-179

  Rand, David G ^a,b  

^a HARVARD UNIVERSITY   (United States)

^b HARVARD UNIVERSITY   (United States)

Author keywords

Cooperation; Economic games; Evolutionary game theory; Experimental economics; Internet

Indexed keywords

COMPUTER SIMULATION; DEMOGRAPHY; EMPIRICAL ANALYSIS; EXPERIMENTAL STUDY; INTERNET; LABOR MARKET;

AMAZON MECHANICAL TURK; ARTICLE; BEHAVIOR; COOPERATION; DEMOGRAPHY; ECONOMICS; EVOLUTIONARY GAME THEORY; MATHEMATICAL ANALYSIS; MATHEMATICAL MODEL; ONLINE MONITORING; ONLINE SYSTEM; PRIORITY JOURNAL; PROBABILITY; PUNISHMENT; QUALITATIVE ANALYSIS; QUANTITATIVE ANALYSIS; REWARD; SELF REPORT; THEORY;

BEHAVIORAL MEDICINE; BIOLOGICAL EVOLUTION; COOPERATIVE BEHAVIOR; GAME THEORY; HUMANS; INTERNET; MODELS, GENETIC; RESEARCH DESIGN; SELECTION, GENETIC;

EID: 84857918066     PISSN: 00225193     EISSN: 10958541     Source Type: Journal    
DOI: 10.1016/j.jtbi.2011.03.004     Document Type: Article

References (77)

1. Abbink K., Sadrieh A. The pleasure of being nasty. Economics Letters 2009, 105:306-308.
2. Abbink K., Herrmann B. The moral costs of nastiness. Economic Inquiry 2011, 49(2):631-633.
3. Almenberg J., Dreber A., Apicella C.L., Rand D.G. Third Party Reward and Punishment: Group Size, Efficiency and Public Goods, Psychology and Punishment 2011, Nova Science Publishers, Hauppauge, NY.
4. Andreoni J., Harbaugh W.T., Vesterlund L. The carrot or the stick: rewards, punishments and cooperation. American Economic Review 2003, 93:893-902.
5. Apicella C.L., Dreber A., Campbell B.C., Gray P.B., Hoffman M., Little A.C. Testosterone and Financial Risk Preferences. Evolution and Human Behavior 2008, 29(6):384-390.
6. Axelrod R. The Evolution of Cooperation 1984, Basic Books, New York.
7. Axelrod R., Hamilton W.D. The evolution of cooperation. Science 1981, 211:1390-1396.
8. Blanco M., Engelmann D., Koch A., Normann H.-T. Belief elicitation in experiments: is there a hedging problem?. Experimental Economics 2010, 13:412-438.
9. Brosnan S.F., Salwiczek L., Bshary R. The interplay of cognition and cooperation. Philosophical Transactions of the Royal Society B: Biological Sciences 2010, 365:2699-2710.
10. Buhrmester, M.D., Kwang, T., and Gosling, S.D., 2011. Amazon's Mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 3 (6), 13-5.
11. Burnham T., Hare B. Engineering human cooperation. Human Nature 2007, 18:88-108.
12. Burnham T.C. Engineering altruism: a theoretical and experimental investigation of anonymity and gift giving. Journal of Economic Behavior and Organization 2003, 50:133-144.
13. Camerer C.F. Behavioral Game Theory: Experiments in Strategic Interaction 2003, Princeton University Press, Princeton, NJ.
14. Cinyabuguma M., Page T., Putterman L. Can second-order punishment deter perverse punishment?. Experimental Economics 2006, 9:265-279.
15. Croson R.T.A. Thinking like a game theorist: factors affecting the frequency of equilibrium play. Journal of economic behavior & organization 2000, 41:299-314.
16. Dal Bó P. Cooperation under the shadow of the future: experimental evidence from infinitely repeated games. American Economic Review 2005, 95:1591-1604.
17. Dal Bó P., Fréchette G.R. The evolution of cooperation in infinitely repeated games: experimental evidence. American Economic Review 2011, 101:411-429.
18. Denant-Boemont L., Masclet D., Noussair C. Punishment, counterpunishment and sanction enforcement in a social dilemma experiment. Economic Theory 2007, 33:145-167.
19. Dreber, A., Fudenberg, D., Rand, D.G., 2011. Who cooperates in repeated games? Available at SSRN: http://ssrn.com/abstract=1752366.
20. Dreber A., Rand D.G., Fudenberg D., Nowak M.A. Winners don't punish. Nature 2008, 452:348-351.
21. Dreber A., Apicella C.L., Eisenberg D.T.A., Garcia J.R., Zamore R., Lum J.K., Campbell B.C. The 7R polymorphism in the dopamine Receptor D4 gene (DRD4) is associated with financial risk-taking in men. Evolution and Human Behavior 2009, 30(2):85-92.
22. Fagerlin A., Zikmund-Fisher B.J., Ubel P.A., Jankovic A., Derry H.A., Smith D.M. Measuring numeracy without a math test: development of the subjective numeracy scale. Medical Decision Making 2007, 27:672-680.
23. Fehr E., Gächter S. Cooperation and punishment in public goods experiments. American Economic Review 2000, 90:980-994.
24. Fischbacher U., Gächter S., Fehr E. Are people conditionally cooperative? Evidence from a public goods experiment. Economics Letters 2001, 71:397-404.
25. Fudenberg D., Maskin E. The folk theorem in repeated games with discounting or with incomplete information. Econometrica 1986, 54:533-554.
26. Fudenberg D., Maskin E.S. Evolution and cooperation in noisy repeated games. American Economic Review 1990, 80:274-279.
27. Fudenberg, D., Rand, D.G., Dreber, A. Slow to anger and fast to forgive: cooperation in an uncertain world. American Economic Review, in press.
28. Gächter S., Herrmann B. Reciprocity, culture and human cooperation: previous insights and a new cross-cultural experiment. Philosophical Transactions of the Royal Society B: Biological Sciences 2009, 364:791-806.
29. Gächter S., Renner E. The effects of (incentivized) belief elicitation in public goods experiments. Experimental Economics 2010, 13:364-377.
30. Gächter, S., Herrmann, B. The limits of self-governance when cooperators get punished: experimental evidence from urban and rural Russia. European Economic Review, in press.
31. Grujić J., Fosco C., Araujo L., Cuesta J.A., Sánchez A. Social experiments in the mesoscale: humans playing a spatial Prisoner's Dilemma. PLoS ONE 2010, 5:e13749.
32. Gurerk O., Irlenbusch B., Rockenbach B. The competitive advantage of sanctioning institutions. Science 2006, 312:108-111.
33. Haley K.J., Fessler D.M.T. Nobody's watching? Subtle cues affect generosity in an anonymous economic game. Evolution and Human Behavior 2005, 26:245-256.
34. Harbaugh W.T., Krause K. Children's altruism in public good and dictator experiments. Economic Inquiry 2000, 38:95-109.
35. Hauert C., De Monte S., Hofbauer J., Sigmund K. Volunteering as red queen mechanism for cooperation in public goods games. Science 2002, 296:1129-1132.
36. Herrmann B., Thoni C., Gächter S. Antisocial punishment across societies. Science 2008, 319:1362-1367.
37. Hofbauer J., Sigmund K. Evolutionary Games and Population Dynamics 1998, Cambridge University Press, Cambridge.
38. Horton, J.J., Rand, D.G., Zeckhauser, R.J. The online laboratory: conducting experiments in a real labor market. Experimental Economics, in press, doi:. http://10.1007/s10683-011-9273-9.
39. Ioannidis J.P.A. Why most published research findings are false. PLoS Med 2005, 2:e124.
40. Janssen M.A., Bushman C. Evolution of cooperation and altruistic punishment when retaliation is possible. Journal of theoretical biology 2008, 254:541-545.
41. Kandori M. Social norms and community enforcement. The Review of Economic Studies 1992, 59:63-80.
42. Kreps D.M., Wilson R. Reputation and imperfect information. Journal of economic theory 1982, 27:253-279.
43. Mason, W., and Suri, S., 2010. Conducting behavioral research on Amazon's Mechanical Turk, Available at SSRN: http://ssrn.com/abstract=1691163.
44. Melis A.P., Semmann D. How is human cooperation different?. Philosophical Transactions of the Royal Society B: Biological Sciences 2010, 365:2663-2674.
45. Milinski M., Semmann D., Krambeck H.J. Reputation helps solve the 'tragedy of the commons'. Nature 2002, 415:424-426.
46. Milinski M., Semmann D., Bakker T.C.M., Krambeck H.-J.r. Cooperation through indirect reciprocity: image scoring or standing strategy?. Proceedings of the Royal Society of London Series B: Biological Sciences 2001, 268:2495-2501.
47. Milinski, M., Rockenbach, B. On the interaction of the stick and the carrot in social dilemmas. Journal of Theoretical Biology, this issue, http://doi:10.1016/j/jtbi.2011.03.014.
48. Nikiforakis N. Punishment and counter-punishment in public goods games: can we still govern ourselves?. Journal of Public Economics 2008, 92:91-112.
49. Nowak M., Sigmund K. A strategy of win-stay, lose-shift that outperforms tit-for-tat in the Prisoner's Dilemma game. Nature 1993, 364:56-58.
50. Nowak M.A. Five rules for the evolution of cooperation. Science 2006, 314:1560-1563.
51. Nowak M.A. Evolutionary Dynamics: Exploring the Equations of Life 2006, Belknap press of Harvard University Press, Cambridge, MA and London, England.
52. Nowak M.A., Sigmund K. Tit for tat in heterogeneous populations. Nature 1992, 355:250-253.
53. Nowak M.A., Sigmund K. Evolution of indirect reciprocity. Nature 2005, 437:1291-1298.
54. Ohtsuki H., Iwasa Y. The leading eight: social norms that can maintain cooperation by indirect reciprocity. Journal of Theoretical Biology 2006, 239:435-444.
55. Paolacci G., Chandler J., Ipeirotis P.G. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making 2010, 5:411-419.
56. Pfeiffer T., Rand D.G., Dreber A. Decision-making in research tasks with sequential testing. PLoS ONE 2009, 4:e4607.
57. Prelec D.e. A Bayesian truth serum for subjective data. Science 2004, 306:462-466.
58. Rand D.G., Ohtsuki H., Nowak M.A. Direct reciprocity with costly punishment: generous tit-for-tat prevails. Journal of Theoretical Biology 2009, 256:45-57.
59. Rand D.G., Armao Iv, J.J., Nakamaru M., Ohtsuki H. Anti-social punishment can prevent the co-evolution of punishment and cooperation. Journal of Theoretical Biology 2010, 265:624-632.
60. Rand D.G., Dreber A., Ellingsen T., Fudenberg D., Nowak M.A. Positive interactions promote public cooperation. Science 2009, 325:1272-1275.
61. Rockenbach B., Milinski M. The efficient interaction of indirect reciprocity and costly punishment. Nature 2006, 444:718-723.
62. Roth A.E., Prasnikar V., Okuno-Fujiwara M., Zamir S. Bargaining and market behavior in Jerusalem, Ljubljana, Pittsburgh, and Tokyo: an experimental study. The American Economic Review 1991, 81:1068-1095.
63. Sefton M., Schupp R., Walker J.M. The Effect of rewards and sanctions in provision of public goods. Economic Inquiry 2007, 45:671-690.
64. Seinen I., Schram A. Social status and group norms: indirect reciprocity in a repeated helping experiment. European Economic Review 2006, 50:581-602.
65. Semmann D., Krambeck H.-J., Milinski M. Volunteering leads to rock-paper-scissors dynamics in a public goods game. Nature 2003, 425:390-393.
66. Semmann D., Krambeck H.-J., Milinski M. Reputation is valuable within and outside one's own social group. Behavioral Ecology and Sociobiology 2005, 57:611-616.
67. Sigmund K., De Silva H., Traulsen A., Hauert C. Social learning promotes institutions for governing the commons. Nature 2010, 466:861-863.
68. Suri S., Watts D.J. Cooperation and contagion in web-based, networked public goods experiments. PLoS ONE 2011, 6(3):e16836. doi:10.1371/journal.pone.0016836.
69. Sutter M., Haigner S., Kocher M.G. Choosing the stick or the carrot? endogenous institutional choice in social dilemma situations. Review of Economic Studies 2010, 77:1540-1566.
70. Traulsen A., Hauert C., De Silva H., Nowak M.A., Sigmund K. Exploration dynamics in evolutionary games. Proceedings of the National Academy of Sciences 2009, 106:709-712.
71. Traulsen A., Semmann D., Sommerfeld R.D., Krambeck H.-J., Milinski M. Human strategy updating in evolutionary games. Proceedings of National Academy of Sciences of the United States of America 2010, 107:2962-2966.
72. Ule A., Schram A., Riedl A., Cason T.N. Indirect punishment and generosity toward strangers. Science 2009, 326:1701-1704.
73. Walker J.M., Halloran M. Rewards and sanctions and the provision of public goods in one-shot settings. Experimental Economics 2004, 7:235-247.
74. Wedekind C., Milinski M. Human cooperation in the simultaneous and the alternating Prisoner's Dilemma: Pavlov versus generous tit-for-tat. Proceedings of the National Academy of Sciences of the United States of America 1996, 93:2686-2689.
75. Wedekind C., Milinski M. Cooperation through image scoring in humans. Science 2000, 288:850-852.
76. Wu J.-J., Zhang B.-Y., Zhou Z.-X., He Q.-Q., Zheng X.-D., Cressman R., Tao Y. Costly punishment does not always increase cooperation. Proceedings of the National Academy of Sciences 2009, 106:17448-17451.
77. Yamagishi T. The provision of a sanctioning system as a public good. Journal of Personality and Social Psychology 1986, 51:110-116.