Conducting interactive experiments online

Experimental Economics

Volumn 21, Issue 1, 2018, Pages 99-131

  Arechar, Antonio A ^a   Gächter, Simon ^b,c,d   Molleman, Lucas ^b,e  

^a YALE UNIVERSITY   (United States)

^b UNIVERSITY OF NOTTINGHAM   (United Kingdom)

^c CESIFO   (Germany)

^d IZA   (Germany)

^e MAX PLANCK INSTITUTE FOR HUMAN DEVELOPMENT   (Germany)

Author keywords

Amazon Mechanical Turk; Behavioral research; Experimental methodology; Internet experiments; Public goods game; Punishment

Indexed keywords

EID: 85019167135     PISSN: 13864157     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10683-017-9527-2     Document Type: Article

References (54)

1. Abeler, J., & Nosenzo, D. (2015). Self-selection into laboratory experiments: pro-social motives versus monetary incentives. Experimental Economics, 18(2), 195–214. doi:10.1007/s10683-014-9397-9.
2. Amir, O., Rand, D. G., & Gal, Y. K. (2012). Economic games on the internet: The effect of $1 stakes. PLoS ONE. doi:10.1371/journal.pone.0031461.
3. Anderhub, V., Muller, R., & Schmidt, C. (2001). Design and evaluation of an economic experiment via the internet. Journal of Economic Behavior & Organization, 46(2), 227–247. doi:10.1016/S0167-2681(01)00195-0.
4. Anderson, J., Burks, S. V., Carpenter, J., Gotte, L., Maurer, K., Nosenzo, D., et al. (2013). Self-selection and variations in the laboratory measurement of other-regarding preferences across subject pools: evidence from one college student and two adult samples. Experimental Economics, 16(2), 170–189. doi:10.1007/s10683-012-9327-7.
5. Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43(3), 800–813. doi:10.3758/s13428-011-0081-0.
6. Belot, M., Duch, R., & Miller, L. (2010). Who should be called to the lab? A comprehensive comparison of students and non-students in classic experimental games. University of Oxford, Nuffield College Discussion Papers.((2010-001)).
7. Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20(3), 351–368. doi:10.1093/pan/mpr057.
8. Bock, O., Baetge, I., & Nicklisch, A. (2014). hroot: hamburg registration and organization online tool. European Economic Review, 71, 117–120. doi:10.1016/j.euroecorev.2014.07.003.
9. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5. doi:10.1177/1745691610393980.
10. Carpenter, J., & Seki, E. (2011). Do social preferences increase productivity? field experimental evidence from fishermen in toyama bay. Economic Inquiry, 49(2), 612–630. doi:10.1111/j.1465-7295.2009.00268.x.
11. Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1), 112–130. doi:10.3758/s13428-013-0365-7.
12. Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7), 1131–1139.
13. Chesney, T., Chuah, S. H., & Hoffmann, R. (2009). Virtual world experimentation: An exploratory study. Journal of Economic Behavior & Organization, 72(1), 618–635. doi:10.1016/j.jebo.2009.05.026.
14. Cleave, B. L., Nikiforakis, N., & Slonim, R. (2013). Is there selection bias in laboratory experiments? The case of social and risk preferences. Experimental Economics, 16(3), 372–382. doi:10.1007/s10683-012-9342-8.
15. Difallah, D. E., Catasta, M., Demartini, G., Ipeirotis, P. G., & Cudre-Mauroux, P. (2015). The Dynamics of Micro-Task Crowdsourcing The Case of Amazon MTurk. Www’15 Companion: Proceedings of the 24th International Conference on World Wide Web, 617–617, doi:10.1145/2740908.2744109.
16. Egas, M., & Riedl, A. (2008). The economics of altruistic punishment and the maintenance of cooperation. Proceedings of the Royal Society B-Biological Sciences, 275(1637), 871–878. doi:10.1098/rspb.2007.1558.
17. Fehr, E., & Gächter, S. (2000). Cooperation and punishment in public goods experiments. American Economic Review, 90(4), 980–994. doi:10.1257/aer.90.4.980.
18. Fehr, E., & Gächter, S. (2002). Altruistic punishment in humans. Nature, 415(6868), 137–140.
19. Fischbacher, U. (2007). z-Tree: Zurich toolbox for ready-made economic experiments. Experimental Economics, 10(2), 171–178. doi:10.1007/s10683-006-9159-4.
20. Gächter, S., & Herrmann, B. (2011). The limits of self-governance when cooperators get punished: Experimental evidence from urban and rural Russia. European Economic Review, 55(2), 193–210. doi:10.1016/j.euroecorev.2010.04.003.
21. Gächter, S., Herrmann, B., & Thoni, C. (2004). Trust, voluntary cooperation, and socio-economic background: survey and experimental evidence. Journal of Economic Behavior & Organization, 55(4), 505–531. doi:10.1016/j.jebo.2003.11.006.
22. Gallo, E., & Yan, C. (2015). The effects of reputational and social knowledge on cooperation. Proceedings of the National Academy of Sciences of the United States of America, 112(12), 3647–3652. doi:10.1073/pnas.1415883112.
23. Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of mechanical turk samples. Journal of Behavioral Decision Making, 26(3), 213–224. doi:10.1002/bdm.1753.
24. Greiner, B. (2015). Subject pool recruitment procedures: organizing experiments with ORSEE. Journal of the Economic Science Association, 1(1), 114–125. doi:10.1007/s40881-015-0004-4.
25. Guillen, P., & Veszteg, R. F. (2012). On “lab rats”. Journal of Socio-Economics, 41(5), 714–720. doi:10.1016/j.socec.2012.07.002.
26. Haigh, M. (2016). Has the standard cognitive reflection test become a victim of its own success? Advances in Cognitive Psychology, 12(3), 145–149. doi:10.5709/acp-0193-5.
27. Hauser, O. P., Hendriks, A., Rand, D. G., & Nowak, M. A. (2016). Think global, act local: Preserving the global commons. [Article]. Scientific reports, 6, 36079, doi:10.1038/srep36079.http://www.nature.com/articles/srep36079#supplementary-information.
28. Hergueux, J., & Jacquemet, N. (2015). Social preferences in the online laboratory: A randomized experiment. Experimental Economics, 18(2), 251–283. doi:10.1007/s10683-014-9400-5.
29. Herrmann, B., Thoni, C., & Gächter, S. (2008). Antisocial punishment across societies. Science, 319(5868), 1362–1367.
30. Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425. doi:10.1007/s10683-011-9273-9.
31. Ipeirotis, P. G. (2010). Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads, The ACM Magazine for Students, 17(2), 16–21.
32. Jenkins, S. P. (1995). Easy estimation methods for discrete-time duration models. Oxford Bulletin of Economics and Statistics, 57(1), 129–138. doi:10.1111/j.1468-0084.1995.tb00031.x.
33. Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahnik, S., Bernstein, M. J., et al. (2014). Investigating variation in replicability a “many labs’’ replication project. Social Psychology, 45(3), 142–152. doi:10.1027/1864-9335/a000178.
34. Krawczyk, M. (2011). What brings your subjects to the lab? A field experiment. Experimental Economics, 14(4), 482–489. doi:10.1007/s10683-011-9277-5.
35. Krupnikov, Y., & Levine, A. S. (2014). Cross-sample comparisons and external validity. Journal of Experimental Political Science, 1(1), 59.
36. List, J. A. (2004). Young, selfish and male: Field evidence of social preferences. Economic Journal, 114(492), 121–149. doi:10.1046/j.0013-0133.2003.00180.x.
37. Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2(02), 109–138. doi:10.1017/XPS.2015.19.
38. Nishi, A., Shirado, H., Rand, D. G., & Christakis, N. A. (2015). Inequality and visibility of wealth in experimental social networks. Nature, 526(7573), 426. doi:10.1038/nature15392.
39. Paolacci, G., & Chandler, J. (2014). Inside the turk: Understanding mechanical turk as a participant pool. Current Directions in Psychological Science, 23(3), 184–188. doi:10.1177/0963721414531598.
40. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.
41. Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163. doi:10.1016/j.jesp.2017.01.006.
42. Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46(4), 1023–1031. doi:10.3758/s13428-013-0434-y.
43. Pfattheicher, S., & Schindler, S. (2015). Understanding the dark side of costly punishment: The impact of individual differences in everyday sadism and existential threat. European Journal of Personality, 29(4), 498–505. doi:10.1002/per.2003.
44. Rand, D. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172–179. doi:10.1016/j.jtbi.2011.03.004.
45. Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G. E., Wurzbacher, O., Nowak, M. A., et al. (2014). Social heuristics shape intuitive cooperation. Nature Communications. doi:10.1038/ncomms4677.
46. Reips, U.-D. (2000). The Web experiment method: Advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psychological experiments on the internet (pp. 89–117). San Diego, CA: Academic Press.
47. Schmelz, K., & Ziegelmeyer, A. (2015). Social distance and control aversion: Evidence from the Internet and the laboratory. Research Paper Series Thurgau Institute of Economics and Department of Economics at the University of Konstanz, TWI-RPS 100.
48. Stagnaro, M. N., Arechar, A. A., & Rand, D. G. (2017). From good institutions to generous citizens: Top-down incentives to cooperate promote subsequent prosociality but not norm enforcement. Cognition. doi:10.1016/j.cognition.2017.01.017.
49. Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., et al. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10(5), 479–491.
50. Stoop, J., Noussair, C. N., & Van Soest, D. (2012). From the lab to the field: Cooperation among fishermen. Journal of Political Economy, 120(6), 1027–1056. doi:10.1086/669253.
51. Suri, S., & Watts, D. J. (2011). Cooperation and contagion in web-based, networked public goods experiments. PLoS ONE. doi:10.1371/journal.pone.0016836.
52. Wang, J., Suri, S., & Watts, D. J. (2012). Cooperation and assortativity with dynamic partner updating. Proceedings of the National Academy of Sciences of the United States of America, 109(36), 14363–14368. doi:10.1073/pnas.1120867109.
53. Weinberg, J. D., Freese, J., & McElhattan, D. (2014). Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociological Science, 1, 292–310.
54. Zhou, H. T., & Fishbach, A. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (Yet False) research conclusions. Journal of Personality and Social Psychology, 111(4), 493–504. doi:10.1037/pspa0000056.