Inside the Turk: Understanding Mechanical Turk as a Participant Pool

Current Directions in Psychological Science

Volumn 23, Issue 3, 2014, Pages 184-188

  Paolacci, Gabriele ^a   Chandler, Jesse ^b,c  

^a ERASMUS UNIVERSITY   (Netherlands)

^b UNIVERSITY OF MICHIGAN   (United States)

^c PRIME Research   (United States)

Author keywords

data collection; Mechanical Turk; online experimentation

Indexed keywords

EID: 84901802988     PISSN: 09637214     EISSN: 14678721     Source Type: Journal    
DOI: 10.1177/0963721414531598     Document Type: Article

References (43)

1. ChandlerJ.MuellerP.PaolacciG. (2014). (See References). An article that discusses and explores empirically several methodological concerns connected to MTurk experimentation.
2. HortonJ. J.RandD. G.ZeckhauserR. J. (2011). (See References). An article that, in addition to validating MTurk for experimental research in economics, discusses extensively issues related to experimental validity on MTurk.
3. MasonW.SuriS. (2012). (See References). An article that provides an overview of MTurk and can be a useful beginner's guide for researchers considering collecting data from MTurk workers.
4. PaolacciG.ChandlerJ.IpeirotisP. G. (2010). (See References). An article that replicates classic findings in the decision-making literature and benchmarks MTurk against other participant pools.
5. Aker A.,El-Haj M.,Albakour M.-D.,Kruschwitz U. Paper presented at the 8th International Conference on Language Resources and Evaluation; 2012Istanbul, Turkey; 2012.
6. Amir O.,Rand D. G.,Gal Y. K.Economic games on the Internet: The effect of $1 stakes.PLoS ONE. 2012;7 (2): e31461.
7. Behrend T. S.,Sharek D. J.,Meade A. W.,Wiebe E. N.The viability of crowdsourcing for survey research.Behavior Research Methods. 2011;43:800-813.
8. Berinsky A.,Margolis M.,Sances M.Separating the shirkers from the workers? Making sure respondents pay attention on Internet surveys.American Journal of Political Science. in press;:.
9. BerinskyA. J.HuberG. A.LenzG. S. (2012). Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis, 20, 351-368.
10. Buhrmester M.,Kwang T.,Gosling S. D.Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data?.Perspectives on Psychological Science. 2011;6:3-5.
11. Chandler D.,Kapelner A.Breaking monotony with meaning: Motivation in crowdsourcing markets.Journal of Economic Behavior & Organization. 2013;90:123-133.
12. Chandler J.,Mueller P.,Paolacci G.Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers.Behavior Research Methods. 2014;46:112-130.
13. Chandler J.,Paolacci G.,Mueller P.Handbook of human computation. Michelucci P., ed. New York, NY: Springer; 2014:377-392.
14. Chilton L. B.,Horton J. J.,Miller R. C.,Azenkot S. Paper presented at the Association for Computing Machinery's Special Interest Group on Knowledge Discovery and Data Mining Workshop on Human Computation; 2010Washington, DC; 2010.
15. CrumpM. J. C.McDonnellJ. V.GureckisT. M. (2013). Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PLoS ONE, 8(3), Article e57410. Retrieved from http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0057410.
16. Eriksson K.,Simpson B.Emotional reactions to losing explain gender differences in entering a risky lottery.Judgment and Decision Making. 2010;5:159-163.
17. Fort K.,Adda G.,Cohen K. B.Amazon Mechanical Turk: Gold mine or coal mine?.Computational Linguistics. 2011;37:413-420.
18. Goodman J. K.,Cryder C. E.,Cheema A.Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples.Journal of Behavioral Decision Making. 2013;26:213-224.
19. Heer J.,Bostock M. Paper presented at the Special Interest Group on Computer-Human Interaction Conference on Human Factors in Computing Systems; 2010Atlanta, GA; 2010.
20. Holden C. J.,Dennie T.,Hicks A. D.Assessing the reliability of the M5-120 on Amazon's Mechanical Turk.Computers in Human Behavior. 2013;29:1749-1754.
21. Horton J. J.,Chilton L. B. Paper presented at the 11th Association for Computing Machinery Conference on Electronic Commerce; 2010Cambridge, MA; 2010.
22. Horton J. J.,Rand D. G.,Zeckhauser R. J.The online laboratory: Conducting experiments in a real labor market.Experimental Economics. 2011;14:399-425.
23. Kazai G. Paper presented at CrowdConf 2010; 2010San Francisco, CA; 2010.
24. Kazai G.,Kamps J.,Milic-Frayling N. Paper presented at the Proceedings of the 21st Association for Computing Machinery International Conference on Information and Knowledge Management; 2012Lahaina, HI; 2012.
25. Kittur A.,Chi E. H.,Suh B.Chi 2008: 26th Annual Chi Conference on Human Factors in Computing Systems Conference Proceedings. Burnett M.Costabile M. F.Catarci T.de Ruyter B.Tan D.Czerwinski M.Lund A., ed. 2008:453-456.
26. Klein R. A.,Ratliff K. A.,Vianello M.,Adams R. B.,Bahník Š.,Bernstein M. J., & Nosek B. A.Investigating variation in replicability: A "many labs" replication project.Social Psychology. in press;:.
27. Komarov S.,Reinecke K.,Gajos K. Z. Paper presented at the Proceedings of the Special Interest Group on Computer-Human Interaction Conference on Human Factors in Computing Systems; 2013Paris, France; 2013.
28. Kosara R.,Ziemkiewicz C. Paper presented at the Proceedings of the 3rd BELIV'10 Workshop: Beyond Time and Errors: Novel Evaluation Methods for Information Visualization; 2010Atlanta, GA; 2010.
29. MargeM.BanerjeeS.RudnickyA. I. (2010). Using the Amazon Mechanical Turk for transcription of spoken language. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (pp. 5270-5273). Retrieved from http://dblp.uni-trier.de/db/conf/icassp/icassp2010.html#MargeBR10.
30. Mason W.,Suri S.Conducting behavioral research on Amazon's Mechanical Turk.Behavior Research Methods. 2012;44:1-23.
31. Mason W.,Watts D. J. Proceedings of the Human Computation Workshop: Knowledge Discovery and Data Mining Conference; 2009Paris, France; 2009.
32. Mason W.,Watts D. J.Collaborative learning in networks.Proceedings of the National Academy of Sciences, USA. 2012;109:764-769.
33. Paolacci G.,Chandler J.,Ipeirotis P. G.Running experiments on Amazon Mechanical Turk.Judgment and Decision Making. 2010;5:411-419.
34. Peer E.,Vosgerau J.,Acquisti A.Reputation as a sufficient condition for data quality on Amazon Mechanical Turk.Behavior Research Methods. 2013;:.
35. Rand D. G.The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments.Journal of Theoretical Biology. 2012;299:172-179.
36. Rand D. G.,Greene J. D.,Nowak M. A.Spontaneous giving and calculated greed.Nature. 2012;489:427-430.
37. Rand D. G.,Peysakhovich A.,Kraft-Todd G. T.,Newman G. E.,Wurzbacher O.,Nowak M. A.,Greene J. D.Intuitive cooperation and the social heuristics hypothesis: Evidence from 15 time constraint studies.Nature Communications. in press;:.
38. Rogstadius J.,Kostakos V.,Kittur A.,Smus B.,Laredo J.,Vukovic M. Paper presented at the 5th International AAAI Conference on Weblogs and Social Media; 2011Barcelona, Spain; 2011.
39. Ross J.,Irani L.,Silberman M.,Zaldivar A.,Tomlinson B. Paper presented at the CHI'10 Extended Abstracts on Human Factors in Computing Systems; 2010Atlanta, GA; 2010.
40. Shapiro D. N.,Chandler J.,Mueller P. A.Using Mechanical Turk to study clinical populations.Clinical Psychological Science. 2013;1:213-220.
41. Simcox T.,Fiez J. A.Collecting response times using Amazon Mechanical Turk and Adobe Flash.Behavior Research Methods. 2014;46:95-111.
42. Sprouse J.A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory.Behavior Research Methods. 2011;43:155-167.
43. Suri S.,Goldstein D. G.,Mason W. A. Paper presented at the 3rd Association for the Advancement of Artificial Intelligence Human Computation Workshop; 2011San Francisco, CA; 2011.