The average laboratory samples a population of 7,300 amazon mechanical turk workers

Judgment and Decision Making

Volumn 10, Issue 5, 2015, Pages 479-491

  Stewart, Neil ^a   Ungemach, Christoph ^b   Harris, Adam J L ^c   Bartels, Daniel M ^d   Newell, Ben R ^e   Paolacci, Gabriele ^f   Chandler, Jesse ^g  

^a UNIVERSITY OF WARWICK   (United Kingdom)

^b Och Spine at New York Presbyterian Hospitals   (United States)

^c UNIVERSITY COLLEGE LONDON   (United Kingdom)

^d UNIVERSITY OF CHICAGO   (United States)

^e UNIVERSITY OF NEW SOUTH WALES   (Australia)

^f ERASMUS UNIVERSITY   (Netherlands)

^g MATHEMATICA POLICY RESEARCH   (United States)

Author keywords

Amazon mechanical turk; Capture recapture; MTurk; Population size

Indexed keywords

EID: 84943144072     PISSN: None     EISSN: 19302975     Source Type: Journal    
DOI: None     Document Type: Article

References (33)

1. Baillargeon, S., & Rivest, L.-P. (2007). Rcapture: Loglinear models for capture-recapture in R. Journal of Statistical Software, 19. Retrieved from http://cran. r-project. org/web/packages/Rcapture/Rcapture. pdf.
2. Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800-813. doi: 10. 3758/s13428-011-0081-0.
3. Berinsky, A. J., Huber, G. A., &Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com's Mechanical Turk. Political Analysis, 20, 351-368. doi: 10. 1093/pan/mpr057.
4. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives On Psychological Science, 6, 3-5. doi: 10. 1177/1745691610393980.
5. Casey, L., Chandler, J., Levine, A. S., Proctor, A., & Strolovitch, D. (2015). Demographic characteristics of a large sample of us workers. Unpublished manuscript.
6. Chandler, J., Mueller, P., & Ipeirotis, P. G. (2013). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods. doi: 10. 3758/s13428-013-0365-7.
7. Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. (2015). Using non-naïve participants can reduce effect sizes. Psychological Science, 26, 1131-1139. doi: 10. 1177/0956797615585115.
8. Cormack, R. M. (1989). Log-linear models for capture recapture. Biometrics, 45, 395-413. doi: 10. 2307/2531485.
9. Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7-29. doi: 10. 1177/0956797613504966.
10. Fisher, N., Turner, S., Pugh, R., & Taylor, C. (1994). Estimating the number of homeless and homeless mentally ill people in north east Westminster by using capturerecapture analysis. British Medical Journal, 308, 27-30.
11. Fort, K., Adda, G., & Cohen, K. B. (2011). Amazon Mechanical Turk: Gold mine or coal mine? Computational Linguistics, 37, 413-420. doi: 10. 1162/COLI_a_00057.
12. Francis, C. M., Barker, R. J., & Cooch, E. G. (2013). Modeling demographic processes in marked populations: Proceedings of the EURING 2013 analytical meeting [Special Issue]. Ecology and Evolution, 5. doi: 10. 1111/2041-210X. 12300.
13. Glinski, R. J., Glinski, B. C., & Slatin, G. T. (1970). Nonnaivety contamination in conformity experiments: Sources, effects, and implications for control. Journal of Personality and Social Psychology, 16, 478-485. doi: 10. 1037/h0030073.
14. Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of mechanical turk samples. Journal of Behavioral Decision Making, 26, 213-224. doi: 10. 1002/bdm. 1753.
15. Hauser, D. J., & Schwarz, N. (2015). Attentive turkers: Mturk participants perform better on online attention checks than subject pool participants. Behavior Research Methods. doi: 10. 3758/s13428-015-0578-z.
16. Hertwig, R., & Ortmann, A. (2001). Experimental practices in economics: A methodological challenge for psychologists? Behavioral and Brain Sciences, 24, 383-451.
17. Hillygus, D. S., Jackson, N., & Young, M. (2014). Professional respondents in non-probability online panels. In M. Callegaro, R. Baker, P. Lavrakas, J. Krosnick, J. Bethlehem, & A. Gritz (Eds.), Online panel research: A data quality perspective (pp. 219-237). West Sussex, UK: Wiley.
18. Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399-425. doi: 10. 1007/s10683-011-9273-9.
19. Klein, R. A., Ratliff, K. A., Vianello, M., Adams, J., Reginald B., Bahnik, S., Bernstein, M. J.,. Nosek, B. A. (2014). Investigating variation in replicability: A "many labs" replication project. Social Psychology, 45, 142-152. doi: 10. 1027/1864-9335/a000178.
20. Laporte, R. E. (1994). Assessing the human condition: Capture-recapture techniques. British Medical Journal, 308, 5-6.
21. Mason, W., Suri, S., & Watts, D. J. (2014, June). Longrun learning in games of cooperation. In Proceedings of the 15th acm conference on economics and computation. ACM. doi: 10. 1145/2600057. 2602892.
22. Mullinix, K., Druckman, J., & Freese, J. (2014). The generalizability of survey experiments [Institute for Policy Research Northwestern University Working Paper Series]. Retrieved from http://www. ipr. northwestern. edu/publications/docs/workingpapers/2014/IPR-WP-14-19. pdf.
23. Orne, M. T. (1962). On the social-psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist, 17, 776-783. doi: 10. 1037/h0043424.
24. Otis, D. L., Burnham, K. P., White, G. C., & Anderson, D. R. (1978). Statistical inference from capture data on closed animal populations. In Wildlife monographs (Vol. 62). Wildlife Society.
25. Paolacci, G., & Chandler, J. (2014). Inside the turk: Understanding mechanical turk as a participant pool. Current Directions in Psychological Science, 23, 184-188. doi: 10. 1177/0963721414531598.
26. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419. Retrieved from http://journal. sjdm. org/10/10630a/jdm10630a. pdf.
27. Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G. E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2014). Social heuristics shape intuitive cooperation. Nature Communications, 5, e3677. doi: 10. 1038/ncomms4677.
28. Rivest, L. P., & Daigle, G. (2004). Loglinear models for the robust design in mark-recapture experiments. Biometrics, 60, 100-107. doi: 10. 1111/j. 0006-341X. 2004. 00157. x.
29. Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006). Experimental study of inequality and unpredictability in an artificial cultural market. Science, 311, 854-856. doi: 10. 1126/science. 1121066.
30. Seber, G. A. F. (1982). The estimation of animal abundance and related parameters. New York: Macmillan.
31. Simonsohn, U. (2013). Evaluating replication results. Unpublished manuscript. Retrieved from http://ssrn. com/abstract=2259879.
32. Williams, B., Nichols, J., & Conroy, M. (2002). Analysis and management of animal populations. San Diego, CA: Elsevier.
33. Woods, A. T., Velasco, C., A., L. C., X., W., & Spence, C. (2015). Conducting perception research over the internet: A tutorial review. Manuscript submitted for publication.