Demographics and dynamics of Mechanical Turk workers

WSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining

Volumn 2018-Febuary, Issue , 2018, Pages 135-143

  Difallah, Djellel ^a   Filatova, Elena ^b   Ipeirotis, Panos ^a  

^a NEW YORK UNIVERSITY   (United States)

^b CITY UNIVERSITY OF NEW YORK   (United States)

Author keywords

Amazon mechanical Turk; Capture recapture; Crowdsourcing; Demographics; Dynamics; Selection bias; Surveys

Indexed keywords

CROWDSOURCING; DATA MINING; DYNAMICS; INFORMATION RETRIEVAL; POPULATION STATISTICS; SURVEYING; WEBSITES;

AMAZON MECHANICAL TURKS; CAPTURE-RECAPTURE; CROWDSOURCING PLATFORMS; DEMOGRAPHICS; HEAVY-TAILED DISTRIBUTION; MECHANICAL TURKS; SELECTION BIAS; SURVEY DATA;

SURVEYS;

EID: 85046905206     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/3159652.3159661     Document Type: Conference Paper

References (36)

1. Steven C. Amstrup, Trent L. McDonald, and Bryan F. J. Manly. 2010. Handbook of Capture-Recapture Analysis. Princeton University Press, Princeton, NJ.
2. Joanna J. Arch and Alaina L. Carr. 2016. Using Mechanical Turk for research on cancer survivors. Psycho-Oncology Forthcoming (2016), -. PON-15-0731. R1.
3. Antonio A. Arechar, Simon Gächter, and Lucas Molleman. 2017. Conducting interactive experiments online. Experimental Economics Forthcoming (09 May 2017), 1-33.
4. K. P. Burnham and W. S. Overton. 1978. Estimation of the Size of a Closed Population when Capture Probabilities vary Among Animals. Biometrika 65, 3(1978), 625-633.
5. Chris Callison-Burch and Mark Dredze. 2010. Creating Speech and Language Data with Amazon's Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk (CSLDAMT '10). Association for Computational Linguistics, Stroudsburg, PA, USA, 1-12.
6. Krista Casler, Lydia Bickel, and Elizabeth Hackett. 2013. Separate but Equal? A Comparison of Participants and Data Gathered via Amazon's MTurk, Social Media, and Face-to-Face Behavioral Testing. Computers in Human Behavior 29(2013), 2156-2160.
7. Jesse Chandler, Gabriele Paolacci, and Pam Mueller. 2013. Risks and Rewards of Crowdsourcing Marketplaces. In Handbook of Human Computation. Springer, New York, 377-392.
8. Jesse Chandler and Danielle Shapiro. 2016. Conducting Clinical Research Using Crowdsourced Convenience Samples. Annual Review of Clinical Psychology 12(2016), 35-81.
9. Anne Chao. 1987. Estimating the population size for capture-recapture data with unequal catchability. Biometrics 43, 4 (Dec. 1987), 783-791.
10. R. M. Cormack. 1964. Estimates of Survival From the Sighting of Marked Animals. Biometrics 51(1964), 429-438.
11. Francisco Cribari-Neto and Achim Zeileis. 2010. Beta Regression in R. Journal of Statistical Software, Articles 34, 2(2010), 1-24.
12. Damien L. Crone and Lisa A. Williams. 2017. Crowdsourcing participants for psychological research in Australia: A test of Microworkers. Australian Psychological Society 69(2017), 39-47. Issue 1.
13. Joost de Winter, Miltos Kyriakidis, Dimitra Dodou, and Riender Happee. 2015. Using CrowdFlower to Study the Relationship between Self-reported Violations and Traffic Accidents. Procedia Manufacturing 3(2015), 2518-2525.
14. Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G Ipeirotis, and Philippe Cudré-Mauroux. 2015. The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, ACM, New York, NY, USA, 238-247.
15. Karën Fort, Gilles Adda, and K. Bretonnel Cohen. 2011. Amazon Mechanical Turk: Gold Mine or Coal Mine? Computational Linguistics 37(2011), 413-420. Issue 2.
16. Ilka H. Gleibs. 2017. Are all research fields equal? Rethinking practice for the use of data from crowd-sourcing market places. Behavior Research Methods 49 (August 2017), 1333-1342. Issue 4.
17. Panagiotis Ipeirotis. 2010. Demographics of Mechanical Turk. Technical Report CeDER-10-01. New York University.
18. G. M. Jolly. 1965. Explicit Estimates from Capture-Recapture Data With Both Death and Immigration-Stochastic Model. Biometrika 52(1965), 225-247.
19. Jeremy Kees, Christopher Berry, Scot Burton, and Kim Sheehan. 2017. An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk. Journal of Advertising 46, 1(2017), 141-155.
20. Frederick Charles Lincoln. 1930. Calculating waterfowl abundance on the basis of banding returns. US Dept. of Agriculture, Washington, DC.
21. Jianguo Lu and Dingding Li. 2010. Estimating deep web data source size by capture-recapture method. Information Retrieval 13, 1 (February 2010), 70-95.
22. Winter Mason and Duncan J. Watts. 2009. Financial Incentives and the "Performance of Crowds". In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '09). ACM, New York, NY, USA, 77-85.
23. Joshua D. Miller, Michael Crowe, Brandon Weiss, Jessica L. Maples-Keller, and Donald R. Lynam. 2017. Using online, crowdsourcing platforms for data collection in personality disorder research: The example of Amazon's Mechanical Turk. Personality Disorders: Theory, Research, and Treatment 8, 1(2017), 26-34.
24. Shirley Pledger, Kenneth H. Pollock, and James L. Norris. 2003. Open Capture-Recapture Models with Heterogeneity: I. Cormack-Jolly-Seber Model. Biometrics 59, 4 (Dec. 2003), 786-794.
25. Shirley Pledger, Kenneth H. Pollock, and James L. Norris. 2010. Open Capture-Recapture Models with Heterogeneity: II. Jolly-Seber Model. Biometrics 66, 3 (September 2010), 883-890.
26. Joel Ross, Lilly Irani, M. Six Silberman, Andrew Zaldivar, and Bill Tomlinson. 2010. Who Are the Crowdworkers?: Shifting Demographics in Mechanical Turk. In CHI '10 Extended Abstracts on Human Factors in Computing Systems (CHI EA '10). ACM, New York, NY, USA, 2863-2872.
27. Carl James Schwarz and A. Neil Arnason. 1996. A General Methodology for the Analysis of Capture-Recapture Experiments in Open Populations. Biometrics 52, 3(1996), 860-873.
28. George Seber. 1982. The Estimation of Animal Abundance and Related Parameters. Charles Griffin, London, UK.
29. G. A. F. Seber. 1964. A Note on the Multiple Recapture Census. Biometrika 52(1964), 249-259.
30. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '08). ACM, New York, NY, USA, 614-622.
31. Michael Smithson and Jay Verkuilen. 2006. A better lemon squeezer? Maximumlikelihood regression with beta-distributed dependent variables. Psychological methods 11(2006), 54-71. Issue 1.
32. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and Fast-but is It Good?: Evaluating Non-expert Annotations for Natural Language Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP '08). Association for Computational Linguistics, Stroudsburg, PA, USA, 254-263.
33. Vanessa V. Sochat, Ian W. Eisenberg, A. Zeynep Enkavi, Jamie Li, Patrick G. Bissett, and Russell A. Poldrack. 2016. The Experiment Factory: Standardizing Behavioral Experiments. Frontiers in Psychology 7(2016), 610.
34. Neil Stewart, Christoph Ungemach, Adam J. L. Harris, Daniel M. Bartels, Ben R. Newell, Gabriele Paolaccik, and Jesse Chandler. 2015. The average laboratory samples a population of 7, 300 Amazon Mechanical Turk workers. Judgment and Decision Making 10, 5(2015), 479-491.
35. Keela S. Thomson and Daniel M. Oppenheimer. 2016. Investigating an alternate form of the cognitive reflection test. Judgment and Decision Making 11, 1(2016), 99-113.
36. Beth Trushkowsky, Tim Kraska, Michael J. Franklin, and Purnamrita Sarkar. 2016. Answering Enumeration Queries with the Crowd. Commun. ACM 59, 1(2016), 118-127.