Are your participants gaming the system? Screening mechanical Turk workers

Conference on Human Factors in Computing Systems - Proceedings

Volumn 4, Issue , 2010, Pages 2399-2402

  Downs, Julie S ^a   Holbrook, Mandy B ^a   Sheng, Steve ^b   Cranor, Lorrie Faith ^b,c  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

^c NONE   (United States)

Author keywords

crowdsourcing; mechanical turk; screening; survey

Indexed keywords

SCREENING PROCESS; SCREENING SURVEY;

HUMAN ENGINEERING;

SURVEYS;

EID: 77954009273     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1753326.1753688     Document Type: Conference Paper

References (8)

1. Amazon Mechanical Turk. https://www.mturk.com
2. Amazon Mechanical Turk: Best Practices Guide. http://mturkpublic.s3. amazonaws.com/docs/MTURK-BP.pdf
3. Atwood, J. Is Amazon's Mechanical Turk a Failure? Coding Horror: Programming and Human Factors, April 9, 2007. http://www.codinghorror.com/blog/ archives/000828.html
4. Callison-Burch, C. Fast, cheap, and creative: Evaluating translation quality using Amazon's Mechanical Turk, In Proc. EMNLP 2009, ACL and AFNLP (2009), 286-295.
5. Ipeirotis, P. Turker demographics vs. Internet demographics. http://behind-the-enemylines.blogspot.com/2009/03/turker-demographics- vsinternet.html. 2009.
6. Kittur, A., Chi, E.H., and Suh, B. Crowdsourcing User Studies with Mechanical Turk. In Proc. CHI 2008, ACM Press, (2008), 453-456.
7. MdFedries, P. Technically Speaking: It's a Wiki, Wiki World. IEEE spectrum, 43, (2006), 88.
8. Sheng, S., Holbrook, M., Kumaraguru, P., Cranor, L., & Downs, J. Who falls for phish? A demographic analysis of phishing susceptibility and effectiveness of interventions. In Proc. CHI 2010, ACM Press, (2010).