Increasing cheat robustness of crowdsourcing tasks

Information Retrieval

Volumn 16, Issue 2, 2013, Pages 121-137

  Eickhoff, Carsten ^a   de Vries, Arjen P ^b  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

^b CWI   (Netherlands)

Author keywords

Crowdsourcing; Human factors; Stability; User experiments

Indexed keywords

EID: 84875647836     PISSN: 13864564     EISSN: 15737659     Source Type: Journal    
DOI: 10.1007/s10791-011-9181-9     Document Type: Article

References (34)

1. Alonso, O., & Mizzaro, S. (2009). Can we get rid of TREC assessors? Using mechanical turk for relevance assessment. In Proceedings of the SIGIR 2009 workshop on the future of IR evaluation (pp. 15-16). Citeseer.
2. Ambati, V., Vogel, S., & Carbonell, J. (2010). Active learning and crowd-sourcing for machine translation. Language Resources and Evaluation (LREC), 7, 2169-2174.
3. Amitay, E., Carmel, D., Lempel, R., & Soffer, A. (2004). Scaling IR-system evaluation using term relevance sets. In Proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval (pp. 10-17), ACM.
4. Baio, A. (2008). The faces of mechanical turk. http://waxy. org/2008/11/the_faces_of_mechanical_turk.
5. Clarke, C. (2009). Overview of the trec 2009 web track. Technical report, Waterloo University.
6. Dawid, A., & Skene, A. (1979). Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society Series C (Applied Statistics), 28(1), 20-28.
7. Eickhoff, C., Serdyukov, P., & de Vries, A. (2011). A combined topical/non-topical approach to identifying web sites for children. In Proceedings of the 4th ACM international conference on Web search and data mining (pp. 505-514). ACM.
8. Grady, C., & Lease, M. (2010). Crowdsourcing document relevance assessment with mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's mechanical turk (pp. 172-179). Association for Computational Linguistics.
9. Harman, D. (1993). Overview of the first trec conference. In Proceedings of the 16th annual international ACM SIGIR conference on research and development in information retrieval (pp. 36-47). ACM.
10. Harris, C. (2011). Youre hired! An examination of crowdsourcing incentive models in human resource tasks. Crowdsourcing for search and data mining (CSDM 2011) (p. 15).
11. Hirth, M., Hoßfeld, T., & Tran-Gia, P. (2010). Cheat-detection mechanisms for crowdsourcing. Technical report, University of Würzburg.
12. Hsueh, P., Melville, P., & Sindhwani, V. (2009). Data quality from crowdsourcing: A study of annotation selection criteria. In Proceedings of the NAACL HLT 2009 workshop on active learning for natural language processing (pp. 27-35). Association for Computational Linguistics.
13. Ipeirotis, P. (2010a). Be a top mechanical turk worker: You need $5 and 5 minutes. http://behind-the-enemy-lines. blogspot. com/2010/10/be-top-mechanical-turk-worker-you-need. html.
14. Ipeirotis, P. (2010b). Demographics of mechanical turk. Center for Digital Economy Research, NYU Stern School of Business, Working paper.
15. Kaufmann, N., Schulze, T., & Veit, D. (2011). More than fun and money. Worker motivation in crowdsourcing-A study on mechanical turk. In Proceedings of the 17th Americas Conference on Information Systems.
16. Kazai, G. (2011). In search of quality in crowdsourcing for search engine evaluation. Advances in Information Retrieval, 6611, 165-176.
17. Kazai, G., Kamps, J., & Milic-Frayling, N. (2011). Worker types and personality traits in crowdsourcing relevance labels. In Proceedings of 20th International Conference on Information and Knowledge Management (CIKM). ACM.
18. Khanna, S., Ratan, A., Davis, J., & Thies, W. (2010). Evaluating and improving the usability of mechanical turk for low-income workers in india. In Proceedings of the first ACM symposium on computing for development (p. 12). ACM.
19. Kittur, A., Chi, E., & Suh, B. (2008). Crowdsourcing user studies with mechanical turk. In Proceeding of the 26th annual SIGCHI conference on human factors in computing systems (pp. 453-456). ACM.
20. Lesher, G., & Sanelli, C. (2000). A web-based system for autonomous text corpus generation. In Proceedings of ISSAAC.
21. Little, G., Chilton, L., Goldman, M., & Miller, R. (2009). Turkit: Tools for iterative tasks on mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation (pp. 29-30). ACM.
22. Marcus, M., Marcinkiewicz, M., & Santorini, B. (1993). Building a large annotated corpus of english: The penn treebank. Computational linguistics 19(2), 313-330.
23. Moore, A., Ellison, R., & Linger, R. (2001). Attack modeling for information security and survivability. Technical report, Carnegie-Mellon University, Software Engineering Institute.
24. MTurk. (2011). Amazon mechanical turk-Artificial artificial intelligence. https://www. mturk. com/.
25. Pfleeger, C., & Pfleeger, S. (2007). Security in computing (Vol. 604). New Yok: Prentice Hall.
26. PuppyIR. (2011). PuppyIR: An open source environment to construct information services for children. http://www. puppyir. eu.
27. Riloff, E. (1996). Automatically generating extraction patterns from untagged text. In Proceedings of the national conference on artificial intelligence (pp. 1044-1049).
28. Ross, J., Irani, L., Silberman, M., Zaldivar, A., & Tomlinson, B. (2010). Who are the crowdworkers? Shifting demographics in mechanical turk. In Proceedings of the 28th of the international conference extended abstracts on human factors in computing systems (pp. 2863-2872). ACM.
29. Shneiderman, B. (1997). Designing the user interface: strategies for effective human-computer interaction. Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA.
30. Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. (2008). Cheap and fast-But is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing (pp. 254-263). Association for Computational Linguistics.
31. Soboroff, I., Nicholas, C., & Cahan, P. (2001). Ranking retrieval systems without relevance judgments. In Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval (pp. 66-73). ACM.
32. Soleymani, M., & Larson, M. (2010). Crowdsourcing for affective annotation of video: Development of a viewer-reported boredom corpus. In Proceedings of the ACM SIGIR 2010 workshop on crowdsourcing for search evaluation (CSE 2010) (pp. 4-8).
33. Sorokin, A., & Forsyth, D. (2008). Utility data annotation with amazon mechanical turk. In Computer vision and pattern recognition workshops, 2008. CVPRW'08. IEEE computer society conference on IEEE (pp. 1-8).
34. Wang, J., Faridani, S., Ipeirotis, P. (2011). Estimating the completion time of crowdsourced tasks using survival analysis models. Crowdsourcing for search and data mining (CSDM 2011) (p. 31).