Understanding malicious behavior in crowdsourcing platforms: The case of online surveys

Conference on Human Factors in Computing Systems - Proceedings

Volumn 2015-April, Issue , 2015, Pages 1631-1640

  Gadiraju, Ujwal ^a   Kawase, Ricardo ^a   Dietze, Stefan ^a   Demartini, Gianluca ^b  

^a L3S RESEARCH CENTER   (Germany)

^b UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Crowdsourcing; Malicious intent; Microtasks; Online surveys; User behavior

Indexed keywords

HUMAN COMPUTER INTERACTION; HUMAN ENGINEERING; QUALITY CONTROL; SURVEYS;

CROWDSOURCING; MALICIOUS INTENT; MICROTASKS; ONLINE SURVEYS; USER BEHAVIORS;

BEHAVIORAL RESEARCH;

EID: 84940423017     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2702123.2702443     Document Type: Conference Paper

References (27)

1. Baba, Y., Kashima, H., Kinoshita, K., Yamaguchi, G., and Akiyoshi, Y. Leveraging crowdsourcing to detect improper tasks in crowdsourcing marketplaces. In Twenty-Fifth IAAI Conference (2013).
2. Behrend, T. S., Sharek, D. J., Meade, A. W., and Wiebe, E. N. The viability of crowdsourcing for survey research. Behavior research methods 43, 3 (2011), 800-813.
3. Brabham, D. C. Crowdsourcing as a Model for Problem Solving. Convergence: The International Journal of Research into New Media Technologies 14, 1 (Feb. 2008), 75-90.
4. Buchholz, S., and Latorre, J. Crowdsourcing preference tests, and how to detect cheating. In INTERSPEECH (2011), 3053-3056.
5. Corbin, J., and Strauss, A. Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage, 2008.
6. Difallah, D. E., Demartini, G., and Cudŕe-Mauroux, P. Mechanical cheat: Spamming schemes and adversarial techniques on crowdsourcing platforms. In CrowdSearch (2012), 26-30.
7. Dow, S., Kulkarni, A., Bunge, B., Nguyen, T., Klemmer, S., and Hartmann, B. Shepherding the crowd: Managing and providing feedback to crowd workers. In CHI'11 Extended Abstracts on Human Factors in Computing Systems, ACM (2011), 1669-1674.
8. Dow, S., Kulkarni, A., Klemmer, S., and Hartmann, B. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, ACM (2012), 1013-1022.
9. Eickhoff, C., and de Vries, A. How crowdsourcable is your task. In Proceedings of the workshop on crowdsourcing for search and data mining (CSDM) at the fourth ACM international conference on web search and data mining (WSDM) (2011), 11-14.
10. Eickhoff, C., and de Vries, A. P. Increasing cheat robustness of crowdsourcing tasks. Information retrieval 16, 2 (2013), 121-137.
11. Estelĺes-Arolas, E., and Gonźalez-Ladŕon-de Guevara, F. Towards an integrated crowdsourcing definition. Journal of Information science 38, 2 (2012), 189-200.
12. Gadiraju, U., Kawase, R., and Dietze, S. A taxonomy of microtasks on the web. In Proceedings of the 25th ACM conference on Hypertext and social media, ACM (2014), 218-223.
13. Gennaro, R., Gentry, C., and Parno, B. Non-interactive verifiable computing: Outsourcing computation to untrusted workers. In Advances in Cryptology-CRYPTO 2010. Springer, 2010, 465-482.
14. Glass, R. L., and Vessey, I. Contemporary application-domain taxonomies. IEEE Software 12, 4 (1995), 63-76.
15. Ipeirotis, P. G., Provost, F., and Wang, J. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation, ACM (2010), 64-67.
16. Kaufmann, N., Schulze, T., and Veit, D. More than fun and money. worker motivation in crowdsourcing - A study on mechanical turk. In AMCIS (2011).
17. Kazai, G., Kamps, J., and Milic-Frayling, N. Worker types and personality traits in crowdsourcing relevance labels. In Proceedings of the 20th ACM international conference on Information and knowledge management, ACM (2011), 1941-1944.
18. Kazai, G., Kamps, J., and Milic-Frayling, N. An analysis of human factors and label accuracy in crowdsourcing relevance judgments. Inf. Retr. 16, 2 (2013), 138-178.
19. Kittur, A., Chi, E. H., and Suh, B. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems, ACM (2008), 453-456.
20. Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., and Horton, J. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work, ACM (2013), 1301-1318.
21. Marshall, C. C., and Shipman, F. M. Experiences surveying the crowd: Reflections on methods, participation, and reliability. In Proceedings of the 5th Annual ACM Web Science Conference, WebSci '13, ACM (New York, NY, USA, 2013), 234-243.
22. Mason, W., and Watts, D. J. Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter 11, 2 (2010), 100-108.
23. Ntalianis, K., Tsapatsoulis, N., Doulamis, A., and Matsatsinis, N. Automatic annotation of image databases based on implicit crowdsourcing, visual concept modeling and evolution. Multimedia Tools and Applications 69, 2 (2014), 397-421.
24. Oleson, D., Sorokin, A., Laughlin, G. P., Hester, V., Le, J., and Biewald, L. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. Human computation 11 (2011), 11.
25. Ross, J., Irani, L., Silberman, M., Zaldivar, A., and Tomlinson, B. Who are the crowdworkers?: Shifting demographics in mechanical turk. In CHI'10 Extended Abstracts on Human Factors in Computing Systems, ACM (2010), 2863-2872.
26. Wang, G., Wilson, C., Zhao, X., Zhu, Y., Mohanlal, M., Zheng, H., and Zhao, B. Y. Serf and turf: Crowdturfing for fun and profit. In Proceedings of the 21st international conference on World Wide Web, ACM (2012), 679-688.
27. Yuen, M.-C., King, I., and Leung, K.-S. A survey of crowdsourcing systems. In Privacy, security, risk and trust (passat), 2011 ieee third international conference on and 2011 ieee third international conference on social computing (socialcom), IEEE (2011), 766-773.