Reputation-based worker filtering in crowdsourcing

Advances in Neural Information Processing Systems

Volumn 3, Issue January, 2014, Pages 2492-2500

  Jagabathula, Srikanth ^a   Subramanian, Lakshminarayanan ^b,c   Venkataraman, Ashwin ^b,c  

^a NEW YORK UNIVERSITY   (United States)

^b POLYTECHNIC UNIVERSITY   (United States)

^c NEW YORK UNIVERSITY ABU DHABI   (United Arab Emirates)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION SCIENCE;

AND FILTERS; COMPUTATIONALLY EFFICIENT; CROWDSOURCING; LABEL AGGREGATION; LABELING STRATEGY; NOISY LABELS; THEORETICAL GUARANTEES; WORST-CASE BEHAVIORS;

ALGORITHMS;

EID: 84937907918     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. B. Biggio, B. Nelson, and P. Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv:1206.6389, 2012.
2. N. Dalvi, A. Dasgupta, R. Kumar, and V. Rastogi. Aggregating crowdsourced binary ratings. In Proceedings of the 22nd international conference on World Wide Web, pages 285-294. International World Wide Web Conferences Steering Committee, 2013.
3. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, pages 20-28, 1979.
4. G. Demartini, D. E. Difallah, and P. Cudré-Mauroux. Zencrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In Proceedings of the 21st international conference on World Wide Web, pages 469-478. ACM, 2012.
5. J. S. Downs, M. B. Holbrook, S. Sheng, and L. F. Cranor. Are your participants gaming the system?: screening mechanical turk workers. In Proceedings of the 28th international conference on Human factors in computing systems, pages 2399-2402. ACM, 2010.
6. A. Ghosh, S. Kale, and P. McAfee. Who moderates the moderators?: crowdsourcing abuse detection in user-generated content. In Proceedings of the 12th ACM conference on Electronic commerce, pages 167-176. ACM, 2011.
7. N. J. Harvey, R. E. Ladner, L. Lovász, and T. Tamir. Semi-matchings for bipartite graphs and load balancing. In Algorithms and data structures, pages 294-306. Springer, 2003.
8. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation, pages 64-67. ACM, 2010.
9. D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. Neural Information Processing Systems, 2011.
10. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 453-456. ACM, 2008.
11. J. Le, A. Edmonds, V. Hester, and L. Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution.
12. K. Lee, P. Tamilarasan, and J. Caverlee. Crowdturfers, campaigns, and social media: tracking and revealing crowdsourced manipulation of social media. In 7th international AAAI conference on weblogs and social media (ICWSM), Cambridge, 2013.
13. V. C. Raykar and S. Yu. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. The Journal of Machine Learning Research, 13:491-518, 2012.
14. V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. The Journal of Machine Learning Research, 99:1297-1322, 2010.
15. B. I. Rubinstein, B. Nelson, L. Huang, A. D. Joseph, S.-h. Lau, S. Rao, N. Taft, and J. Tygar. Antidote: understanding and defending against poisoning of anomaly detectors. In Proceedings of the 9th ACM SIGCOMM conference on Internet measurement conference, pages 1-14. ACM, 2009.
16. P. Smyth, U. Fayyad, M. Burl, P. Perona, and P. Baldi. Inferring ground truth from subjective labelling of venus images. Advances in neural information processing systems, pages 1085-1092, 1995.
17. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing, pages 254-263. Association for Computational Linguistics, 2008.
18. N. Tran, B. Min, J. Li, and L. Subramanian. Sybil-resilient online content voting. In Proceedings of the 6th USENIX symposium on Networked systems design and implementation, pages 15-28. USENIX Association, 2009.
19. J. Vuurens, A. P. de Vries, and C. Eickhoff. How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In Proc. ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR11), pages 21-26, 2011.
20. J. B. Vuurens and A. P. de Vries. Obtaining high-quality relevance judgments using crowdsourcing. Internet Computing, IEEE, 16(5):20-27, 2012.
21. G. Wang, T. Wang, H. Zheng, and B. Y. Zhao. Man vs. machine: Practical adversarial detection of malicious crowdsourcing workers. In 23rd USENIX Security Symposium, USENIX Association, CA, 2014.
22. G. Wang, C. Wilson, X. Zhao, Y. Zhu, M. Mohanlal, H. Zheng, and B. Y. Zhao. Serf and turf: crowdturfing for fun and profit. In Proceedings of the 21st international conference on World Wide Web, pages 679-688. ACM, 2012.
23. P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. Advances in Neural Information Processing Systems, 23:2424-2432, 2010.
24. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in Neural Information Processing Systems, 22(2035-2043):7-13, 2009.