Exploiting document content for efficient aggregation of crowdsourcing votes

International Conference on Information and Knowledge Management, Proceedings

Volumn 19-23-Oct-2015, Issue , 2015, Pages 783-790

  Davtyan, Martin ^a   Eickhoff, Carsten ^a   Hofmann, Thomas ^a  

^a ETH ZURICH   (Switzerland)

Author keywords

Clustering hypothesis; Crowdsourcing; Relevance assessment

Indexed keywords

KNOWLEDGE MANAGEMENT;

AGGREGATION METHODS; AGGREGATION SCHEMES; CLUSTERING HYPOTHESIS; CORPUS ANNOTATIONS; DOCUMENT CONTENTS; DOCUMENT SIMILARITY; RELEVANCE ASSESSMENTS; RELEVANCE JUDGMENT;

CROWDSOURCING;

EID: 84958231872     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2806416.2806460     Document Type: Conference Paper

References (33)

1. Omar Alonso, Daniel E Rose, and Benjamin Stewart. Crowdsourcing for relevance evaluation. In ACM SigIR Forum, volume 42, pages 9-15. ACM, 2008.
2. Roi Blanco, Harry Halpin, Daniel M Herzig, Peter Mika, Jeffrey Pound, Henry S Thompson, and Thanh Tran Duc. Repeatable and reliable search system evaluation using crowdsourcing. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 923-932. ACM, 2011.
3. Daren C Brabham. Moving the crowd at threadless: Motivations for participation in a crowdsourcing application. Information, Communication & Society, 13(8):1122-1145, 2010.
4. Chris Buckley and Ellen M Voorhees. Retrieval evaluation with incomplete information. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 25-32. ACM, 2004.
5. Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. Clueweb09 data set, 2009.
6. Ben Carterette, James Allan, and Ramesh Sitaraman. Minimal test collections for retrieval evaluation. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 268-275. ACM, 2006.
7. Vitor R Carvalho, Matthew Lease, and Emine Yilmaz. Crowdsourcing for search evaluation. In ACM Sigir forum, volume 44, pages 17-22. ACM, 2011.
8. Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics, pages 20-28, 1979.
9. Djellel Eddine Difallah, Gianluca Demartini, and Philippe Cudré-Mauroux. Pick-a-crowd: tell me what you like, and i'll tell you what to do. In Proceedings of the 22nd international conference on World Wide Web, pages 367-374. International World Wide Web Conferences Steering Committee, 2013.
10. Carsten Eickhoff and Arjen P de Vries. Increasing cheat robustness of crowdsourcing tasks. Information retrieval, 16(2):121-137, 2013.
11. Carsten Eickhoff, Christopher G Harris, Arjen P de Vries, and Padmini Srinivasan. Quality through flow and immersion: gamifying crowdsourced relevance assessments. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, pages 871-880. ACM, 2012.
12. Catherine Grady and Matthew Lease. Crowdsourcing document relevance assessment with mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's mechanical turk, pages 172-179. Association for Computational Linguistics, 2010.
13. Matthias Hirth, Tobias Hoßfeld, and Phuoc Tran-Gia. Cheat-detection mechanisms for crowdsourcing. University of Würzburg, Tech. Rep, 4, 2010.
14. Mehdi Hosseini, Ingemar J Cox, Nataša Milić-Frayling, Gabriella Kazai, and Vishwa Vinay. On aggregating labels from multiple crowd workers to infer relevance of documents. In Advances in information retrieval, pages 182-194. Springer, 2012.
15. Panos Ipeirotis. Crowdsourcing using mechanical turk: quality management and scalability. In Proceedings of the 8th International Workshop on Information Integration on the Web: in conjunction with WWW 2011, page 1. ACM, 2011.
16. Thorsten Joachims. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133-142. ACM, 2002.
17. David R Karger, Sewoong Oh, and Devavrat Shah. Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 62(1):1-24, 2014.
18. Gabriella Kazai, Jaap Kamps, and Natasa Milic-Frayling. Worker types and personality traits in crowdsourcing relevance labels. In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 1941-1944. ACM, 2011.
19. Gabriella Kazai, Jaap Kamps, and Natasa Milic-Frayling. An analysis of human factors and label accuracy in crowdsourcing relevance judgments. Information retrieval, 16(2):138-178, 2013.
20. Gabriella Kazai and Natasa Milic-Frayling. On the evaluation of the quality of relevance assessments collected through crowdsourcing. In SIGIR 2009 Workshop on the Future of IR Evaluation, page 21, 2009.
21. Aniket Kittur, Ed H Chi, and Bongwon Suh. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 453-456. ACM, 2008.
22. Matthew Lease and Gabriella Kazai. Overview of the trec 2011 crowdsourcing track. In Proceedings of the text retrieval conference (TREC), 2011.
23. Matthew Lease and Emine Yilmaz. Crowdsourcing for information retrieval. In ACM SIGIR Forum, volume 45, pages 66-75. ACM, 2012.
24. Catherine C Marshall and Frank M Shipman. The ownership and reuse of visual media. In Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries, pages 157-166. ACM, 2011.
25. Thomas P. Minka. Expectation propagation for approximate bayesian inference. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI'01, pages 362-369, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc.
26. Wei Tang and Matthew Lease. Semi-supervised consensus labeling for crowdsourcing. In SIGIR 2011 workshop on crowdsourcing for information retrieval (CIR), 2011.
27. Cornelis J van Rijsbergen and Karen Spärck Jones. A test for the separation of relevant and non-relevant documents in experimental retrieval collections. Journal of Documentation, 29(3):251-257, 1973.
28. Ellen M Voorhees. The philosophy of information retrieval evaluation. In Evaluation of cross-language information retrieval systems, pages 355-370. Springer, 2002.
29. Ellen M Voorhees, Donna K Harman, et al. TREC: Experiment and evaluation in information retrieval, volume 1. MIT press Cambridge, 2005.
30. Jeroen Vuurens, Arjen P de Vries, and Carsten Eickhoff. How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In Proc. ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR'11), pages 21-26, 2011.
31. Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. the MIT Press, 2(3):4, 2006.
32. Yan Yan, Glenn M Fung, Rómer Rosales, and Jennifer G Dy. Active learning from crowds. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 1161-1168, 2011.
33. Yu Zhang and Mihaela van der Schaar. Reputation-based incentive protocols in crowdsourcing applications. In INFOCOM, 2012 Proceedings IEEE, pages 2140-2148. IEEE, 2012.