Identifying careless workers in crowdsourcing platforms: A game theory approach

SIGIR 2016 - Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2016, Pages 857-860

  Moshfeghi, Yashar ^a   Huertas Rosero, Alvaro F ^a   Jose, Joemon M ^a  

^a UNIVERSITY OF GLASGOW   (United Kingdom)

Author keywords

Chicken game; Crowdsourcing; Game theory; Relevance assessment

Indexed keywords

CROWDSOURCING; INFORMATION RETRIEVAL; QUALITY ASSURANCE; RISK PERCEPTION;

CHICKEN GAME; CROWDSOURCING PLATFORMS; EXPERIMENTAL EVIDENCE; GAME SCENARIO; RELEVANCE ASSESSMENTS; RISK AVERSE; RISK NEUTRALS; TASK COMPLETION TIME;

GAME THEORY;

EID: 84980378623     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2911451.2914756     Document Type: Conference Paper

References (14)

1. O. Alonso. Implementing crowdsourcing-based relevance experimentation: an industrial perspective. Inf. Retr., pages 1-20, 2013.
2. B. Carterette and I. Soboroff. The Effect of Assessor Error on IR System Evaluation. In SIGIR '10, pages 539-546, 2010.
3. C. Eickhoff, C. G. Harris, A. P. de Vries, and P. Srinivasan. Quality through flow and immersion: gamifying crowdsourced relevance assessments. In SIGIR '12, pages 871-880, 2012.
4. C. Grady and M. Lease. Crowdsourcing document relevance assessment with Mechanical Turk. In CSLDAMT '10, pages 172-179, 2010.
5. P. G. Ipeirotis, F. Provost, and J. Wang.
6. G. Kazai, J. Kamps, and N. Milic-Frayling. The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy. In CIKM '12, pages 2583-2586, 2012.
7. G. Kazai, J. Kamps, and N. Milic-Frayling. An Analysis of Human Factors and Label Accuracy in Crowdsourcing Relevance Judgments. Inf. Retr., 16(2):138-178, Apr. 2013.
8. J. Le, A. Edmonds, V. Hester, and L. Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 workshop on crowdsourcing for search evaluation, pages 21-26, 2010.
9. Y. Moshfeghi, A. F. H. Rosero, and J. M. Jose. A Game-Theory Approach for Effective Crowdsource-Based Relevance Assessment. ACM Trans. Intell. Syst. Technol., 7(4):55:1-55:25, Mar. 2016.
10. T. Straub, H. Gimpel, and F. Teschner. The negative effect of feedback on performance in crowd labor tournaments. Collective Intelligence 2014, 2014.
11. M. Szilagyi. Agent-Based Simulation of the N-Person Chicken Game. In Advances in Dynamic Game Theory, Volume 9, pages 696-703. Birkhäuser Boston, 2007.
12. L. Von Ahn and L. Dabbish. Designing games with a purpose. Communications of the ACM, 51(8):58-67, 2008.
13. E. Voorhees, D. Harman, N. I. of Standards, and T. (US). TREC: Experiment and evaluation in information retrieval, Volume 63. MIT press Cambridge, 2005.
14. Y. Zhao and Q. Zhu. Evaluation on crowdsourcing research: Current status and future direction. Information Systems Frontiers, pages 1-18, 2012.