Beyond independent agreement: A tournament selection approach for quality assurance of human computation tasks

AAAI Workshop - Technical Report

Volumn WS-11-11, Issue , 2011, Pages 113-118

  Sun, Yu An ^a,b   Roy, Shourya ^a,b   Little, Greg D ^c  

^a XEROX CORPORATION   (United States)

^b INDIAN INSTITUTE OF TECHNOLOGY MADRAS   (India)

^c MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

HUMAN COMPUTATION; KEY TOPICS; RESEARCH FIELDS; TOURNAMENT SELECTION;

QUALITY ASSURANCE;

QUALITY CONTROL;

EID: 80055050754     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (12)

1. Su, Q., Pavlov, D., Chow, J., and Baker, W.C. 2007. Internet-scale collection of human-reviewed data. In Proc. Of the 16th international Conference on World Wide Web (WWW-2007).
2. Sorokin, A. and Forsyth, D. 2008. Utility data annotation with Amazon Mechanical Turk. In Proc. of the First IEEE Workshop on Internet Vision at CVPR-2008.
3. Kittur, A., Chi, E. H., and Suh, B. 2008. Crowdsourcing user studies with Mechanical Turk. In Proc. of the 26th Annual ACM Conference on Human Factors in Computing Systems (CHI-2008).
4. Snow, R., O'Connor, B., Jurafsky, D., and Ng, A.Y. 2008. Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In Proc. Of EMNLP-2008.
5. Callison-Burch, C. 2009. Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk. In Proc. of EMNLP-2009.
6. Andrew DeOrio and Valeria Bertacco. Human computing for EDA. In Proc. Of the: 46th Annual Design Automation Conference (DAC-2009)
7. Le, J., Edmonds, A., Hester, V., and Biewald, Lukas. Ensuring quality in crowdsourced search relevance evaluation. Workshop on Crowdsourcing for Search Evaluation. ACM SIGIR 2010
8. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast|but is it good?: evaluating non-expert annotations for natural language tasks. In EMNLP'08: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254-263. Association for Computational Linguistics, 2008.
9. P. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In KDD-HCOMP'10, 2010.
10. G. Kazai and N. Milic-Frayling. On the evaluation of the quality of relevance assessments collected through crowdsourcing. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pages 21-22, 2009.
11. Vamshi Ambati and Stephan Vogel. 2010. Can crowds build parallel corpora for machine translation systems? In Proceedings of the NAACL HLT Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk, pages 62-65.
12. G. Little and Y. Sun, Human OCR: Insights from a Complex Human Computation Process, Workshop on Crowdsourcing and Human Computation, Services, Studies and Platforms, ACM CHI, 2011