Assessing crowdsourcing quality through objective tasks

Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012

Volumn , Issue , 2012, Pages 1456-1461

  Aker, Ahmet ^a   El Haj, Mahmoud ^b   Albakour, M Dyaa ^b   Kruschwitz, Udo ^b  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

^b UNIVERSITY OF ESSEX   (United Kingdom)

Author keywords

Evaluation; Mechanical Turk; Objective Metrics

Indexed keywords

ACADEMIC RESEARCH; AMAZON'S MECHANICAL TURKS; CROWDSOURCING PLATFORMS; EVALUATION; HUMAN ASSESSMENT; MECHANICAL TURKS; OBJECTIVE METRICS; PARADIGM SHIFTS;

CROWDSOURCING;

EID: 84984834296     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. M-Dyaa Albakour, Udo Kruschwitz, and Simon Lucas. 2010. Sentence-level Attachment Prediction. In Proceedings of the 1st Information Retrieval Facility Conference, volume 6107 of Lecture Notes in Computer Science, pages 6-19, Vienna. Springer.
2. Omar Alonso and Stefano Mizzaro. 2009. Can we get rid of TREC Assessors? Using Mechanical Turk for Relevance Assessment. In SIGIR '09: Workshop on The Future of IR Evaluation.
3. Omar Alonso, Daniel E. Rose, and Benjamin Stewart. 2008. Crowdsourcing for relevance evaluation. SIGIR Forum, 42(2):9-15.
4. Vikas Bhardwaj, Rebecca Passonneau, Ansaf Salleb-Aouissi, and Nancy Ide. 2010. Anveshan: a framework for analysis of multiple annotators' labeling behavior. In Proceedings of the Fourth Linguistic Annotation Workshop, LAWIV '10, pages 47-55, Stroudsburg, PA, USA. Association for Computational Linguistics.
5. Jon Chamberlain, Massimo Poesio, and Udo Kruschwitz. 2008. Phrase Detectives-A Web-based Collaborative Annotation Game. In In Proceedings of I-Semantics.
6. Wisam Dakka and Panagiotis G. Ipeirotis. 2008. Automatic extraction of useful facet hierarchies from text databases. In ICDE, pages 466-475. IEEE.
7. Pinar Donmez, Jaime Carbonell, and Jeff Schneider. 2009. Efficiently learning the accuracy of labeling sources for selective sampling. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '09, pages 259-268, New York, NY, USA.
8. Mahmoud El-Haj, Udo Kruschwitz, and Chris Fox. 2010. Using Mechanical Turk to Create a Corpus of Arabic Summaries. In Proceedings of the LREC Workshop on Semitic Languages, pages 36-39, Valletta, Malta. Donghui Feng, Sveva Besana, and Remi Zajac. 2009.
9. Acquiring High Quality Non-expert Knowledge from On-demand Workforce. In Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources, People's Web '09, pages 51-56, Morristown, NJ, USA. Association for Computational Linguistics.
10. Karën Fort, Gilles Adda, and K. Bretonnel Cohen. 2011. Amazon Mechanical Turk: Gold Mine or Coal Mine? Computational Linguistics.
11. Panagiotis G. Ipeirotis. 2010. Demographics of Mechanical Turk. SSRN eLibrary.
12. Michael Kaisser and John Lowe. 2008. Creating a Research Collection of Question Answer Sentence Pairs with Amazons Mechanical Turk. In Proceedings of the Sixth International Language Resources and Evaluation (LREC'08), Marrakech, Morocco, may. European Language Resources Association (ELRA). http://www. lrecconf. org/proceedings/lrec2008/.
13. Michael Kaisser, Marti A. Hearst, and John B. Lowe. 2008. Improving search results quality by customizing summary lengths.
14. Gabriella Kazai, Natasa Milic-Frayling, and Jamie Costello. 2009. Towards methods for the collective gathering and quality control of relevance assessments. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, SIGIR '09, pages 452-459, New York, NY, USA. ACM.
15. Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with mechanical turk. In CHI '08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 453-456, New York, NY, USA. ACM.
16. Winter Mason and Duncan J. Watts. 2010. Financial Incentives and the "Performance of Crowds". SIGKDD Explor. Newsl., 11:100-108, May.
17. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and Fast-But is it Good?: Evaluating Non-expert Annotations for Natural Language Tasks. In EMNLP '08: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254-263, Morristown, NJ, USA. Association for Computational Linguistics.
18. Alexander Sorokin and David Forsyth. 2008. Utility Data Annotation with Amazon Mechanical Turk. In Proceedings of the First IEEE Workshop on Internet Vision at CVPR 08, pages 1-8.
19. Qi Su, Dmitry Pavlov, Jyh-Herng Chow, and Wendell C. Baker. 2007. Internet-scale collection of humanreviewed data. In Proceedings of the 16th international conference on World Wide Web, WWW '07, pages 231-240, New York, NY, USA. ACM.
20. Jiayu Tang and Mark Sanderson. 2010. Evaluation and user preference study on spatial diversity. In ECIR, volume 5993 of Lecture Notes in Computer Science, pages 179-190. Springer.
21. Luis von Ahn. 2006. Games With A Purpose. IEEE Computer Magazine, pages 96-98.
22. Paul Wais, Shivaram Lingamneni, Duncan Cook, Jason Fennell, Benjamin Goldenberg, Daniel Lubarov, David Marin, and Hari Simons. 2010. Towards Building a High-QualityWorkforce with Mechanical Turk. In NIPS Workshop on Computational Social Science and the Wisdom of Crowds.
23. Yin Yang, Nilesh Bansal, Wisam Dakka, Panagiotis Ipeirotis, Nick Koudas, and Dimitris Papadias. 2009. Query by Document. In Proceedings of the Second ACM International Conference on Web Search and Data Mining, WSDM '09, pages 34-43, New York, NY, USA. ACM.
24. Omar Zaidan and Chris Callison-Burch. 2011. Crowdsourcing Translation: Professional Quality from Non-Professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1220-1229, Portland, Oregon, USA. Association for Computational Linguistics.