Crowdsourcing document relevance assessment with Mechanical Turk

Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010 - Proceedings

Volumn , Issue , 2010, Pages 172-179

  Grady, Catherine ^a   Lease, Matthew ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; HUMAN ENGINEERING; SEARCH ENGINES;

AMAZON'S MECHANICAL TURKS; HUMAN INTELLIGENCE; MECHANICAL TURKS; RELATED WORKS; RELEVANCE ASSESSMENTS; SEARCH QUERIES;

CROWDSOURCING;

EID: 85018109911     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (12)

1. Omar Alonso, Daniel E. Rose, and Benjamin Stewart. 2008. Crowdsourcing for relevance evaluation. SIGIR Forum, 42(2):9-15.
2. J. Aslam and V. Pavlu. 2008. A practical sampling strategy for efficient retrieval evaluation. Technical report, Northeastern University.
3. C. Buckley and E.M. Voorhees. 2004. Retrieval evaluation with incomplete information. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 25-32. ACM New York, NY, USA.
4. B. Carterette, J. Allan, and R. Sitaraman. 2006. Minimal test collections for retrieval evaluation. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 268-275.
5. B. Carterette, V. Pavlu, E. Kanoulas, J.A. Aslam, and J. Allan. 2009. If I Had a Million Queries. In Proceedings of the 31st European Conference on Information Retrieval, pages 288-300.
6. D. Horowitz and S.D. Kamvar. 2010. The Anatomy of a Large-Scale Social Search Engine. In Proc. of the 19th international conference on World wide web (WWW).
7. Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In KDD'02: Proceedings of the 8th SIGKDD international conference on Knowledge discovery and data mining, pages 133-142.
8. W. Mason and D.J. Watts. 2009. Financial incentives and the performance of crowds. In Proceedings of the ACM SIGKDD Workshop on Human Computation, pages 77-85. ACM.
9. R. Snow, B. O'Connor, D. Jurafsky, and A.Y. Ng. 2008. Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254-263. Association for Computational Linguistics.
10. L. Von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum. 2008. recaptcha: Human-based character recognition via web security measures. Science, 321(5895):1465.
11. E.M. Voorhees. 2002. The philosophy of information retrieval evaluation. Lecture Notes in Computer Science, pages 355-370.
12. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. 2009. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. Proceedings of the 2009 Neural Information Processing Systems (NIPS) Conference.