Collecting image annotations using Amazon's Mechanical Turk

Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010 - Proceedings

Volumn , Issue , 2010, Pages 139-147

  Rashtchian, Cyrus ^a   Young, Peter ^a   Hodosh, Micah ^a   Hockenmaier, Julia ^a  

^a University of Illinois at Urbana Champaign   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS; QUALITY CONTROL;

AMAZON'S MECHANICAL TURKS; CROWD SOURCING; FREEFORMS; HIGH SPEED; LARGE AMOUNTS; LINGUISTIC DATA; LOW-COSTS; LOW-HIGH; MECHANICAL TURKS; TEXT ENTRY;

IMAGE ANNOTATION;

EID: 85090348677     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (10)

1. Chris Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using Amazon's Mechanical Turk. In Proceedings of EMNLP 2009.
2. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 2008. The PASCAL Visual Object Classes Challenge 2008 (VOC2008) Results. http://www.pascalnetwork.org/challenges/VOC/voc2008/workshop/.
3. Michael Kaisser and John Lowe. 2008. Creating a research collection of question answer sentence pairs with amazons mechanical turk. In LREC 2008.
4. Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with mechanical turk. In Proceedings of SIGCHI 2008.
5. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2009. Turkit: tools for iterative tasks on mechanical turk. In HCOMP'09: Proceedings of the ACM SIGKDD Workshop on Human Computation.
6. Rada Mihalcea and Carlo Strapparava. 2009. The lie detector: Explorations in the automatic recognition of deceptive language. In Proceedings of the ACLIJCNLP 2009 Conference Short Papers.
7. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP 2008.
8. Alexander Sorokin and David Forsyth. 2008. Utility data annotation with amazon mechanical turk. In Computer Vision and Pattern Recognition Workshop.
9. Martin I. Tietze, Andi Winterboer, and Johanna D. Moore. 2009. The effect of linguistic devices in information presentation messages on comprehension and recall. In Proceedings of ENLG 2009.
10. Omar F. Zaidan and Chris Callison-Burch. 2009. Feasibility of human-in-the-loop minimum error rate training. In Proceedings of EMNLP 2009.