How good is the crowd at "real" WSD?

ACL HLT 2011 - LAW 2011: 5th Linguistic Annotation Workshop, Proceedings

Volumn , Issue , 2011, Pages 30-37

  Hong, Jisup ^a   Baker, Collin F ^a  

^a INTERNATIONAL COMPUTER SCIENCE INSTITUTE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AMAZON'S MECHANICAL TURKS; FRAMENET; LINGUISTIC DATA; LOCAL EXPERTS; SEMANTIC ROLES; SENTIMENT ANALYSIS; WISDOM OF THE CROWDS; WORD SENSE;

DATA MINING; SNOW;

LINGUISTICS;

EID: 84866743828     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (10)

1. Chris Callison-Burch and Mark Dredze, editors. 2010. Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Los Angeles, June. Association for Computational Linguistics.
2. Katrin Erk and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 440-449, Singapore, August. Association for Computational Linguistics.
3. Michael Heilman and Noah A. Smith. 2010. Rating computer-generated questions with Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 35-40, Los Angeles, June. Association for Computational Linguistics.
4. Jeff Howe. 2008. Crowdsourcing. Crown Business, New York.
5. Adam Kilgarriff, Pavel Rychly, Pavel Smrz, and David Tugwell. July 2004. The Sketch Engine. In Proceedings of EURALEX 2004, Lorient, France.
6. Bart Mellebeek, Francesc Benavent, Jens Grivolla, Joan Codina, Marta R. Costa-Juss̀a, and Rafael Banchs. 2010. Opinion mining of spanish customer comments with non-expert annotations on mechanical turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 114-121, Los Angeles, June. Association for Computational Linguistics.
7. Josef Ruppenhofer, Michael Ellsworth, Miriam R. L. Petruck, Christopher R. Johnson, and Jan Scheffczyk. 2006. FrameNet II: Extended Theory and Practice. International Computer Science Institute, Berkeley, California. Distributed with the FrameNet data.
8. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 254-263, Honolulu, Hawaii, October. Association for Computational Linguistics.
9. Lúis von Ahn and Laura Dabbish. 2008. Designing games with a purpose. Communications of the ACM, 51:58- 67., August.
10. Lúis von Ahn, Benjamin Maurer, Colin McMillen, David Abraham, and Manuel Blum. 2008. reCAPTCHA: Human-based character recognition via web security measures. Science, 321(5895):1465-1468.