Crowdsourcing and language studies: The new generation of linguistic data

Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010 - Proceedings

Volumn , Issue , 2010, Pages 122-130

  Munro, Robert ^a   Bethard, Steven ^b   Kuperman, Victor ^a   Lai, Vicky Tzuyin ^c   Melnick, Robin ^a   Potts, Christopher ^a   Schnoebelen, Tyler ^a   Tily, Harry ^a  

^a STANFORD UNIVERSITY   (United States)

^b Stanford University   (United States)

^c University of Colorado   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

SEMANTICS;

CONTROLLED LABORATORIES; CURRENT PROJECTS; DATA QUALITY; DATAPOINTS; LABORATORY EXPERIMENTS; LANGUAGE STUDY; LINGUISTIC DATA;

CROWDSOURCING;

EID: 85121234075     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. Joan Bresnan. 2007. Is syntactic knowledge probabilistic? Experiments with the English dative alternation. In Sam Featherston and Wolfgang Sternefeld, editors, Roots: Linguistics in search of its evidential base, pages 75-96. Mouton de Gruyter, Berlin.
2. Chris Callison-Burch. 2009. Fast, cheap, and creative: evaluating translation quality using Amazon's Mechanical Turk. In EMNLP'09: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 286-295.
3. Carol Chomsky. 1969. The Acquisition of Syntax in Children from 5 to 10. MIT Press, Cambridge, MA.
4. Thierry Dutoit, Vincent Pagel, Nicolas Pierret, Franois Bataille, and Olivier van der Vrecken. 1996. The MBROLA project: Towards a set of high quality speech synthesizers free of use for non commercial purposes. In Fourth International Conference on Spoken Language Processing, pages 75-96.
5. Michael Frank, Harry Tily, Inbal Arnon, and Sharon Goldwater. submitted. Beyond transitional probabilities: Human learners impose a parsimony bias in statistical word segmentation.
6. Michael Frank, Sharon Goldwater, Thomas Griffiths, and Joshua Tenenbaum. under review. Modeling human performance in statistical word segmentation.
7. Pei-Yun Hsueh, Prem Melville, and Vikas Sindhwani. 2009. Data quality from crowdsourcing: a study of annotation selection criteria. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, pages 27-35.
8. Florian Jaeger. 2006. Redundancy and syntactic reduction in spontaneous speech. Ph.D. thesis, Stanford University, Stanford, CA.
9. Florian Jaeger. in press. Redundancy and reduction: Speakers manage syntactic information density. Cognitive Psychology.
10. Vicky Tzuyin Lai, Tim Curran, and Lise Menn. 2009. Comprehending conventional and novel metaphors: An ERP study. Brain Research, 1284:145-155, August.
11. Richard Landis and Gary Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1).
12. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. Annotating noun argument structure for NomBank. In Proceedings of LREC-2004.
13. Prashant Parikh. 2010. Language and Equilibrium. MIT Press, Cambridge, MA.
14. Stefanie Regel, Seana Coulson, and Thomas C. Gunter. 2010. The communicative style of a speaker can affect language comprehension? ERP evidence from the comprehension of irony. Brain Research, 1311:121-135.
15. Erik D. Reichle, Alexander Pollatsek, Donald L. Fisher, and Keith Rayner. 1998. Toward a model of eye movement control in reading. Psychological Review, 105:125-157.
16. Tom Roeper. 2007. The Prism of Grammar: How Child Language Illuminates Humanism. MIT Press, Cambridge, MA.
17. Jenny R. Saffran, Richard N. Aslin, and Elissa L. Newport. 1996. Word segmentation: The role of distributional cues. Journal of memory and language, 35:606-621.
18. Tyler Schnoebelen and Victor Kuperman. submitted. Using Amazon Mechanical Turk for linguistic research: Fast, cheap, easy, and reliable.
19. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew T. Ng. 2008. Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks. In EMNLP'08: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254-263.
20. Wilson Taylor. 1953. Cloze procedure: A new tool for measuring readability. Journalism Quarterly, 30:415-433.
21. Tom Wasow. 2008. Gradient data and gradient grammars. In Proceedings of the 43rd Annual Meeting of the Chicago Linguistics Society, pages 255-271.