Using Amazon Mechanical Turk for transcription of non-native speech

Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010 - Proceedings

Volumn , Issue , 2010, Pages 53-56

  Evanini, Keelan ^a   Higgins, Derrick ^a   Zechner, Klaus ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AMAZON'S MECHANICAL TURKS; GOLD STANDARDS; NON-NATIVE SPEECH; WORKERS';

EID: 84877321285     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (8)

1. Chris Callison-Burch. 2009. Fast, cheap and creative: Evaluating translation quality using Amazon's Mechanical Turk. In Proc. EMNLP.
2. Neeraj Deshmukh, Richard Jennings Duncan, Aravind Ganapathiraju, and Joseph Picone. 1996. Benchmarking human performance for continuous speech recognition. In Proc. ICSLP.
3. Jonathan G. Fiscus. 1997. A post-processing system to yield word error rates: Recognizer Ooutput Voting Error Reduction (ROVER). In Proc. ASRU.
4. Meghan Lammie Glenn and Stephanie Strassel. 2008. Shared linguistic resources for the meeting domain. In Lecture Notes in Computer Science, volume 4625, pages 401-413. Springer.
5. Matthew Marge, Satanjeev Banerjee, and Alexander I. Rudnicky. to appear. Using the Amazon Mechanical Turk for transcription of spoken language. In Proc. ICASSP.
6. Scott Novotney and Chris Callison-Burch. 2010. Cheap, fast, and good enough: Automatic speech recognition with non-expert transcription. In Proc. NAACL.
7. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast - But is it good? Evaluating non-expert annotations for natural language tasks. In Proc. EMNLP.
8. Klaus Zechner. 2009. What did they actually say? Agreement and disagreement among transcribers of non-native spontaneous speech responses in an English proficiency test. In Proc. ISCA-SLaTE.