Creating speech and language data with Amazon's Mechanical Turk

Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, Mturk 2010 at the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2010 - Proceedings

Volumn , Issue , 2010, Pages 1-12

  Callison Burch, Chris ^a   Dredze, Mark ^a  

^a Johns Hopkins University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AMAZON'S MECHANICAL TURKS; CROWDSOURCING PLATFORMS; HUMAN LANGUAGE TECHNOLOGIES;

EID: 84957855792     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (61)

1. Cem Akkaya, Alexander Conrad, Janyce Wiebe, and Rada Mihalcea. 2010. Amazon Mechanical Turk for subjectivity word sense disambiguation. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
2. Vamshi Ambati and Stephan Vogel. 2010. Can crowds build parallel corpora for machine translation systems? In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
3. Vamshi Ambati, Stephan Vogel, and Jamie Carbonell. 2010. Active learning and crowd-sourcing for machine translation. Language Resources and Evaluation (LREC).
4. Kathy Baker, Steven Bethard, Michael Bloodgood, Ralf Brown, Chris Callison-Burch, Glen Coppersmith, Bonnie Dorr, Wes Filardo, Kendall Giles, Ann Irvine, Mike Kayser, Lori Levin, Justin Martineau, Jim May-field, Scott Miller, Aaron Phillips, Andrew Philpot, Christine Piatko, Lane Schwartz, and David Zajic. 2009. Semantically-informed machine translation. Technical Report 002, Johns Hopkins Human Language Technology Center of Excellence, Summer Camp for Applied Language Exploration, Johns Hopkins University, Baltimore, MD.
5. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Association for Computational Linguistics (ACL).
6. Michael Bloodgood and Chris Callison-Burch. 2010a. Bucking the trend: Large-scale cost-focused active learning for statistical machine translation. In 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden.
7. Michael Bloodgood and Chris Callison-Burch. 2010b. Using Mechanical Turk to build machine translation evaluation sets. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
8. Cristina Butnariu, Su Nam Kim, Preslav Nakov, Diarmuid Ó Séaghdha, Stan Szpakowicz, and Tony Veale. 2009. Semeval-2010 Task 9: The interpretation of noun compounds using paraphrasing verbs and prepositions. In Workshop on Semantic Evaluations.
9. Olivia Buzek, Philip Resnik, and Ben Bederson. 2010. Error driven paraphrase annotation using Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
10. Chris Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using amazon's mechanical turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP-2009), Singapore.
11. Jonathan Chang, Jordan Boyd-Graber, Chong Wang, Sean Gerrish, and David M. Blei. 2009. Reading tea leaves: How humans interpret topic models. In Neural Information Processing Systems.
12. Jonathan Chang. 2010. Not-so-Latent Dirichlet Allocation: Collapsed Gibbs sampling using human judgments. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
13. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR, Miami Beach, Floriday.
14. Michael Denkowski and Alon Lavie. 2010. Exploring normalization techniques for human judgments of machine translation adequacy collected using Amazon Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
15. Michael Denkowski, Hassan Al-Haj, and Alon Lavie. 2010. Turker-assisted paraphrasing for English-Arabic machine translation. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
16. Mark Dredze, Partha Pratim Talukdar, and Koby Crammer. 2009. Sequence learning from data with multiple labels. In ECML/PKDD Workshop on Learning from Multi-Label Data (MLD).
17. Keelan Evanini, Derrick Higgins, and Klaus Zechner. 2010. Using Amazon Mechanical Turk for transcription of non-native speech. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
18. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books.
19. Tim Finin, William Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Touring PER, ORG and LOC on $100 a day. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
20. Qin Gao and Stephan Vogel. 2010. Semi-supervised word alignment with Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
21. Dan Gillick and Yang Liu. 2010. Non-expert evaluation of summarization systems is risky. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
22. Jonathan Gordon, Benjamin Van Durme, and Lenhart Schubert. 2010. Evaluation of commonsense knowledge with Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
23. Matthew R. Gormley, Adam Gerber, Mary Harper, and Mark Dredze. 2010. Non-expert correction of automatically generated relation annotations. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
24. Catherine Grady and Matthew Lease. 2010. Crowdsourcing document relevance assessment with Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
25. Michael Heilman and Noah A. Smith. 2010. Rating computer-generated questions with Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
26. Chiara Higgins, Elizabeth McGrath, and Laila Moretto. 2010. AMT crowdsourcing: A viable method for rapid discovery of Arabic nicknames? In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
27. Paul J. Hopper and Sandra A. Thompson. 1980. Transitivity in grammar and discourse. Language, 56:251-299.
28. Pei-Yun Hsueh, Prem Melville, and Vikas Sindhwani. 2009. Data quality from crowdsourcing: A study of annotation selection criteria. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, pages 27-35, Boulder, Colorado, June. Association for Computational Linguistics.
29. Panos Ipeirotis. 2010. New demographics of Mechanical Turk. http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk. html.
30. Ann Irvine and Alexandre Klementiev. 2010. Using Mechanical Turk to annotate lexicons for less commonly used languages. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
31. Mukund Jha, Jacob Andreas, Kapil Thadani, Sara Rosenthal, and Kathleen McKeown. 2010. Corpus creation for new genres: A crowdsourced approach to PP attachment. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
32. Michael Kaisser and John Lowe. 2008. Creating a research collection of question answer sentence pairs with Amazons Mechanical Turk. In Proceedings of the Sixth International Language Resources and Evaluation (LREC'08), Marrakech, Morocco.
33. Stephen Kunath and Steven Weinberger. 2010. The wisdom of the crowd's ear: Speech accent rating and annotation with Amazon Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
34. Ian Lane, Matthias Eck, Kay Rottmann, and Alex Waibel. 2010. Tools for collecting speech corpora via Mechanical-Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
35. Nolan Lawson, Kevin Eustice, Mike Perkowitz, and Meliha Yetisgen Yildiz. 2010. Annotating large email datasets for named entity recognition with Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
36. Greg Little, Lydia B. Chilton, Rob Miller, and Max Goldman. 2009. Turkit: Tools for iterative tasks on mechanical turk. In Proceedings of the Workshop on Human Computation at the International Conference on Knowledge Discovery and Data Mining (KDD-HCOMP'09), Paris.
37. Nitin Madnani and Jordan Boyd-Graber. 2010. Measuring transitivity using untrained annotators. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
38. Mitch Marcus., Beatrice Santorini, and Mary Ann Marcinkiewicz. 1994. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313-330.
39. Matthew Marge, Satanjeev Banerjee, and Alexander Rudnicky. 2010a. Using the Amazon Mechanical Turk for transcription of spoken language. ICASSP, March.
40. Matthew Marge, Satanjeev Banerjee, and Alexander Rudnicky. 2010b. Using the Amazon Mechanical Turk to transcribe and annotate meeting speech for extractive summarization. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
41. Winter Mason and Duncan J. Watts. 2009. Financial incentives and the “performance of crowds”. In Proceedings of the Workshop on Human Computation at the International Conference on Knowledge Discovery and Data Mining (KDD-HCOMP'09), Paris.
42. Ian McGraw, Chia ying Lee, Lee Hetherington, and Jim Glass. 2010. Collecting voices from the crowd. LREC, May.
43. Bart Mellebeek, Francesc Benavent, Jens Grivolla, Joan Codina, Marta R. Costa-Jussà, and Rafael Banchs. 2010. Opinion mining of spanish customer comments with non-expert annotations on Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
44. Robert Munro, Steven Bethard, Victor Kuperman, Vicky Tzuyin Lai, Robin Melnick, Christopher Potts, Tyler Schnoebelen, and Harry Tily. 2010. Crowdsourcing and language studies: the new generation of linguistic data. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
45. Matteo Negri and Yashar Mehdad. 2010. Creating a bi-lingual entailment corpus through translations with Mechanical Turk: $100 for a 10 days rush. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
46. Scott Novotney and Chris Callison-Burch. 2010a. Cheap, fast and good enough: Automatic speech recognition with non-expert transcription. NAACL, June.
47. Scott Novotney and Chris Callison-Burch. 2010b. Crowdsourced accessibility: Elicitation of Wikipedia articles. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
48. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In Empirical Methods in Natural Language Processing (EMNLP).
49. Gabriel Parent and Maxine Eskenazi. 2010. Clustering dictionary definitions using Amazon Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
50. Dragomir R. Radev, Pradeep Muthukrishnan, and Vahed Qazvinian. 2009. The acl anthology network. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries, pages 54-61, Suntec City, Singapore, August. Association for Computational Linguistics.
51. Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. 2010. Collecting image annotations using Amazon's Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
52. Joel Rose. 2010. Some turn to 'Mechanical' job search. http://marketplace.publicradio.org/display/web/2009/06/30/pm_turking/. Marketplace public radio. Air date: June 30, 2009.
53. Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In CoNLL-2003.
54. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP-2008), Honolulu, Hawaii.
55. Audrey Tong, Jerome Ajot, Mark Przybocki, and Stephanie Strassel. 2010. Document image collection using Amazon's Mechanical Turk. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
56. Luis von Ahn and Laura Dabbish. 2008. General techniques for designing games with a purpose. Communications of the ACM.
57. Rui Wang and Chris Callison-Burch. 2010. Cheap facts and counter-facts. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
58. Tae Yano, Philip Resnik, and Noah A Smith. 2010. Shedding (a thousand points of) light on biased language. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
59. Meliha Yetisgen-Yildiz, Imre Solti, Scott Halgrim, and Fei Xia. 2010. Preliminary experiments with Amazon's Mechanical Turk for annotating medical named entities. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.
60. Omar F. Zaidan and Chris Callison-Burch. 2009. Feasibility of human-in-the-loop minimum error rate training. In Proceedings of EMNLP 2009, pages 52-61, August.
61. Omar Zaidan and Juri Ganitkevitch. 2010. An enriched MT grammar for under $100. In NAACL Workshop on Creating Speech and Language Data With Amazon's Mechanical Turk.