Amazonmechanical turk: Gold mine or coalmine?

Computational Linguistics

Volumn 37, Issue 2, 2011, Pages 413-420

  Fort, Kar̈en ^a   Adda, Gilles ^b   Cohen, K Bretonnel ^c  

^a INIST CNRS/LIPN   (France)

^b UFR 919 Laboratoire d'Informatique Pour la Mécanique et les Sciences de l'Ingénieur   (France)

^c UNIVERSITY OF COLORADO SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 79958245410     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/COLI_a_00057     Document Type: Article

References (19)

1. Adda, Gilles and Joseph Mariani. 2010. Language resources and Amazon Mechanical Turk: Legal, ethical and other issues. In LISLR2010, ''Legal Issues for Sharing Language Resources workshop,'' LREC2010, Malta.
2. Akerlof, George A. 1970. The market for 'lemons': Quality uncertainty and the market mechanism. Quarterly Journal of Economics, 84(3):488-500.
3. Bhardwaj, Vikas, Rebecca Passonneau, Ansaf Salleb-Aouissi, and Nancy Ide. 2010. Anveshan: A tool for analysis of multiple annotators' labeling behavior. In Proceedings of The Fourth Linguistic Annotation Workshop (LAW IV), pages 47-55, Uppsala.
4. Biadsy, Fadi, Julia Hirschberg, and Elena Filatova. 2008. An unsupervised approach to biography production usingWikipedia. In Proceedings of ACL 2008, pages 807-815, Columbus.
5. Callison-Burch, Chris and Mark Dredze. 2010. Creating speech and language data with Amazon's Mechanical Turk. In CSLDAMT'10: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 1-12, Morristown, NJ.
6. Chamberlain, J., M. Poesio, and U. Kruschwitz. 2008. Phrase detectives: aWeb-based collaborative annotation game. In Proceedings of the International Conference on Semantic Systems (I-Semantics'08), pages 42-49, Graz.
7. Deneme. 2009. How many Turkers are there? Available at http://groups.csail.mit.edu/uid/deneme/?p=502. Accessed December 2009.
8. Gillick, Dan and Yang Liu. 2010. Non-expert evaluation of summarization systems is risky. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, CSLDAMT'10, pages 148-151, Stroudsburg, PA.
9. Ipeirotis, Panos. 2010a. Analyzing the Amazon Mechanical Turk marketplace. CeDER Working Papers No. CeDER-10-04. Available at http://hdl.handle.net/2451/29801.
10. Ipeirotis, Panos. 2010b. Demographics of Mechanical Turk. CeDER Working Papers No. CeDER-10-01. Available at http://hdl.handle.net/2451/29585.
11. Lafourcade, Mathieu. 2007. Making people play for lexical acquisition. In Proceedings of SNLP 2007, 7th Symposium on Natural Language Processing, Pattaya.
12. Marge, Matthew, Satanjeev Banerjee, and Alexander I. Rudnicky. 2010. Using the Amazon Mechanical Turk for transcription of spoken language. In IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pages 5270-5273, Dallas, TX.
13. Novotney, Scott and Chris Callison-Burch. 2010. Cheap, fast and good enough: Automatic speech recognition with non-expert transcription. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, HLT '10, pages 207-215, Morristown, NJ.
14. Ross, Joel, Lilly Irani, M. Six Silberman, Andrew Zaldivar, and Bill Tomlinson. 2010. Who are the crowdworkers? Shifting demographics in Mechanical Turk. In Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA '10, pages 2863-2872, New York, NY.
15. Ross, Joel, Andrew Zaldivar, Lilly Irani, and Bill Tomlinson. 2009. Who are the Turkers?Worker demographics in Amazon Mechanical Turk. Social Code Report 2009-01. Available at http://www.ics.uci.edu/jwross/pubs/SocialCode-2009-01.pdf.
16. Snow, Rion, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast-but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP 2008, pages 254-263, Waikiki.
17. TheQuill. 2010. Making money on-line, part 2: Microworking. Available at http://thequill.org/personalfinance/9-making-money-online-part-2-microworking-.html. Accessed 21 June 2010.
18. Tratz, Stephen and Eduard Hovy. 2010. A taxonomy, dataset, and classifier for automatic noun compound interpretation. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 678-687, Uppsala.
19. Wais, Paul, Shivaram Lingamneni, Duncan Cook, Jason Fennell, Benjamin Goldenberg, Daniel Lubarov, David Marin, and Hari Simons. 2010. Towards building a high-quality workforce with Mechanical Turk. In Proceedings of Computational Social Science and the Wisdom of Crowds (NIPS), pages 1-5, Whister.