Corpus annotation through crowdsourcing: Towards best practice guidelines

Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014

Volumn , Issue , 2014, Pages 859-866

  Sabou, Marta ^a   Bontcheva, Kalina ^b   Derczynski, Leon ^b   Scharl, Arno ^a  

^a MODUL UNIVERSITY VIENNA   (Austria)

^b UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Corpus Annotation; Crowdsourcing; Guidelines; Human Computation; Survey

Indexed keywords

MERGERS AND ACQUISITIONS; SURVEYING;

BEST PRACTICE GUIDELINES; BEST PRACTICES; COLLABORATIVE APPROACH; CORPUS ANNOTATIONS; GAMES WITH A PURPOSE; GUIDELINES; HUMAN COMPUTATION; LINGUISTIC RESOURCES;

CROWDSOURCING;

EID: 84961700014     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (57)

1. Abekawa, T., Utiyama, M., Sumita, E., and Kageura, K. (2010). Community-based Construction of Draft and Final Translation Corpus through a Translation Hosting Site Minna no Hon'yaku (MNH). In Proc. LREC.
2. ACE, (2004). Annotation Guidelines for Event Detection and Characterization (EDC), Feb. Available at http://www.ldc.upenn.edu/Projects/ACE/.
3. Aker, A., El-Haj, M., Albakour, M.-D., and Kruschwitz, U. (2012). Assessing crowdsourcing quality through objective tasks. In Proc. LREC, pages 1456-1461.
4. Attardi, G. (2010). Phratris - A Phrase Annotation Game. In INSEMTIVES Game Idea Challenge.
5. Behrend, T., Sharek, D., Meade, A., and Wiebe, E. (2011). The viability of crowdsourcing for survey research. Behav. Res., 43(3).
6. Biewald, L. (2012). Massive multiplayer human computation for fun, money, and survival. In Current Trends in Web Engineering, pages 171-176. Springer.
7. Bontcheva, K., Derczynski, L., and Roberts, I. (2014a). Crowdsourcing named entity recognition and entity linking corpora. In Handbook of Linguistic Annotation. Springer.
8. Bontcheva, K., Roberts, I., and Derczynski, L. (2014b). The GATE Crowdsourcing Plugin: Crowdsourcing Annotated Corpora Made Easy. In Proc. EACL.
9. Callison-Burch, C. and Dredze, M., editors. (2010). Proc. of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk.
10. Chamberlain, J., Poesio, M., and Kruschwizt, U. (2009). A new life for a dead parrot: Incentive structures in the Phrase Detectives game. In Proc. of the Webcentives Workshop.
11. Chernova, S., Orkin, J., and Breazeal, C. (2010). Crowdsourcing HRI through Online Multiplayer Games. In Dialog with Robots: Papers from the AAAI Fall Symposium (FS-10-05).
12. Chklovski, T. and Mihalcea, R. (2002). Building a Sense Tagged Corpus with Open Mind Word Expert. In Proc. of the ACL Workshop on Word Sense Disambiguation: Recent Successes and Future Directions.
13. Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popovic, Z., and players, F. (2010). Predicting protein structures with a multiplayer online game. Nature, 466(7307).
14. Cunningham, H., Tablan, V., Roberts, A., and Bontcheva, K. (2013). Getting more out of biomedical documents with GATE's full lifecycle open source text analytics. PLoS computational biology, 9(2):e1002854.
15. Doan, A., Ramakrishnan, R., and Halevy, A. Y. (2011). Crowdsourcing Systems on the World-Wide Web. Commun. ACM, 54(4), April.
16. El-Haj, M., Kruschwitz, U., and Fox, C. (2010). Using Mechanical Turk to Create a Corpus of Arabic Summaries. In Proc. LREC.
17. Feng, D., Besana, S., and Zajac, R. (2009). Acquiring High Quality Non-Expert Knowledge from On-Demand Workforce. In Proc. of The People's Web Meets NLP: Collaboratively Constructed Semantic Resources.
18. Finin, T., Murnane, W., Karandikar, A., Keller, N., Martineau, J., and Dredze, M. (2010). Annotating Named Entities in Twitter Data with Crowdsourcing. In Callison-Burch and Dredze (Callison-Burch and Dredze, 2010).
19. Fort, K. and Sagot, B. (2010). Influence of Pre-annotation on POS-tagged Corpus Development. In Proc. of the Fourth Linguistic Annotation Workshop.
20. Fort, K., Adda, G., and Cohen, K. (2011). Amazon Mechanical Turk: Gold Mine or Coal Mine? Computational Linguistics, 37(2):413-420.
21. Freitas, J., Calado, A., Braga, D., Silva, P., and Dias, M. (2010). Crowdsourcing platform for large-scale speech data collection. Proceedings of FALA, Vigo.
22. Gruenstein, E., Mcgraw, I., and Sutherl, A. (2009). A Self-Transcribing Speech Corpus: Collecting Continuous Speech with an Online Educational Game. In Proc. of The Speech and Language Technology in Education (SLaTE) Workshop.
23. Heilman, M. and Smith, N. A. (2010). Rating ComputerGenerated Questions with Mechanical Turk. In Callison-Burch and Dredze (Callison-Burch and Dredze, 2010).
24. Hong, J. and Baker, C. F. (2011). How Good is the Crowd at "real" WSD? In Proc. of the 5th Linguistic Annotation Workshop.
25. Hovy, E., and Marcus., M. P., Palmer, M., Ramshaw, L. A., and Weischedel, R. M. (2006). OntoNotes: The 90% Solution. In Proc. NAACL.
26. Hovy, D., Berg-Kirkpatrick, T., Vaswani, A., and Hovy, E. (2013). Learning Whom to trust with MACE. In Proc. of NAACL-HLT, pages 1120-1130.
27. Hovy, E. (2010). Annotation. In Tutorial Abstracts of ACL.
28. Irvine, A. and Klementiev, A. (2010). Using Mechanical Turk to Annotate Lexicons for Less Commonly Used Languages. In Callison-Burch and Dredze (Callison-Burch and Dredze, 2010).
29. Jha, M., Andreas, J., Thadani, K., Rosenthal, S., and McKeown, K. (2010). Corpus Creation for New Genres: A Crowdsourced Approach to PP Attachment. In Callison-Burch and Dredze (Callison-Burch and Dredze, 2010).
30. Kawrykow, A., Roumanis, G., Kam, A., Kwak, D., Leung, C., Wu, C., Zarour, E., and players, P. (2012). Phylo: A Citizen Science Approach for Improving Multiple Sequence Alignment. PLoS ONE, 7(3):e31362.
31. Khanna, S., Ratan, A., Davis, J., and Thies, W. (2010). Evaluating and improving the usability of Mechanical Turk for low-income workers in India. In Proceedings of the first ACM symposium on Computing for Development. ACM.
32. Kittur, A., and Chi., E. H., and Suh, B. (2008). Crowdsourcing User Studies with Mechanical Turk. In Proc. of the 26th Conference on Human Factors in Computing Systems.
33. Laws, F., Scheible, C., and Schütze, H. (2011). Active Learning with Amazon Mechanical Turk. In Proc. EMNLP.
34. Lawson, N., Eustice, K., Perkowitz, M., and Yetisgen-Yildiz, M. (2010). Annotating Large Email Datasets for Named Entity Recognition with Mechanical Turk. In Callison-Burch and Dredze (Callison-Burch and Dredze, 2010).
35. Mason, W. and Watts, D. J. (2010). Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter, 11(2):100-108.
36. McCreadie, R., Macdonald, C., and Ounis, I. (2012). Identifying Top News Using Crowdsourcing. Information Retrieval. 10.1007/s10791-012-9186-z.
37. McGraw, I., Gruenstein, A., and Sutherland, A. (2009). A Self-Labeling Speech Corpus: Collecting Spoken Words with an Online Educational Game. In Proc. of INTERSPEECH.
38. Mellebeek, B., Benavent, F., Grivolla, J., Codina, J., Costajussà, M. R., and Banchs, R. (2010). Opinion Mining of Spanish Customer Comments with Non-Expert Annotations on Mechanical Turk. In Callison-Burch and Dredze (Callison-Burch and Dredze, 2010).
39. Munro, R., Bethard, S., Kuperman, V., Lai, V. T., Melnick, R., Potts, C., Schnoebelen, T., and Tily, H. (2010). Crowdsourcing and Language Studies: The New Generation of Linguistic Data. In Callison-Burch and Dredze (Callison-Burch and Dredze, 2010).
40. Negri, M. and Mehdad, Y. (2010). Creating a Bi-lingual Entailment Corpus through Translations with Mechanical Turk: 100 for a 10-day Rush. In Callison-Burch and Dredze (Callison-Burch and Dredze, 2010).
41. Negri, M., Bentivogli, L., Mehdad, Y., Giampiccolo, D., and Marchetti, A. (2011). Divide and Conquer: Crowd-sourcing the Creation of Cross-Lingual Textual Entailment Corpora. In Proc. EMNLP.
42. Parent, G. and Eskenazi, M. (2010). Clustering Dictionary Definitions Using Amazon Mechanical Turk. In Callison-Burch and Dredze (Callison-Burch and Dredze, 2010).
43. Poesio, M., Kruschwitz, U., Chamberlain, J., Robaldo, L., and Ducceschi, L. (2012). Phrase Detectives: Utilizing Collective Intelligence for Internet-Scale Language Resource Creation. Transactions on Interactive Intelligent Systems.
44. Poesio, M., Chamberlain, J., and Kruschwitz, U. (2014). Crowdsourcing. In Handbook of Linguistic Annotation. Springer.
45. Rafelsberger, W. and Scharl, A. (2009). Games with a Purpose for Social Networking Platforms. In Proc. ACM conference on Hypertext and Hypermedia.
46. Rosenthal, S., Lipovsky, W., McKeown, K., Thadani, K., and Andreas, J. (2010). Towards Semi-Automated Annotation for Prepositional Phrase Attachment. In Proc. LREC.
47. Sabou, M., Bontcheva, K., Scharl, A., and Föls, M. (2013a). Games with a Purpose or Mechanised Labour? A Comparative Study. In Proc. International Conference on Knowledge Management and Knowledge Technologies.
48. Sabou, M., Scharl, A., and Föls, M. (2013b). Crowd-sourced Knowledge Acquisition: Towards Hybrid-Genre Workflows. International Journal on Semantic Web and Information Systems, 9(3).
49. Sayeed, A. B., Rusk, B., Petrov, M., Nguyen, H. C., and Meyer., T. J., and Weinberg, A. (2011). Crowdsourcing syntactic relatedness judgements for opinion mining in the study of information technology adoption. In Proc. of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (LaT-eCH '11).
50. Scharl, A., Sabou, M., Gindl, S., Rafelsberger, W., and Weichselbraun, A. (2012). Leveraging the wisdom of the crowds for the acquisition of multilingual language resources. In Proc. LREC.
51. Snow, R., O'Connor, B., Jurafsky, D., and Ng, A. Y. (2008). Cheap and Fast - but is it Good?: Evaluating Non-Expert Annotations for Natural Language Tasks. In Proc. EMNLP.
52. Vertanen, K. and Kristensson, P. O. (2011). The Imagination of Crowds: Conversational AAC Language Modeling using Crowdsourcing and Large Data Sources. In Proc. EMNLP.
53. Voyer, R., Nygaard, V., Fitzgerald, W., and Copperman, H. (2010). A Hybrid Model for Annotating Named Entity Training Corpora. In Proc. of the Fourth Linguistic Annotation Workshop (LAW IV '10).
54. Wang, A., Hoang, C., and Kan, M. Y. (2012). Perspectives on Crowdsourcing Annotations for Natural Language Processing. Language Resources and Evaluation.
55. Yano, T., Resnik, P., and Smith, N. A. (2010). Shedding (a Thousand Points of) Light on Biased Language. In Callison-Burch and Dredze (Callison-Burch and Dredze, 2010).
56. Yetisgen-Yildiz, M., Solti, I., Xia, F., and Halgrim, S. R. (2010). Preliminary Experience with Amazon's Mechanical Turk for Annotating Medical Named Entities. In Callison-Burch and Dredze (Callison-Burch and Dredze, 2010).
57. Zaidan, O. F. and Callison-Burch, C. (2011). Crowd-sourcing Translation: Professional Quality from NonProfessionals. In Proc. ACL.