Modeling the complexity of manual annotation tasks: A grid of analysis

24th International Conference on Computational Linguistics - Proceedings of COLING 2012: Technical Papers

Volumn , Issue , 2012, Pages 895-910

  Fort, Karën ^a,b   Nazarenko, Adeline ^a   Rosset, Sophie ^c  

^a UNIVERSITÉ PARIS 13   (France)

^b LORIA   (France)

^c CNRS   (France)

Author keywords

Annotation campaign cost estimate; Annotation campaign management; Manual corpus annotation

Indexed keywords

CAMPAIGN MANAGEMENTS; CORPUS ANNOTATIONS; COST ESTIMATE; IN-DEPTH ANALYSIS; ITS APPLICATIONS; MANUAL ANNOTATION; NATURAL LANGUAGE PROCESSING; REAL-WORLD;

COMPUTATIONAL LINGUISTICS; COST ESTIMATING; NATURAL LANGUAGE PROCESSING SYSTEMS;

COST BENEFIT ANALYSIS;

EID: 84876798631     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (26)

1. Abeillé, A., Clément, L., and Toussenel, F. (2003). Building a treebank for French. In Abeillé, A., editor, Treebanks, pages 165 -187. Kluwer, Dordrecht.
2. Bayerl, P. S. and Paul, K. I. (2011). What determines inter-coder agreement in manual annotations? a meta-analytic investigation. Computational Linguistics, 37(4):699-725.
3. Bonneau-Maynard, H., Rosset, S., Ayache, C., Kuhn, A., and Mostefa, D. (2005). Semantic annotation of the French media dialog corpus. In Proceedings of the InterSpeech, Lisboa, Portugal.
4. Bontcheva, K., Cunningham, H., Roberts, I., and Tablan, V. (2010). Web-based collaborative corpus annotation: Requirements and a framework implementation. In Witte, R., Cunningham, H., Patrick, J., Beisswanger, E., Buyko, E., Hahn, U., Verspoor, K., and Coden, A. R., editors, Proceedings of the workshop on New Challenges for NLP Frameworks (NLPFrameworks 2010), Valletta, Malta. ELRA.
5. Cook, P. and Stevenson, S. (2010). Automatically identifying changes in the semantic orientation of words. In Chair, N. C. C., Choukri, K., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S., Rosner, M., and Tapias, D., editors, Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10), Valletta, Malta.
6. Dandapat, S., Biswas, P., Choudhury, M., and Bali, K. (2009). Complex linguistic annotation - no easy way out! a case from bangla and hindi POS labeling tasks. In Proceedings of the third ACL Linguistic Annotation Workshop, Singapore.
7. Erk, K., Kowalski, A., Padó, S., and Pinkal, M. (2003). Towards a resource for lexical semantics: A large German corpus with extensive semantic annotation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL 03, pages 537-544, Morristown, NJ, USA. Association for Computational Linguistics.
8. Fort, K., François, C., and Ghribi, M. (2010). Evaluer des annotations manuelles dispersées : les coefficients sont-ils suffisants pour estimer l'accord inter-annotateurs ? In Proceedings of the Traitement Automatique des Langues Naturelles (TALN), Montreal, Canada. 10 pages.
9. Fort, K. and Sagot, B. (2010). Influence of pre-annotation on POS-tagged corpus development. In Proceedings of the Fourth ACL Linguistic Annotation Workshop, pages 56-63, Uppsala, Sweden.
10. Goecke, D., Lüngen, H., Metzing, D., Stührenberg, M., and Witt, A. (2010). Different views on markup. In Witt, A., Metzing, D., and Ide, N., editors, Linguistic Modeling of Information and Markup Languages, volume 40 of Text, Speech and Language Technology, pages 1-21. Springer Netherlands.
11. Grouin, C., Rosset, S., Zweigenbaum, P., Fort, K., Galibert, O., and Quintard, L. (2011). Proposal for an extension of traditional named entities: From guidelines to evaluation, an overview. In Proceedings of the 5th Linguistic Annotation Workshop, pages 92-100, Portland, Oregon, USA. (poster).
12. Gut, U. and Bayerl, P. S. (2004). Measuring the reliability of manual annotations of speech corpora. In Proceedings of the Speech Prosody, pages 565-568, Nara, Japan.
13. Ide, N. and Romary, L. (2006). Representing linguistic corpora and their annotations. In Proceedings of the Fifth International Conference on Language Resources and Evaluation(LREC 2006), Genoa, Italy.
14. Jourde, J., Manine, A.-P., Veber, P., Fort, K., Bossy, R., Alphonse, E., and Bessières, P. (2011). BioNLP shared task 2011 - bacteria gene interactions and renaming. In Proceedings of the BioNLP Shared Task 2011 Workshop, pages 65-73, Portland, Oregon, USA. Association for Computational Linguistics.
15. Kaplan, D., Iida, R., and Tokunaga, T. (2010). Annotation process management revisited. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC 2010), pages 365 - 366.
16. Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology, second edition, chapter 11. Sage, Thousand Oaks, CA., USA.
17. Leech, G. (1993). Corpus annotation schemes. Literary and Linguistic Computing, 8(4):275- 281.
18. Leech, G. (2005). Developing Linguistic Corpora: A Guide to Good Practice, chapter Adding Linguistic Annotation, pages 17-29. Oxford: Oxbow Books.
19. Makhoul, J., Kubala, F., Schwartz, R., and Weischedel, R. (1999). Performance measures for information extraction. In Proceedings of DARPA Broadcast News Workshop, pages 249-252.
20. Marcus, M., Santorini, B., and Marcinkiewicz, M. A. (1993). Building a large annotated corpus of English : The Penn Treebank. Computational Linguistics, 19(2):313-330.
21. Ogren, P. (2006). Knowtator: A plug-in for creating training and evaluation data sets for biomedical natural language systems. In Protégé Conference, Stanford, USA.
22. Palmer, M., Gildea, D., and Kingsbury, P. (2005). The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71-106.
23. Popescu-Belis, A. (2007). Le rôle des métriques d'évaluation dans le processus de recherche en tal. T.A.L. : Traitement automatique de la langue, vol. 48, n. 1:67-91.
24. Rosset, S., Grouin, C., Fort, K., Galibert, O., Kahn, J., and Zweigenbaum, P. (2012). Structured named entities in two distinct press corpora: Contemporary broadcast news and old newspapers. In Proceedings of the 6th Linguistic Annotation Workshop (LAW VI), pages 40-48, Jeju, Republic of Korea.
25. Santorini, B. (1990). Part-of-speech tagging guidelines for the Penn Treebank Project. Technical Report MS-CIS-90-47, Department of Computer and Information Science, University of Pennsylvania, USA.
26. Tomanek, K., Hahn, U., Lohmann, S., and Ziegler, J. (2010). A cognitive cost model of annotations based on eye-tracking data. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL '10, pages 1158-1167, Stroudsburg, PA, USA. Association for Computational Linguistics.