Inter-annotator agreement on a multilingual semantic annotation task

Proceedings of the 5th International Conference on Language Resources and Evaluation, LREC 2006

Volumn , Issue , 2006, Pages 1951-1956

  Passonneau, Rebecca ^a   Habash, Nizar ^a   Rambow, Owen ^a  

^a Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALUMINUM; ONTOLOGY; SEMANTICS; SYNTACTICS;

LEARNING EFFECTS; LEXICAL ITEMS; MACHINE TRANSLATIONS; MULTILINGUAL TEXTS; NEWS ARTICLES; PARTS OF SPEECH; SEMANTIC ANNOTATIONS; WORDNET;

TRANSLATION (LANGUAGES);

EID: 57349129738     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (20)

1. 3=Alpha (or Beta). Technical Report NLE Technote 2005-01, University of Essex. Essex.
2. Bruce, R. and J. Wiebe. (1998). Word-sense distinguishability an inter-coder agreement. In Proceedings of Empirical Methods in Natural Language Processing.
3. Di Eugenio, B. and M. Glass. (2004). The kappa statistic: A second look. Computational Linguistics, 30(1):95-101.
4. Di Eugenio, B. (2000). On the usage of Kappa to evaluate agreement on coding tasks, Proceedings of the Second International Conference on Language Resources and Evaluation (LREC). Athens, Greece.
5. Dorr, B.; Green, R.; Levin, L.; Rambow, O.; Farwell, D.; Habash, N.; Helmreich, S.; Hovy, E.; Miller, K.J.; Mitamura, T.; Reeder, F.; Siddharthan, A. (2004). Semantic annotation and lexico-syntactic paraphrase. In Proceedings of the Workshop on Building Lexical Resources from Semantically Annotated Corpora, (LREC) Portugal.
6. Farwell, D.; Helmreich, S.; Dorr, B. J.; Habash, N.; Reeder, F.; Miller, K.; Levin, L.; Mitamura, T.; Hovy, E.; Rambow, O.; Siddharthan, A. (2004). Interlingual annotation of multilingual text corpora. In Proceedings of the North American Chapter of the Association for Computational Linguistics Workshop on Frontiers in Corpus Annotation, Boston, MA, pp. 55-62, 2004.
7. Fellbaum, C. (1998). WordNet: An Electronic Lexical Database. MIT Press.
8. Jaccard, P. (1908). Nouvelles recherches sur la distribution florale. Bulletin de la Societe Vaudoise des Sciences Naturelles 44:223-270.
9. Krippendorff, Klaus. (1980) Content Analysis: An Introduction to its Methodology. Sage Publications, Beverly Hills, CA.
10. Mitamura, T.; Miller, K.; Dorr, B. J.; Farwell, D.; Habash, N.; Levin, L.; Helmreich, S.; Hovy, E.; Levin, L.; Rambow, O.; Reeder, F.; Siddharthan, A. (2004). Semantic annotation of multilingual text corpora. In Proceedings of the Workshop on Beyond Named Entity Recognition: Semantic Labeling for NLP Tasks, LREC, Portugal.
11. Nirenburg, S., S. Beale, K. Mahesh, B. Onyshkevych, V. Raskin, E. Viegas, Y. Wilks and R. Zajac. (1996). Lexicons in the Mikrokosmos project. Proceedings of the Society for Artificial Intelligence and Simulated Behavior Workshop on Multilinguality in the Lexicon, Brighton, UK.
12. Passonneau, R.; Nenkova, A.; McKeown, K.; Sigelman, S. (2005). Applying the pyramid method in DUC 2005. In Proceedings of the Workshop of the Document Understanding Conference. Vancouver, B.C.
13. Passonneau, R. (2006). Measuring agreement on setvalued items (MASI) for semantic and pragmatic annotation. In Proceedings of the International Conference on Language Resources and Evaluation (LREC). Genoa, Italy.
14. Passonneau, R. (2004). Computing reliability for coreference annotation. Proceedings of the International Conference on Language Resources and Evaluation (LREC). Portugal.
15. Philpot, A.; Fleischman, M.; Hovy, E.H. (2003). Semi-automatic construction of a general purpose ontology. Proceedings of the International Lisp Conference. New York, NY. Invited.
16. Rosenberg, A. and Binkowski, E. (2004). Augmenting the kappa statistic to determine inter-annotator reliability for multiply labeled data points. In Proceedings of the Human Language Technology Conference and Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL).
17. Scott, W. A. (1955). Reliability of content analysis: the case of nominal scale coding. Public Opinion Quarterly 19:321-325.
18. nd edition. McGraw-Hill, New York.
19. Teufel, S. and H. van Halteren. (2004). Evaluating information content by factoid analysis: human annotation and stability. Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 419-426.
20. Uebersax, J. (1988). Validity inferences from interobserver agreement. Psychological Bulletin, 104(3):405-416.