Analyzing the impact of prevalence on the evaluation of a manual annotation campaign

Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012

Volumn , Issue , 2012, Pages 1474-1480

  Fort, Karen ^a,b   Francois, Claire ^a   Galibert, Olivier ^c   Ghribi, Maha ^a  

^a INIST CNRS   (France)

^b UNIVERSITÉ PARIS 13   (France)

^c LABORATOIRE NATIONAL DE MÉTROLOGIE ET D ESSAIS LNE   (France)

Author keywords

Inter annotator agreement; Manual annotation evaluation; Prevalence

Indexed keywords

AGREEMENT COEFFICIENT; COMPUTING DISTANCES; CONTINGENCY TABLE; F MEASURE; MANUAL ANNOTATION; PREVALENCE; WEIGHTED COEFFICIENTS;

EID: 84941310079     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. Beatrice Alex, Malvina Nissim, and Claire Grover. 2006. The Impact of Annotation on the Performance of Protein Tagging in Biomedical Text. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC), pages 595-600, Genoa, Italy, 24-26 May.
2. Beatrice Alex, Claire Grover, Rongzhou Shen, and Mijail Kabadjov. 2010. Agile corpus annotation in practice: An overview of manual and automatic annotation of cvs. In Proceedings of the Fourth Linguistic Annotation Workshop (LAW), pages 29-37, Uppsala, Sweden. Association for Computational Linguistics.
3. Erick Alphonse, Sophie Aubin, Philippe Bessières, Gilles Bisson, Thierry Hamon, Sandrine Laguarigue, Adeline Nazarenko, Alain-Pierre Manine, Claire Nédellec, Mohamed Ould Abdel Vetah, Thierry Poibeau, and Davy Weissenbacher. 2004. Event-based Information Extraction for the Biomedical the CADERIGE Project. In Proceedings of the JNLPBA COLING 2004 Workshop, Geneva, Switzerland.
4. Ron Artstein and Massimo Poesio. 2008. Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4):555-596.
5. Edward M. Bennett, R. Alpert, and A. C. Goldstein. 1954. Communications through Limited Questioning. Public Opinion Quarterly, 18(3):303-308.
6. Hélène Bonneau-Maynard, Sophie Rosset, Christelle Ayache, Anne Kuhn, and Djamel Mostefa. 2005. Semantic Annotation of the French Media Dialog Corpus. In Proceedings of the InterSpeech, Lisboa, Portugal.
7. Jean Carletta. 1996. Assessing Agreement on Classification Tasks: the Kappa Statistic. Computational Linguistics, 22:249-254.
8. Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37-46.
9. Jacob Cohen. 1968. Weighted Kappa: Nominal Scale Agreement with Provision for Scaled Disagreement or Partial Credit. Psychological Bulletin, 70(4):213-220.
10. Barbara Di Eugenio and Michael Glass. 2004. The Kappa Statistic: a Second Look. Computational Linguistics, 30(1):95-101.
11. R. H. Finn. 1970. A Note on Estimating the Reliability of Categorical Data. Educational and Psychological Measurement, 30:71-76.
12. Olivier Galibert, Ludovic Quintard, Sophie Rosset, Pierre Zweigenbaum, Claire Nédellec, Sophie Aubin, Laurent Gillard, Jean-Pierre Raysz, Delphine Pois, Xavier Tannier, Louise Deléger, and Dominique Laurent. 2010. Named and Specific Entity Detection in Varied Data: the Quaero Named Entity Baseline Evaluation. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC 2010).
13. George Hripcsak and Daniel F. Heitjan. 2002. Measuring Agreement in Medical Informatics Reliability Studies. Journal of Biomedical Informatics, 35(2):99-110.
14. George Hripcsak and Adam S Rothschild. 2005. Agreement, the f measure, and reliability in information retrieval. Journal of the American Medical Informatics Association (JAMIA), 12(3):2968.
15. Klaus Krippendorff, 1980. Content Analysis: An Introduction to Its Methodology, chapter 12. Sage, Beverly Hills, CA., USA.
16. Klaus Krippendorff, 2004. Content Analysis: An Introduction to Its Methodology, second edition, chapter 11. Sage, Thousand Oaks, CA., USA.
17. Marion Laignelet and François Rioult. 2009. Repérer automatiquement les segments obsolescents à l'aide d'indices sémantiques et discursifs. In Proceedings of the Traitement Automatique des Langues Naturelles (TALN 2009), Senlis, France.
18. John Makhoul, Francis Kubala, Richard Schwartz, and Ralph Weischedel. 1999. Performance measures for information extraction. In In Proceedings of DARPA Broadcast News Workshop, pages 249-252.
19. Rebecca Passonneau. 2006. Measuring Agreement on Set-Valued Items (MASI) for Semantic and Pragmatic Annotation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC 2006).
20. Denis Reidsma and Jean Carletta. 2008. Reliability Measurement Without Limits. Computational Linguistics, 34(3):319-326.
21. William A Scott. 1955. Reliability of Content Analysis: The Case of Nominal Scale Coding. Public Opinion Quaterly, 19(3):321-325.
22. Sidney Siegel and N. John Castellan. 1988. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill, New York, USA, 2nd edition.