From annotator agreement to noise models

Computational Linguistics

Volumn 35, Issue 4, 2009, Pages 495-503

  Klebanov, Beata Beigman ^a   Beigman, Eyal ^a  

^a NORTHWESTERN UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AGREEMENT COEFFICIENT; DATA SET; GOLD STANDARDS; HIGH CONFIDENCE; NOISE LEVELS; NOISE MODELS;

BENCHMARKING;

EID: 71749114162     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/coli.2009.35.4.35402     Document Type: Article

References (28)

1. Aickin, Mikel. 1990. Maximum likelihood estimation of agreement in the constant predictive probability model, and its relation to Cohen's kappa. Biometrics, 46(2):293-302.
2. Albert, Paul and Lori Dodd. 2004. A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard. Biometrics, 60(2):427-435.
3. Albert, Paul, Lisa McShane, and Joanna Shih. 2001. Latent class modeling approaches for assessing diagnostic error without a gold standard: With applications to p53 immunohistochemical assays in bladder tumors. Biometrics, 57(2):610-619.
4. Artstein, Ron and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555-596.
5. Beigman, Eyal and Beata Beigman Klebanov. 2009. Learning with annotation noise. In Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics, Singapore.
6. Beigman Klebanov, Beata, Eyal Beigman, and Daniel Diermeier. 2008. Analyzing disagreements. In COLING 2008 Workshop on Human Judgments in Computational Linguistics, pages 2-7, Manchester.
7. Blum, Avrim, Alan Frieze, Ravi Kannan, and Santosh Vempala. 1996. A polynomial-time algorithm for learning noisy linear threshold functions. In Proceedings of the 37th Annual IEEE Symposium on Foundations of Computer Science, pages 330-338, Burlington, VT.
8. Carletta, Jean. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249-254.
9. Cohen, Edith. 1997. Learning noisy perceptrons by a perceptron in polynomial time. In Proceedings of the 38th Annual Symposium on Foundations of Computer Science, pages 514-523, Miami Beach, FL.
10. Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46.
11. Dagan, Ido, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognizing textual entailment challenge. In The PASCAL Recognising Textual Entailment Challenge, Springer, Berlin, pages 177-190.
12. Di Eugenio, Barbara and Michael Glass. 2004. The kappa statistic: A second look. Computational Linguistics, 30(1):95-101.
13. Freund, Y. and R. E. Schapire. 1999. Large margin classification using the perceptron algorithm. Machine Learning, 37(3):277-296.
14. Gildea, Daniel and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245-288.
15. Girju, Roxana, Adriana Badulescu, and Dan Moldovan. 2006. Automatic discovery of part-whole relations. Computational Linguistics, 32(1):83-135.
16. Hui, Siu and Xiao Zhou. 1998. Evaluation of diagnostic tests without gold standards. Statistical Methods in Medical Research, 7(4):354-370.
17. Lapata, Maria. 2002. The disambiguation of nominalizations. Computational Linguistics, 28(3):357-388.
18. Litman, Diane, Julia Hirschberg, and Marc Swerts. 2006. Characterizing and predicting corrections in spoken dialogue systems. Computational Linguistics, 32(3):417-438.
19. Markert, Katja and Malvina Nissim. 2002. Metonymy resolution as a classification task. In Proceedings of the Empirical Methods in Natural Language Processing Conference, pages 204-213, Philadelphia, PA.
20. Palmer, Martha, Paul Kingsbury, and Daniel Gildea. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71-106.
21. Poesio, Massimo and Renata Vieira. 1998. A corpus-based investigation of definite description use. Computational Linguistics, 24(2):183-216.
22. Reidsma, Dennis and Jean Carletta. 2008. Reliability measurement without limit. Computational Linguistics, 34(3):319-326.
23. Reidsma, Dennis and Rieks op den Akker. 2008. Exploiting subjective annotations. In COLING 2008 Workshop on Human Judgments in Computational Linguistics, pages 8-16, Manchester.
24. Shen, Libin, Giorgio Satta, and Aravind Joshi. 2007. Guided learning for bidirectional sequence classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 760-767, Prague.
25. Siegel, Sidney and N. John Castellan Jr. 1988. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill, 2nd edition.
26. Snyder, Benjamin and Martha Palmer. 2004. The English all-words task. In Senseval-3: 3rd International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 41-43, Barcelona.
27. Vieira, Renata and Massimo Poesio. 2000. An empirically based system for processing definite descriptions. Computational Linguistics, 26(4):539-593.
28. Wiebe, Janyce, Teresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2):165-210.