Reliability measurement without limits

Computational Linguistics

Volumn 34, Issue 3, 2008, Pages 319-326

  Reidsma, Dennis ^a   Carletta, Jean ^b  

^a UNIVERSITY OF TWENTE   (Netherlands)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

LINGUISTICS; MACHINE LEARNING;

FIT FOR PURPOSE; MACHINE LEARNERS; RANDOM NOISE; REAL PATTERNS; RELIABILITY MEASURE; RELIABILITY MEASUREMENTS;

RELIABILITY;

EID: 50849119091     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/coli.2008.34.3.319     Document Type: Article

References (18)

1. Aron, Arthur and Elaine N. Aron. 2003. Statistics for Psychology. Prentice Hall, Upper Saddle River, NJ.
2. Artstein, Ron and Massimo Poesio. 2005. Bias decreases in proportion to the number of annotators. In Proceedings of FG-MoL 2005, pages 141-150, Edinburgh.
3. Artstein, Ron and Massimo Poesio. In press. Inter-coder agreement for computational linguistics. Computational Linguistics.
4. Bayerl, Petra Saskia and Karsten Ingmar Paul. 2007. Identifying sources of disagreement: Generalizability theory in manual annotation studies. Computational Linguistics, 33(43-8.
5. Carletta, Jean C. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249-254.
6. Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46.
7. Cohen, Jacob. 1988. Statistical power analysis for the behavioral sciences, 2nd edition. Lawrence Erlbaum, Hillsdale, NJ.
8. Craggs, Richard and Mary McGee Wood. 2005. Evaluating discourse and dialogue coding schemes. Computational Linguistics, 31(3):289-296.
9. Di Eugenio, Barbara and Michael Glass. 2004. The kappa statistic: A second look. Computational Linguistics, 30(1):95-101.
10. Galley, Michel, Kathleen McKeown, Julia Hirschberg, and Elizabeth Shriberg. 2004. Identifying agreement and disagreement in conversational speech: Use of Bayesian networks to model pragmatic dependencies. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL-04), pages 669-676, Barcelona.
11. Krippendorff, Klaus. 1980. Content Analysis: An Introduction to its Methodology, volume 5 of The Sage Comm Text Series. Sage Publications, London.
12. Krippendorff, Klaus. 2004. Reliability in content analysis. Some common misconceptions and recommendations. Human Communication Research, 30(3):411-433.
13. Marcu, Daniel, Estibaliz Amorrortu, and Magdalena Romera. 1999. Experiments in constructing a corpus of discourse trees. In Marilyn Walker, editor, Towards Standards and Tools for Discourse Tagging: Proceedings of the Workshop. Association for Computational Linguistics, Somerset, NJ, pages 48-57.
14. Pearl, Judea. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann Publishers Inc., San Francisco, CA.
15. Reithinger, Norbert and Michael Kipp. 1998. Large scale dialogue annotation in Verbmobil. In Workshop Proceedings of ESSLLI 98, pages 1-6, Saarbrücken.
16. Shriberg, Elizabeth, Rebecca Bates, Paul Taylor, Andreas Stolcke, Daniel Jurafsky, Klaus Ries, Noah Coccaro, Rachel Martin, Marie Meteer, and Carol Van Ess-Dykema. 1998. Can prosody aid the automatic classification of dialog acts in conversational speech? Language and Speech, 41(3-4):443-492.
17. Wiebe, Janyce M., Rebecca F. Bruce, and Thomas P. O'Hara. 1999. Development and use of a gold-standard data set for subjectivity classifications. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 246-253, Morristown, NJ.
18. Witten, Ian H. and Eibe Frank. 2005. Data Mining: Practical Machine Learning Tools and Techniques, 2nd edition. Morgan Kaufmann, San Francisco, CA.