Inter-coder agreement for computational linguistics

Computational Linguistics

Volumn 34, Issue 4, 2008, Pages 555-596

  Artstein, Ron ^a,c   Poesio, Massimo ^a,b  

^a UNIVERSITY OF ESSEX   (United Kingdom)

^b UNIVERSITY OF TRENTO   (Italy)

^c University of Southern California ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AGREEMENT COEFFICIENT; COHEN'S KAPPAS; CORPUS ANNOTATIONS; KRIPPENDORFF'S ALPHAS;

COMPUTATIONAL LINGUISTICS;

EID: 57349126313     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/coli.07-034-R2     Document Type: Review

References (110)

1. Allen, James and Mark Core. 1997. DAMSL: Dialogue act markup in several layers. Draft contribution for the Discourse Resource Initiative, University of Rochester. Available at http://www.cs.rochester.edu/ research/cisd/resources/ dams1/.
2. Artstein, Ron and Massimo Poesio. 2005. Bias decreases in proportion to the number of annotators. In Proceedings of FG-MoL 2005, pages 141-150, Edinburgh.
3. Artstein, Ron and Massimo Poesio. 2006. Identifying reference to abstract objects in dialogue. In brandial 2006: Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue, pages 56-63, Potsdam.
4. Atkins, Sue. 1992. Tools for computer-aided corpus lexicography: The Hector project. Acta Linguistica Hungarica, 41:5-71.
5. Babarczy, Anna, John Carroll, and Geoffrey Sampson. 2006. Definitional, personal, and mechanical constraints on part of speech annotation performance. Natural Language Engineering, 12(1):77-90.
6. Bartko, John J. and William T. Carpenter, Jr. 1976. On the methods and theory of reliability. Journal of Nervous and Mental Disease, 163(5):307-317.
7. Beeferman, Doug, Adam Berger, and John Lafferty. 1999. Statistical models for text segmentation. Machine Learning, 34(1-3):177-210.
8. Bennett, E. M., R. Alpert, and A. C. Goldstein. 1954. Communications through limited questioning. Public Opinion Quarterly, 18(3):303-308.
9. Bloch, Daniel A. and Helena Chmura Kraemer. 1989. 2 × 2 kappa coefficients: Measures of agreement or association. Biometrics, 45(1):269-287.
10. Brennan, Robert L. and Dale J. Prediger. 1981. Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3):687-699.
11. Bruce, Rebecca and Janyce Wiebe. 1998. Word-sense distinguishability and inter-coder agreement. In Proceedings of EMNLP, pages 53-60, Granada.
12. Bruce, Rebecca F. and Janyce M. Wiebe. 1999. Recognizing subjectivity: A case study in manual tagging. Natural Language Engineering, 5(2):187-205.
13. Buitelaar, Paul. 1998. CoreLex : Systematic Polysemy and Underspecification. Ph.D. thesis, Brandeis University, Waltham, MA.
14. Bunt, Harry C. 2000. Dynamic interpretation and dialogue theory. In Martin M. Taylor, Françoise Néel, and Don G. Bouwhuis, editors, The Structure of Multimodal Dialogue II. John Benjamins, Amsterdam, pages 139-166.
15. Bunt, Harry C. 2005. A framework for dialogue act specification. In Proceedings of the Joint ISO-ACL Workshop on the Representation and Annotation of Semantic Information, Tilburg. Available at: http://let.uvt.nl/research/ti/sigsem/wg/discussionnotes4.htm.
16. Byron, Donna K. 2002. Resolving pronominal reference to abstract entities. In Proceedings of the 40th Annual Meeting of the ACL, pages 80-87, Philadelphia, PA.
17. Byrt, Ted, Janet Bishop, and John B. Carlin. 1993. Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46(5):423-429.
18. Carletta, Jean. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249-254.
19. Carletta, Jean, Amy Isard, Stephen Isard, Jacqueline C. Kowtko, Gwyneth Doherty-Sneddon, and Anne H. Anderson. 1997. The reliability of a dialogue structure coding scheme. Computational Linguistics, 23(1):13-32.
20. Carlson, Lynn, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Jan C. J. van Kuppevelt and Ronnie W. Smith, editors, Current and New Directions in Discourse and Dialogue. Kluwer, Dordrecht, pages 85-112.
21. Cicchetti, Domenic V. and Alvan R. Feinstein. 1990. High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6)551-558.
22. Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46.
23. Cohen, Jacob. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213-220.
24. Core, Mark G. and James F. Allen. 1997. Coding dialogs with the DAMSL annotation scheme. In Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines, AAAI, Cambridge, MA. Available at: http://www.cs.umd.edu/~traum/CA/fpapers.html.
25. Craggs, Richard and Mary McGee Wood. 2004. A two-dimensional annotation scheme for emotion in dialogue. In Papers from the 2004 AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, Stanford, pages 44-49.
26. Craggs, Richard and Mary McGee Wood. 2005. Evaluating discourse and dialogue coding schemes. Computational Linguistics, 31(3):289-295.
27. Davies, Mark and Joseph L. Fleiss. 1982. Measuring agreement for multinomial data. Biometrics, 38(4):1047-1051.
28. Di Eugenio, Barbara. 2000. On the usage of Kappa to evaluate agreement on coding tasks. In Proceedings of LREC, volume 1, pages 441-444, Athens.
29. Di Eugenio, Barbara and Michael Glass. 2004. The kappa statistic: A second look. Computational Linguistics, 30(1):95-101.
30. Di Eugenio, Barbara, Pamela W. Jordan, Johanna D. Moore, and Richmond H. Thomason. 1998. An empirical investigation of proposals in collaborative dialogues. In Proceedings of 36th Annual Meeting of the ACL, pages 325-329, Montreal.
31. Dice, Lee R. 1945. Measures of the amount of ecologic association between species. Ecology, 26(3):297-302.
32. Donner, Allan and Michael Eliasziw. 1987. Sample size requirements for reliability studies. Statistics in Medicine, 6:441-448.
33. Doran, Christine, John Aberdeen, Laurie Damianos, and Lynette Hirschman. 2001. Comparing several aspects of human-computer and human-human dialogues. In Proceedings of the 2nd SIGdial Workshop on Discourse and Dialogue, Aalborg, Denmark. Available at: http://www.sigdial.org/workshops/workshop2/ proceedings.
34. Eckert, Miriam and Michael Strube. 2000. Dialogue acts, synchronizing units, and anaphora resolution. Journal of Semantics, 17(1):51-89.
35. Feinstein, Alvan R. and Domenic V. Cicchetti. 1990. High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6)543-549.
36. Fellbaum, Christiane, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA.
37. Fleiss, Joseph L. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378-382.
38. Fleiss, Joseph L. 1975. Measuring agreement between two judges on the presence or absence of a trait. Biometrics, 31(3):651-659.
39. Francis, W. Nelson and Henry Kucera. 1982. Frequency Analysis of English Usage: lexicon and grammar. Houghton Mifflin, Boston, MA.
40. Geertzen, Jeroen and Harry Bunt. 2006. Measuring annotator agreement in a complex hierarchical dialogue act annotation scheme. In Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue, pages 126-133, Sydney.
41. Gross, Derek, James F. Allen, and David R. Traum. 1993. The Trains 91 dialogues. TRAINS Technical Note 92-1, University of Rochester Computer Science Department, Rochester, NY.
42. Grosz, Barbara J. and Candace L. Sidner. 1986. Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3):175-204.
43. Hayes, Andrew F. and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1):77-89.
44. Hearst, Marti A. 1997. TextTiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics, 23(1):33-64.
45. Hovy, Eduard, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of HLT-NAACL, Companion Volume: Short Papers, pages 57-60, New York.
46. n, Cohen's kappa, Scott's π, and Aickin's α.. Understanding Statistics, 2(3):205-219.
47. Jaccard, Paul. 1912. The distribution of the flora in the Alpine zone. New Phytologist, 11(2):37-50.
48. Jekat, Susanne, Alexandra Klein, Elisabeth Maier, Ilona Maleck, Marion Mast, and J. Joachim Quantz. 1995. Dialogue acts in VERBMOBIL. VM-Report 65, Universität Hamburg, DFKI GmbH, Universität Erlangen, and TU Berlin.
49. Jurafsky, Daniel, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. Technical Report 97-02, University of Colorado at Boulder, Institute for Cognitive Science.
50. Kilgarriff, Adam. 1999. 95% replicability for manual word sense tagging. In Proceedings of the Ninth Conference of the European Chapter of the Association for Computational Linguistics, pages 277-278, Bergen, Norway.
51. Klavans, Judith L., Samuel Popper, and Rebecca Passonneau. 2003. Tackling the internet glossary glut: Automatic extraction and evaluation of genus phrases. In Proceedings of the SIGIR-2003 Workshop on the Semantic Web, Toronto.
52. Kowtko, Jacqueline C., Stephen D. Isard, and Gwyneth M. Doherty. 1992. Conversational games within dialogue. Research Paper HCRC/RP-31, Human Communication Research Centre, University of Edinburgh.
53. Krippendorff, Klaus. 1970. Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1):61-70.
54. Krippendorff, Klaus. 1978. Reliability of binary attribute data. Biometrics, 34(1):142-144. Letter to the editor, with a reply by Joseph L. Fleiss.
55. Krippendorff, Klaus. 1980. Content Analysis: An Introduction to Its Methodology, chapter 12. Sage, Beverly Hills, CA.
56. Krippendorff, Klaus. 1995. On the reliability of unitizing contiguous data. Sociological Methodology, 25:47-76.
57. Krippendorff, Klaus. 2004a. Content Analysis: An Introduction to Its Methodology, second edition, chapter 11. Sage, Thousand Oaks, CA.
58. Krippendorff, Klaus. 2004b. Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3):411-133.
59. Landis, J. Richard and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159-174.
60. Leech, Geoffrey, Roger Garside, and Michael Bryant. 1994. CLAWS4: The tagging of the British National Corpus. In Proceedings of COLING 1994: The 15th International Conference on Computational Linguistics, Volume 1, pages 622-628, Kyoto.
61. Levin, James A. and James A. Moore. 1978. Dialogue-games: Metacommunication structures for natural language interaction. Cognitive Science, 1(4):395-420.
62. Manning, Christopher D. and Hinrich Schuetze. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA.
63. Marcu, Daniel, Magdalena Romera, and Estibaliz Amorrortu. 1999. Experiments in constructing a corpus of discourse trees: Problems, annotation choices, issues. In Workshop on Levels of Representation in Discourse, pages 71-78, University of Edinburgh.
64. Marcus, Mitchell P., Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313-330.
65. Marion, Rodger. 2004. The whole art of deduction. Unpublished manuscript.
66. Melamed, I. Dan and Philip Resnik. 2000. Tagger evaluation given hierarchical tagsets. Computers and the Humanities, 34(1-2):79-84. Available at: http://www.sahs/utmb.edu/PELLIMORE/Intro-to-research/wad/wad/- home.htm.
67. Mieskes, Margot and Michael Strube. 2006. Part-of-speech tagging of transcribed speech. In Proceedings of LREC, pages 935-938, Genoa.
68. Mihalcea, Rada, Timothy Chklovski, and Adam Kilgarriff. 2004. The SENSEVAL-3 English lexical sample task. In Proceedings of SENSEVAL-3, pages 25-28, Barcelona.
69. Miltsakaki, Eleni, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. Annotating discourse connectives and their arguments. In Proceedings of the HLT-NAACL Workshop on Frontiers in Corpus Annotation, pages 9-16, Boston, MA.
70. Moser, Megan G., Johanna D. Moore, and Erin Glendening. 1996. Instructions for Coding Explanations: Identifying Segments, Relations and Minimal Units. Technical Report 96-17, University of Pittsburgh, Department of Computer Science.
71. Navarretta, Costanza. 2000. Abstract anaphora resolution in Danish. In Proceedings of the 1st SIGdial Workshop on Discourse and Dialogue, Hong Kong, pages 56-65.
72. Nenkova, Ani and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of HLT-NAACL 2004, pages 145-152, Boston, MA.
73. Neuendorf, Kimberly A. 2002. The Content Analysis Guidebook. Sage, Thousand Oaks, CA.
74. Palmer, Martha, Hoa Trang Dang, and Christiane Fellbaum. 2007. Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Natural Language Engineering, 13(2):137-163.
75. Passonneau, Rebecca J. 2004. Computing reliability for coreference annotation. In Proceedings of LREC, volume 4, pages 1503-1506, Lisbon.
76. Passonneau, Rebecca J. 2006. Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In Proceedings of LREC, Genoa, pages 831-836.
77. Passonneau, Rebecca J., Nizar Habash, and Owen Rambow. 2006. Inter-annotator agreement on a multilingual semantic annotation task. In Proceedings of LREC, Genoa, pages 1951-1956.
78. Passonneau, Rebecca J. and Diane J. Litman. 1993. Intention-based segmentation: Human reliability and correlation with linguistic cues. In Proceedings of 31st Annual Meeting of the ACL, pages 148-155, Columbus, OH.
79. Passonneau, Rebecca J. and Diane J. Litman. 1996. Empirical analysis of three dimensions of spoken discourse: Segmentation, coherence and linguistic devices. In Eduard H. Hovy and Donia R. Scott, editors, Computational and Conversational Discourse: Burning Issues - An Interdisciplinary Account, volume 151 of NATO ASI Series F: Computer and Systems Sciences. Springer, Berlin, chapter 7, pages 161-194.
80. Passonneau, Rebecca J. and Diane J. Litman. 1997. Discourse segmentation by human and automated means. Computational Linguistics, 23(1):103-139.
81. Pevzner, Lev and Marti A. Hearst. 2002. A critique and improvement of an evaluation metric for text segmentation. Computational Linguistics, 28(1):19-36.
82. Poesio, Massimo. 2004a. Discourse annotation and semantic annotation in the GNOME corpus. In Proceedings of the 2004 ACL Workshop on Discourse Annotation, pages 72-79, Barcelona.
83. Poesio, Massimo. 2004b. The MATE/GNOME proposals for anaphoric annotation, revisited. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue, pages 154-162, Cambridge, MA.
84. Poesio, Massimo and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the Workshop on Frontiers in Corpus Annotation II: Pie in the Sky, pages 76-83, Ann Arbor, MI.
85. Poesio, Massimo and Natalia N. Modjeska. 2005. Focus, activation, and this-noun phrases: An empirical study. In António Branco, Tony McEnery, and Ruslan Mitkov, editors, Anaphora Processing, volume 263 of Current Issues in Linguistic Theory. John Benjamins, pages 429-442, Amsterdam and Philadelphia.
86. Poesio, Massimo, A. Patel, and Barbara Di Eugenio. 2006. Discourse structure and anaphora in tutorial dialogues: An empirical analysis of two theories of the global focus. Research in Language and Computation, 4(2-3):229-257.
87. Poesio, Massimo and Renata Vieira. 1998. A corpus-based investigation of definite description use. Computational Linguistics, 24(2):183-216.
88. Popescu-Belis, Andrei. 2005. Dialogue acts: One or more dimensions? Working Paper 62, ISSCO, University of Geneva.
89. Posner, Karen L., Paul D. Sampson, Robert A. Caplan, Richard J. Ward, and Frederick W. Cheney. 1990. Measuring interrater reliability among multiple raters: An example of methods for nominal data. Statistics in Medicine, 9:1103-1115.
90. Rajaratnam, Nageswari. 1960. Reliability formulas for independent decision data when reliability data are matched. Psychometrika, 25(3):261-271.
91. Reidsma, Dennis and Jean Carletta. 2008. Reliability measurement without limits. Computational Linguistics, 34(3):319-326.
92. Reinhart, T. 1981. Pragmatics and linguistics: An analysis of sentence topics. Philosophica, 27(1)53-93.
93. Reynar, Jeffrey C. 1998. Topic Segmentation: Algorithms and Applications. Ph.D. thesis, University of Pennsylvania, Philadelphia.
94. Ries, Klaus. 2002. Segmenting conversations by topic, initiative and style. In Anni R. Coden, Eric W. Brown, and Savitha Srinivasan, editors, Information Retrieval Techniques for Speech Applications, volume 2273 of Lecture Notes in Computer Science. Springer, Berlin, pages 51-66.
95. Rietveld, Toni and Roeland van Hout. 1993. Statistical Techniques for the Study of Language and Language Behaviour. Mouton de Gruyter, Berlin.
96. Rosenberg, Andrew and Ed Binkowski. 2004. Augmenting the kappa statistic to determine interannotator reliability for multiply labeled data points. In Proceedings of HLT-NAACL 2004: Short Papers, pages 77-80, Boston, MA.
97. Scott, William A. 1955. Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3):321-325.
98. Shriberg, Elizabeth, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue, pages 97-100, Cambridge, MA.
99. Siegel, Sidney and N. John Castellan, Jr. 1988. Nonparametric Statistics for the Behavioral Sciences, 2nd edition, chapter 9.8. McGraw-Hill, New York.
100. Stent, Amanda J. 2001. Dialogue Systems as Conversational Partners: Applying Conversation Acts Theory to Natural Language Generation for Task-Oriented Mixed-Initiative Spoken Dialogue. Ph.D. thesis, Department of Computer Science, University of Rochester.
101. Stevenson, Mark and Robert Gaizauskas. 2000. Experiments on sentence boundary detection. In Proceedings of 6th ANLP, pages 84-89, Seattle, WA.
102. Stolcke, Andreas, Noah Coccaro, Rebecca Bates, Paul Taylor, Carol Van Ess-Dykema, Klaus Ries, Elizabeth Shriberg, Daniel Jurafsky, Rachel Martin, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339-373.
103. Stuart, Alan. 1955. A test for homogeneity of the marginal distributions in a two-way classification. Biometrika, 42(3/4):412-416.
104. Teufel, Simone, Jean Carletta, and Marc Moens. 1999. An annotation scheme for discourse-level argumentation in research articles. In Proceedings of Ninth Conference of the EACL, pages 110-117, Bergen.
105. Teufel, Simone and Marc Moens. 2002. Summarizing scientific articles: Experiments with relevance and rhetorical status. Computational Linguistics, 28(4):409-445.
106. Traum, David R. and Elizabeth A. Hinkelman. 1992. Conversation acts in task-oriented spoken dialogue. Computational Intelligence, 8(3)575-599.
107. Vallduví, Enric. 1993. Information packaging: A survey. Research Paper RP-44, University of Edinburgh, HCRC.
108. Véronis, Jean. 1998. A study of polysemy judgments and inter-annotator agreement. In Proceedings of SENSEVAL-1, Herstmonceux Castle, England. Available at: http://www.itri.brighton.ac.uk/events/senseval/ ARCHIVE/PROCEEDINGS/
109. Vilain, Marc, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the Sixth Message Understanding Conference, pages 45-52, Columbia, MD.
110. Zwick, Rebecca. 1988. Another look at interrater agreement. Psychological Bulletin, 103(3):374-378.