On the usage of Kappa to evaluate agreement on coding tasks

2nd International Conference on Language Resources and Evaluation, LREC 2000

Volumn , Issue , 2000, Pages

  Di Eugenio, Barbara ^a  

^a 1120 Science and Engineering Offices   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DE FACTO STANDARD; DIALOGUE PROCESSING; KAPPA COEFFICIENT; KAPPA VALUES; SYSTEM'S PERFORMANCE;

EID: 84930926511     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (20)

1. Austin, John L., 1962. How to Do Things With Words. Oxford: Oxford University Press.
2. Berry, Charles C., 1992. The K statistic - to the editor. Letters, Journal of the American Medical Association, 268 (18).
3. Carletta, Jean, 1996. Assessing agreement on classification tasks: the kappa statistic. Computational Lingustics, 22(2):249-254.
4. Cohen, Jacob, 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37-46.
5. DAMSL, 1997. DAMSL: Dialog act markup in several layers. Available under Tools and resources, at http://www.georgetown.edu/luperfoy/Discourse-Treebank/dri-home.html.
6. Di Eugenio, Barbara, Pamela W. Jordan, Johanna D. Moore, and Richmond H. Thomason, 1998. An empirical investigation of collaborative dialogues. In ACL-COLING98, Proceedings of the Thirty-sixth Conference of the Association for Computational Linguistics. Montreal, Canada.
7. Di Eugenio, Barbara, Pamela W. Jordan, Richmond H. Thomason, and Johanna D. Moore, 2000. The agreement process: An empirical investigation of humanhuman computer-mediated collaborative dialogues. International Journal of Human Computer Studies. To appear.
8. Goldman, Ronald L., 1992. The K statistic - to the editor (in reply). Letters, Journal of the American Medical Association, 268 (18).
9. Grove, William M., Nancy C. Andreasen, Patricia McDonald-Scott, Martin B. Keller, and Robert W. Shapiro, 1981. Reliability Studies of Psychiatric Diagnosis. Theory and Practice. Archives of General Psychiatry, 38:408-413.
10. Krippendorff, Klaus, 1980. Content Analysis: an Introduction to its Methodology. Beverly Hills: Sage Publications.
11. Mitchell, Tom, 1997. Machine Learning. McGraw Hill.
12. Passonneau, Rebecca J., 1997. Applying reliability metrics to co-reference annotation. Technical Report CUCS-017-097, Columbia University.
13. Rietveld, T. and R. van Hout, 1993. Statistical Techniques for the Study of Language and Language Behaviour. Mouton de Gruyter.
14. Searle, John R., 1965. What Is a Speech Act. In Max Black (ed.), Philosophy in America. Ithaca, New York: Cornell University Press, pages 615-628.
15. Reprinted in Pragmatics. A Reader, Steven Davis editor, Oxford University Press, 1991.
16. Searle, John R., 1975. Indirect Speech Acts. In P. Cole and J. L. Morgan (eds.), Syntax and Semantics 3. Speech Acts. Academic Press.
17. Reprinted in Pragmatics. A Reader, Steven Davis editor, Oxford University Press, 1991.
18. Siegel, Sidney and N. John Castellan, Jr., 1988. Nonparametric statistics for the behavioral sciences. McGraw Hill.
19. Traum, David and James Allen, 1994. Discourse Obligations in Dialogue Processing. In ACL94, Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics.
20. Wiebe, Janyce M., Rebecca F. Bruce, and Thomas P. O'Hara, 1999. Development and use of a gold-standard data set for subjectivity classifications. In ACL99, Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics. College Park, MD.