Measuring annotator agreement in a complex hierarchical dialogue act annotation scheme

SIGDIAL 2006 - 7th SIGdial Workshop on Discourse and Dialogue, Proceedings

Volumn , Issue , 2006, Pages 126-133

  Geertzen, Jeroen ^a   Bunt, Harry ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

ANNOTATION SCHEME; DIALOG ACTS; KAPPA STATISTIC;

EID: 85123803949     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (23)

1. James Allen and Mark Core. 1997. Draft of DAMSL: Dialog act markup in several layers. Unpublished manuscript.
2. J. Allwood, J. Nivre, and E. Ahlsén. 1993. Manual for coding interaction management. Technical report, Göteborg University. Project report: Semantik och talspråk.
3. Harry C. Bunt and Yann Girard. 2005. Designing an open, multidimensional dialogue act taxonomy. In Proceedings of the 9th Workshop on the Semantics and Pragmatics of Dialogue (DIALOR 2005), pages 37–44, Nancy, France, June.
4. Harry C. Bunt. 2000. Dialogue pragmatics and context specification. In Harry C. Bunt and William Black, editors, Abduction, Belief and Context in Dialogue; Studies in Computational Pragmatics, pages 81–150. John Benjamins, Amsterdam, The Netherlands.
5. Harry C. Bunt. 2005. A framework for dialogue act specification. In Joint ISO-ACL Workshop on the Representation and Annotation of Semantic Information, Tilburg, The Netherlands, January.
6. Harry C. Bunt. 2006. Dimensions in dialogue annotation. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006), Genova, Italy, May.
7. Jean Carletta. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249–254.
8. Johanneke Caspers. 2000. Pitch accents, boundary tones and turn-taking in dutch map task dialogues. In Proceedings of the 6th International Conference on Spoken Language Processing (ICSLP 2000), volume 1, pages 565–568, Beijing, China.
9. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Education and Psychological Measurement, 20:37–46.
10. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70:213–220.
11. Mark G. Core and James F. Allen. 1997. Coding dialogues with the DAMSL annotation scheme. In David Traum, editor, Working Notes: AAAI Fall Symposium on Communicative Action in Humans and Machines, pages 28–35, Menlo Park, CA, USA. American Association for Artificial Intelligence.
12. Mark Davies and J.L. Fleiss. 1982. Measuring agreement for multinomial data. Biometrics, 38:1047–1051.
13. Barbara Di Eugenio and Michael Glass. 2004. The kappa statistic: a second look. Computational Linguistics, 30(1):95–101.
14. Barbara Di Eugenio, Pamela W. Jordan, Johanna D. Moore, and Richmond H. Thomason. 1998. An empirical investigation of proposals in collaborative dialogues. In Proceedings of the 17th International Conference on Computational Linguistics and the 36th Annual Meeting of the Association for Computational Linguistics (COLING-ACL 1998), pages 325–329, Montreal, Canada.
15. Jeroen Geertzen, Yann Girard, Roser Morante, Ielka van der Sluis, Hans Van Dam, Barbara Suijkerbuijk, Rintse van der Werf, and Harry Bunt. 2004. The diamond project (poster,project description). In The 8th Workshop on the Semantics and Pragmatics of Dialogue (Catalog’04). Barcelona, Spain.
16. ++ dimensions. Technical report, Tilburg University, Tilburg, The Netherlands.
17. Derek Gross, James F. Allen, and David R. Traum. 1993. The TRAINS 91 dialogues. Technical Report TN92-1, University of Rochester, Rochester, NY, USA.
18. Peter A. Heeman and James F. Allen. 1995. The TRAINS 93 dialogues. Technical Report TN94-2, University of Rochester, Rochester, NY, USA.
19. Klaus Krippendorff. 1980. Content Analysis: An Introduction to its Methodology. Sage Publications, Beverly Hills, CA, USA.
20. Stephan Lesch, Thomas Kleinbauer, and Jan Alexandersson. 2005. A new metric for the evaluation of dialog act classification. In Proceedings of the 9th Workshop on the Semantics and Pragmatics of Dialogue (DIALOR 2005), pages 143–146, Nancy, France, june.
21. Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue, pages 97–100, Boston, USA, April-May.
22. Amanda J. Stent. 2000. The monroe corpus. Technical Report TR728/TN99-2, University of Rochester, Rochester, UK.
23. Helmer Strik, Albert Russel, Henk van den Heuvel, Catia Cucchiarini, and Lou Boves. 1997. A spoken dialog system for the dutch public transport information service. International Journal of Speech Technology, 2(2):119–129.