Ranking the annotators: An agreement study on argumentation structure

LAW 2013 and ID 2013 - 7th Linguistic Annotation Workshop and Interoperability with Discourse, Proceedings of the Workshop

Volumn , Issue , 2020, Pages 196-204

  Peldszus, Andreas ^a   Stede, Manfred ^a  

^a UNIVERSITY OF POTSDAM   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

LINGUISTICS;

ARGUMENTATION STRUCTURES; LARGE GROUPS; SHORT TEXTS;

INTEROPERABILITY;

EID: 84924942152     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (16)

1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555-596, December.
2. Vikas Bhardwaj, Rebecca J. Passonneau, Ansaf Salleb-Aouissi, and Nancy Ide. 2010. Anveshan: a framework for analysis of multiple annotators' labeling behavior. In Proceedings of the Fourth Linguistic Annotation Workshop, LAW IV '10, pages 47-55, Stroudsburg, PA, USA. Association for Computational Linguistics.
3. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Jan van Kuppevelt and Ronnie Smith, editors, Current Directions in Discourse and Dialogue. Kluwer, Dordrecht.
4. Silvie Cinková, Martin Holub, and Vincent Kríž. 2012. Managing uncertainty in semantic tagging. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL '12, pages 840-850, Stroudsburg, PA, USA. Association for Computational Linguistics.
5. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378-382.
6. James B. Freeman. 1991. Dialectics and the Macrostructure of Argument. Foris, Berlin.
7. James B. Freeman. 2011. Argument Structure: Representation and Theory. Argumentation Library (18). Springer.
8. Jeroen Geertzen and Harry Bunt. 2006. Measuring annotator agreement in a complex hierarchical dialogue act annotation scheme. In Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue, SigDIAL '06, pages 126-133, Stroudsburg, PA, USA. Association for Computational Linguistics.
9. Klaus Krippendorff. 1980. Content Analysis: An Introduction to its Methodology. Sage Publications, Beverly Hills, CA.
10. J Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorial data. Biometrics, 33(1):159-174, March.
11. William Mann and Sandra Thompson. 1988. Rhetorical structure theory: Towards a functional theory of text organization. TEXT, 8:243-281.
12. Andreas Peldszus and Manfred Stede. to appear. From argument diagrams to automatic argument mining: A survey. International Journal of Cognitive Informatics and Natural Intelligence, 7(1).
13. William A. Scott. 1955. Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3):321-325.
14. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast-but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '08, pages 254-263, Stroudsburg, PA, USA. Association for Computational Linguistics.
15. Simone Teufel. 2010. The Structure of Scientific Articles: Applications to Citation Indexing and Summarization. CSLI Studies in Computational Linguistics. CSLI Publications.
16. Stephen Toulmin. 1958. The Uses of Argument. Cambridge University Press, Cambridge.