Identifying sources of disagreement: Generalizability theory in manual annotation studies

Computational Linguistics

Volumn 33, Issue 1, 2007, Pages 3-8

  Bayerl, Petra Saskia ^a,c   Paul, Karsten Ingmar ^b,d  

^a UNIVERSITY OF OKLAHOMA   (United States)

^b UNIVERSITY OF ERLANGEN NUREMBERG   (Germany)

^c Tulsa Community College   (United States)

^d Organizational and Social Psychology ^*  (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

BASIC CONCEPTS; GENERALIZABILITY THEORIES; ITS APPLICATIONS; MANUAL ANNOTATION;

QUALITY CONTROL;

EID: 34047098376     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/coli.2007.33.1.3     Document Type: Article

References (12)

1. Agresti, Alan. 1992. Modelling patterns of agreement and disagreement. Statistical Methods in Medical Research, 1(2):201-218.
2. Brennan, Robert L. 2001. Generalizability Theory. Statistics for Social Science and Public Policy. Springer Verlag, New York.
3. Carletta, Jean. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249-254.
4. Chiu, Christopher W. T. and Edward W. Wolfe. 2003. A method for analyzing sparse data matrices in the generalizability theory framework. Applied Psychological Measurement, 26(3):321-338.
5. Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46.
6. Cronbach, Lee J., Goldine C. Gleser, Harinder Nanda, and Nageswari Rajaratnam. 1972. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. John Wiley & Sons, Inc., New York.
7. Eugenio, Barbara Di and Michael Glass. 2004. The kappa statistic: A second look. Computational Linguistics, 30(1):95-101.
8. Matt, Georg E. 2001. Generalizabilty theory. In Neil J. Smelser and Paul B. Baltes, editors, International Encyclopedia of the Social and Behavioral Sciences. Elsevier, Oxford.
9. Shavelson, Richard J. and Noreen M. Webb. 1981. Generalizability Theory: A Primer. Sage Publications, Newbury Park, London, New Delhi.
10. Shriberg, Lawrence D. and Gregory L. Lof. 1991. Reliability studies in broad and narrow phonetic transcription. Clinical Linguistics and Phonetics, 5(3):225-279.
11. Spearman, Charles. 1904. The proof and measurement of association between two things. American Journal of Psychology, 15:72-101.
12. Thompson, Bruce. 2002. A brief introduction to Generalizability Theory. In Bruce Thompson, editor, Score Reliability: Contemporary Thinking on Reliability Issues. Sage Publication, Thousand Oaks, CA, pages 43-58.