Including omission mistakes in the calculation of Cohen's Kappa and an analysis of the coefficient's paradox features

Educational and Psychological Measurement

Volumn 66, Issue 5, 2006, Pages 765-777

  Simon, Patricia ^a,b  

^a UNIVERSITY OF REGENSBURG   (Germany)

^b NONE   (Germany)

Author keywords

Base rate problem; Kappa coefficient; Observer bias; Omission mistakes; Sequential observation

Indexed keywords

EID: 33748088163     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164405285548     Document Type: Article

References (24)

1. Asendorpf, J., & Wallbott, H. G. (1979). Maße der Beobachterübereinstimmung: Ein systematischer Vergleich [Measures of observer agreement: A systematic comparison]. Zeitschrift für Sozialpsychologie, 10, 243-252.
2. Bakeman, R., & Gottman, J. M. (1986). Observing interaction: An introduction to sequential analysis. London: Cambridge University Press.
3. Bales, R. F. (1950). Interaction process analysis: A method for the study of small groups. Cambridge, MA: Addison-Wesley.
4. Brennan, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses and misuses, and alternatives. Educational and Psychological Measurement, 41, 687-699.
5. Byrt, T., Bishop, J., & Carlin, J. B. (1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46 (5). 423-429.
6. Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43, 551-558.
7. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
8. Cook, R. J. (1998). Kappa and its dependence on marginal rates. In T. P. Armitage & T. Colton (Eds.), The encyclopedia of biostatictics (pp. 2166-2168). New York: John Wiley.
9. Cranach, M. v., & Frenz, H. G. (1969). Systematische Beobachtung [Systematic observation]. In K. Gottschaldt, P. Lersch, F. Sander, & H. Thomae (Eds.), Handbuch der Psychologie in 12 Bänden (Bd. 7,1, pp. 269-331). Göttingen, Germany: Hogrefe.
10. Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43, 543-549.
11. Fleiss, J. L. (1981). Statistical methods for raters and proportions. New York: John Wiley.
12. Grove, W. M., Andreason, N. C., & McDonald-Scott, P. (1981). Reliability studies of psychiatric diagnosis. Archives of General Psychiatry, 38, 408-413.
13. Hollenbeck, A. R. (1978). Problems of reliability in observational research. In G. P. Sackett (Ed.), Observing behavior, Vol. 2: Data collection and analysis methods (pp. 79-98). Baltimore: University Park Press.
14. Kauffeld, S. (2003). Gruppensitzungen unter der Lupe - Das Kasseler-Kompetenz-Raster als prozessanalytische Diagnosemethode zur Teamentwicklung [Group sessions under a magnifying glass - The Kasseler-Competence-Grid as a process-analytical method of diagnosis in team development]. In S. Stumpf & A. Thomas (Eds.), Teamarbeit und Teamentwicklung. Göttingen, Germany: Hogrefe.
15. Kvalseth, T. O. (1991). A coefficient of agreement for nominal scales: An asymmetric version of kappa. Educational and Psychological Measurement, 51, 95-101.
16. Lantz, C. A., & Nebenzahl, E. (1996). Behavior and interpretation of the kappa statistic: Resolution of the two paradoxes. Journal of Clinical Epidemiology, 49 (4). 431-434.
17. Light, R. J. (1971). Measures of response agreement for qualitative data: Some generalizations and alternatives. Psychological Bulletin, 76, 365-377.
18. Scharpf, U. (1988). Entscheidungsfindung im Gruppenprozess [Decision-making in group processes]. Konstanz, Germany: Hartung-Gorre.
19. Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19, 321-325.
20. Simon, P. (2002). Die Entwicklung eines Modells der Gruppeneffektivität und eines Analyse-Instruments zur Erfassung des Leistungspotentials von Arbeitsgruppen [The development of a group effectiveness model and an analysis instrument for measuring the output potential of work groups]. Landau, Germany: Verlag Empirische Pädagogik.
21. Spitznagel, E. L., & Helzer, J. E. (1985). A proposed solution to the base rate problem in the kappa statistic. Archives of General Psychiatry, 42 (7). 725-728.
22. Thompson, W. D., & Walter, S. D. (1988). A reappraisal of the kappa coefficient. Journal of Clinical Epidemiology, 41, 949-958.
23. Übersax, J. S. (1987). Diversity of decision making models and the measurement of interrater agreement. Psychological Bulletin, 101, 140-146.
24. Zwick, R. (1988). Another look at interrater agreement. Psychological Bulletin, 103 (4). 374-378.