Cohen's Linearly Weighted Kappa is a Weighted Average of 2×2 Kappas

Psychometrika

Volumn 76, Issue 3, 2011, Pages 471-486

  Warrens, Matthijs J ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

Cohen's kappa; Hubert's weighted kappa; linear weights; merging categories; Mielke, Berry and Johnston's weighted kappa; quadratic weights

Indexed keywords

EID: 79959373939     PISSN: 00333123     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11336-011-9210-z     Document Type: Article

References (48)

1. Agresti, A. (1990). Categorical data analysis. New York: Wiley.
2. 3 = alpha (or beta). Colchester: University of Essex.
3. Berry, K. J., & Mielke, P. W. (1988). A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters. Educational and Psychological Measurement, 48, 921-933.
4. Brennan, R. L., & Prediger, D. J. (1981). Coefficient kappa: some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687-699.
5. Brenner, H., & Kliebsch, U. (1996). Dependence of weighted kappa coefficients on the number of categories. Epidemiology, 7, 199-202.
6. Cicchetti, D., & Allison, T. (1971). A new procedure for assessing reliability of scoring EEG sleep recordings. The American Journal of EEG Technology, 11, 101-109.
7. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 213-220.
8. Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213-220.
9. Conger, A. J. (1980). Integration and generalization of kappas for multiple raters. Psychological Bulletin, 88, 322-328.
10. Davies, M., & Fleiss, J. L. (1982). Measuring agreement for multinomial data. Biometrics, 38, 1047-1051.
11. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382.
12. Fleiss, J. L. (1981). Statistical methods for rates and proportions. New York: Wiley.
13. Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613-619.
14. Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72, 323-327.
15. Heuvelmans, A. P. J. M., & Sanders, P. F. (1993). Beoordelaarsovereenstemming. In Eggen, T. J. H. M., & Sanders, P. F. (Eds.) Psychometrie in de Praktijk (pp. 443-470). Arnhem: Cito Instituut voor Toestontwikkeling.
16. Holmquist, N. S., McMahon, C. A., & Williams, E. O. (1968). Variability in classification of carcinoma in situ of the uterine cervix. Obstetrical & Gynecological Survey, 23, 580-585.
17. n, Cohen's kappa, Scott's π and Aickin's α. Understanding Statistics, 2, 205-219.
18. Hubert, L. (1977). Kappa revisited. Psychological Bulletin, 84, 289-297.
19. Jakobsson, U., & Westergren, A. (2005). Statistical methods for assessing agreement for ordinal data. Scandinavian Journal of Caring Sciences, 19, 427-431.
20. Janson, H., & Olsson, U. (2001). A measure of agreement for interval or nominal multivariate observations. Educational and Psychological Measurement, 61, 277-289.
21. Kraemer, H. C. (1979). Ramifications of a population model for κ as a coefficient of reliability. Psychometrika, 44, 461-472.
22. Kraemer, H. C., Periyakoil, V. S., & Noda, A. (2004). Tutorial in biostatistics: kappa coefficients in medical research. Statistics in Medicine, 21, 2109-2129.
23. Krippendorff, K. (2004). Reliability in content analysis: some common misconceptions and recommendations. Human Communication Research, 30, 411-433.
24. Kundel, H. L., & Polansky, M. (2003). Measurement of observer agreement. Radiology, 288, 303-308.
25. Landis, J. R., & Koch, G. G. (1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33, 363-374.
26. Mielke, P. W., & Berry, K. J. (2009). A note on Cohen's weighted kappa coefficient of agreement with linear weights. Statistical Methodology, 6, 439-446.
27. Mielke, P. W., Berry, K. J., & Johnston, J. E. (2007). The exact variance of weighted kappa with multiple raters. Psychological Reports, 101, 655-660.
28. Mielke, P. W., Berry, K. J., & Johnston, J. E. (2008). Resampling probability values for weighted kappa with multiple raters. Psychological Reports, 102, 606-613.
29. Nelson, J. C., & Pepe, M. S. (2000). Statistical description of interrater variability in ordinal ratings. Statistical Methods in Medical Research, 9, 475-496.
30. Popping, R. (1983). Overeenstemmingsmaten voor Nominale Data. Unpublished doctoral dissertation, Rijksuniversiteit Groningen, Groningen.
31. Popping, R. (2010). Some views on agreement to be used in content analysis studies. Quality & Quantity, 44, 1067-1078.
32. Schouten, H. J. A. (1986). Nominal scale agreement among observers. Psychometrika, 51, 453-466.
33. Schuster, C. (2004). A note on the interpretation of weighted kappa and its relations to other rater agreement statistics for metric scales. Educational and Psychological Measurement, 64, 243-253.
34. Scott, W. A. (1955). Reliability of content analysis: the case of nominal scale coding. Public Opinion Quarterly, 19, 321-325.
35. Vanbelle, S., & Albert, A. (2009a). Agreement between two independent groups of raters. Psychometrika, 74, 477-491.
36. Vanbelle, S., & Albert, A. (2009b). Agreement between an isolated rater and a group of raters. Statistica Neerlandica, 63, 82-100.
37. Vanbelle, S., & Albert, A. (2009c). A note on the linearly weighted kappa coefficient for ordinal scales. Statistical Methodology, 6, 157-163.
38. Visser, H., & de Nijs, T. (2006). The map comparison kit. Environmental Modelling & Software, 21, 346-358.
39. Warrens, M. J. (2008a). On similarity coefficients for 2×2 tables and correction for chance. Psychometrika, 73, 487-502.
40. Warrens, M. J. (2008b). On the equivalence of Cohen's kappa and the Hubert-Arabie adjusted Rand index. Journal of Classification, 25, 177-183.
41. Warrens, M. J. (2009). k-adic similarity coefficients for binary (presence/absence) data. Journal of Classification, 26, 227-245.
42. Warrens, M. J. (2010a). Inequalities between kappa and kappa-like statistics for k×k tables. Psychometrika, 75, 176-185.
43. Warrens, M. J. (2010b). Cohen's kappa can always be increased and decreased by combining categories. Statistical Methodology, 7, 673-677.
44. Warrens, M. J. (2010c). A Kraemer-type rescaling that transforms the odds ratio into the weighted kappa coefficient. Psychometrika, 75, 328-330.
45. Warrens, M. J. (2010d). A formal proof of a paradox associated with Cohen's kappa. Journal of Classification, 27, 322-332.
46. Warrens, M. J. (2010e). Inequalities between multi-rater kappas. Advances in Data Analysis and Classification, 4, 271-286.
47. Warrens, M. J. (2011). Weighted kappa is higher than Cohen's kappa for tridiagonal agreement tables. Statistical Methodology, 4, 271-286.
48. Zwick, R. (1988). Another look at interrater agreement. Psychological Bulletin, 103, 374-378.