Equivalences of weighted kappas for multiple raters

Statistical Methodology

Volumn 9, Issue 3, 2012, Pages 407-422

  Warrens, Matthijs J ^a  

^a LEIDEN UNIVERSITY   (Netherlands)

Author keywords

Cohen's kappa; Cohen's weighted kappa; G agreement; Hubert's kappa; Inter rater reliability; Mielke, Berry and Johnston's weighted kappa; Multiple raters; Ordinal agreement

Indexed keywords

EID: 84855201526     PISSN: 15723127     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.stamet.2011.11.001     Document Type: Article

References (57)

1. Abraira V., Pérez de Vargas A. Generalization of the kappa coefficient for ordinal categorical data, multiple observers and incomplete designs. QÜESTIIÓ 1999, 23:561-571.
2. Banerjee M., Capozzoli M., McSweeney L., Sinha D. Beyond kappa: A review of interrater agreement measures. Canadian Journal of Statistics 1999, 27:3-23.
3. Berry K.J., Mielke P.W. A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters. Educational and Psychological Measurement 1988, 48:921-933.
4. Berry K.J., Johnston J.E., Mielke P.W. Weighted kappa for multiple raters. Perceptual and Motor Skills 2008, 107:837-848.
5. Brennan R.L., Prediger D.J. Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement 1981, 41:687-699.
6. Brenner H., Kliebsch U. Dependence of weighted kappa coefficients on the number of categories. Epidemiology 1996, 7:199-202.
7. Cicchetti D., Allison T. A new procedure for assessing reliability of scoring EEG sleep recordings. The American Journal of EEG Technology 1971, 11:101-109.
8. Cicchetti D., Bronen R., Spencer S., Haut S., Berg A., Oliver P., Tyrer P. Rating scales, scales of measurement, issues of reliability, Resolving some critical issues for clinicians and researchers. The Journal of Nervous and Mental Disease 2006, 194:557-564.
9. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960, 20:213-220.
10. Cohen J. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 1968, 70:213-220.
11. Conger A.J. Integration and generalization of kappas for multiple raters. Psychological Bulletin 1980, 88:322-328.
12. Crewson P.E. Fundamentals of clinical research for radiologists, Reader agreement studies. American Journal of Roentgenology 2005, 184:1391-1397.
13. Davies M., Fleiss J.L. Measuring agreement for multinomial data. Biometrics 1982, 38:1047-1051.
14. Fleiss J.L. Statistical Methods for Rates and Proportions 1981, Wiley, New York.
15. Fleiss J.L., Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement 1973, 33:613-619.
16. Gower J.C., De Rooij M. A comparison of the multidimensional scaling of triadic and dyadic distances. Journal of Classification 2003, 20:115-136.
17. Graham P., Jackson R. The analysis of ordinal agreement data: Beyond weighted kappa. Journal of Clinical Epidemiology 1993, 46:1055-1062.
18. Heiser W.J., Bennani M. Triadic distance models: Axiomatization and least squares representation. Journal of Mathematical Psychology 1997, 41:189-206.
19. A.P.J.M. Heuvelmans, P.F. Sanders, Beoordelaarsovereenstemming, in: T.J.H.M. Eggen, P.F. Sanders (Eds.), Psychometrie in de Praktijk, Arnhem: Cito Instituut voor Toestontwikkeling, 1993, pp. 443-470.
20. Holmquist N.D., McMahan C.A., Williams E.O. Variability in classification of carcinoma in situ of the uterine cervix. Obstetrical & Gynecological Survey 1967, 23:580-585.
21. n, Cohen's kappa, Scott's π and Aickin's α. Understanding Statistics 2003, 2:205-219.
22. Hubert L. Kappa revisited. Psychological Bulletin 1977, 84:289-297.
23. Janson H., Olsson U. A measure of agreement for interval or nominal multivariate observations. Educational and Psychological Measurement 2001, 61:277-289.
24. Kraemer H.C. Ramifications of a population model for κ as a coefficient of reliability. Psychometrika 1979, 44:461-472.
25. Kraemer H.C., Periyakoil V.S., Noda A. Tutorial in biostatistics: Kappa coefficients in medical research. Statistics in Medicine 2004, 21:2109-2129.
26. Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics 1977, 33:159-174.
27. Landis J.R., Koch G.G. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics 1977, 33:363-374.
28. Light R.J. Measures of response agreement for qualitative data: Some generalizations and alternatives. Psychological Bulletin 1971, 76:365-377.
29. Maclure M., Willett W.C. Misinterpretation and misuse of the kappa statistic. American Journal of Epidemiology 1987, 126:161-169.
30. Mielke P.W., Berry K.J. A note on Cohen's weighted kappa coefficient of agreement with linear weights. Statistical Methodology 2009, 6:439-446.
31. Mielke P.W., Berry K.J., Johnston J.E. The exact variance of weighted kappa with multiple raters. Psychological Reports 2007, 101:655-660.
32. Mielke P.W., Berry K.J., Johnston J.E. Resampling probability values for weighted kappa with multiple raters. Psychological Reports 2008, 102:606-613.
33. Nelson J.C., Pepe M.S. Statistical description of interrater variability in ordinal ratings. Statistical Methods in Medical Research 2000, 9:475-496.
34. R. Popping, Overeenstemmingsmaten Voor Nominale Data, Ph.D. Thesis, Rijksuniversiteit Groningen, Groningen, 1983.
35. Popping R. Some views on agreement to be used in content analysis studies. Quality & Quantity 2010, 44:1067-1078.
36. Schouten H.J.A. Nominal scale agreement among observers. Psychometrika 1986, 51:453-466.
37. Schuster C. A note on the interpretation of weighted kappa and its relations to other rater agreement statistics for metric scales. Educational and Psychological Measurement 2004, 64:243-253.
38. Schuster C., Smith D.A. Dispersion-weighted kappa: An integrative framework for metric and nominal scale agreement coefficients. Psychometrika 2005, 70:135-146.
39. Vanbelle S., Albert A. Agreement between two independent groups of raters. Psychometrika 2009, 74:477-491.
40. Vanbelle S., Albert A. Agreement between an isolated rater and a group of raters. Statistica Neerlandica 2009, 63:82-100.
41. Vanbelle S., Albert A. A note on the linearly weighted kappa coefficient for ordinal scales. Statistical Methodology 2009, 6:157-163.
42. Von Eye A., Mun E.Y. Analyzing Rater Agreement. Manifest Variable Methods 2006, Lawrence Erlbaum Associates.
43. Warrens M.J. On the equivalence of Cohen's kappa and the Hubert-Arabie adjusted rand index. Journal of Classification 2008, 25:177-183.
44. Warrens M.J. On similarity coefficients for 2×2 tables and correction for chance. Psychometrika 2008, 73:487-502.
45. Warrens M.J. On multi-way metricity, minimality and diagonal planes. Advances in Data Analysis and Classification 2008, 2:109-119.
46. Warrens M.J. k-Adic similarity coefficients for binary (presence/absence) data. Journal of Classification 2009, 26:227-245.
47. Warrens M.J. Inequalities between kappa and kappa-like statistics for k×k tables. Psychometrika 2010, 75:176-185.
48. Warrens M.J. Cohen's kappa can always be increased and decreased by combining categories. Statistical Methodology 2010, 7:673-677.
49. Warrens M.J. A formal proof of a paradox associated with Cohen's kappa. Journal of Classification 2010, 27:322-332.
50. Warrens M.J. Inequalities between multi-rater kappas. Advances in Data Analysis and Classification 2010, 4:271-286.
51. Warrens M.J. n-Way metrics. Journal of Classification 2010, 27:173-190.
52. Warrens M.J. A family of multi-rater kappas that can always be increased and decreased by combining categories. Statistical Methodology 2012, 9(3):330-340.
53. Warrens M.J. Cohen's linearly weighted kappa is a weighted average of 2×2 kappas. Psychometrika 2011, 76:471-486.
54. Warrens M.J. Weighted kappa is higher than Cohen's kappa for tridiagonal agreement tables. Statistical Methodology 2011, 8:268-272.
55. M.J. Warrens, Cohen's linearly weighted kappa is a weighted average, Advances in Data Analysis and Classification (2011) (in press).
56. M.J. Warrens, Some paradoxical results for the quadratically weighted kappa, Psychometrika (2011) (in press).
57. Zwick R. Another look at interrater agreement. Psychological Bulletin 1988, 103:374-378.