The Effect of the Raters' Marginal Distributions on Their Matched Agreement: A Rescaling Framework for Interpreting Kappa

Multivariate Behavioral Research

Volumn 48, Issue 6, 2013, Pages 923-952

  Karelitz, Tzur M ^a   Budescu, David V ^b  

^a National Institute for Testing and Evaluation Jerusalem   (Israel)

^b FORDHAM UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84890691291     PISSN: 00273171     EISSN: None     Source Type: Journal    
DOI: 10.1080/00273171.2013.830064     Document Type: Article

References (47)

1. Agresti, A. 1992. Analysis of ordinal paired comparison data. Applied Statistics, 41: 287 - 297.
2. Agresti, A. 2002. Categorical data analysis New York, NY: Wiley.
3. Beasley, W. H. and Rodgers, J. L. 2009. " Re-sampling methods. ". In Sage handbook of quantitative methods in psychology Edited by: Millsap, R. and Maydeu Olivares, A. 362 - 386. London, UK: Sage.
4. Bennett, E. M., Alpert, R. and Goldstein, A. C. 1954. Communications through limited response questioning. Public Opinion Quarterly, 18: 303 - 308.
5. Brennan, P. and Silman, A. 1992. Statistical methods for assessing observer variability in clinical measures. British Medical Journal, 304: 1491 - 1494.
6. Brennan, R. L. and Prediger, D. J. 1981. Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41: 687 - 699.
7. Broomell, S. and Budescu, D. V. 2009. Why are experts correlated? Decomposing correlations between judges. Psychometrika, 74: 531 - 553.
8. Budescu, D. V. 1980. Some new measures of profile dissimilarity. Applied Psychological Measurement, 4: 261 - 272.
9. Byrt, T., Bishop, J. and Carlin, J. B. 1993. Bias, prevalence, and kappa. Journal of Clinical Epidemiology, 46: 423 - 429.
10. Carroll, J. B. 1961. The nature of the data, or how to choose a correlation coefficient. Psychometrika, 26: 347 - 372.
11. Cha, S. H. 2007. Comprehensive survey of distance/similarity measures between probability density functions. International Journal of Mathematical Models and Methods in Applied Sciences, 1: 300 - 307.
12. Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20: 37 - 46.
13. Collis, G. M. 1985. Kappa, measures of marginal symmetry and intraclass correlations. Educational and Psychological Measurement, 45: 55 - 62.
14. Cook, R. 1998. " Kappa and its dependence on marginal rates. ". In Encyclopedia of Biostatistics, Vol. 3, 2166 - 2168. Chichester, UK: Wiley.
15. Cureton, E. E. 1959. Note on phi/phi(max). Psychometrika, 24: 89 - 91.
16. Dates, B. and King, J. E. SPSS algorithms for bootstrapping and jackknifing generalized measures of agreement. Paper presented at the annual meeting of the Southwest Educational Research Association. New Orleans, LA.
17. Davenport, E. C. and El-Sanhurry, N. A. 1991. Phi/phimax: Review and synthesis. Educational and Psychological Measurement, 51: 821 - 828.
18. Dunn, G. 1989. Design and analysis of reliability studies New York, NY: Oxford University Press.
19. Efron, B. 1982. The jackknife, the bootstrap, and other resampling plans. Philadelphia, PA: Society for Industrial and Applied Mathematics.
20. Efron, B. and Tibshirani, R. J. 1993. An introduction to the bootstrap New York, NY: Chapman & Hall.
21. Feinstein, A. R. and Cicchetti, D. V. 1990. High agreement but low kappa: I. The problems of the two paradoxes. Journal of Clinical Epidemiology, 43: 543 - 549.
22. Fienberg, S. E. 2000. Contingency tables and log-linear models: Basic results and new developments. Journal of the American Statistical Association, 95: 643 - 647.
23. Fleiss, J. L. and Cicchetti, D. V. 1978. Inference about weighted kappa in the non-null case. Applied Psychological Measurement, 2: 113 - 117.
24. Fleiss, J. L., Cohen, J. and Everitt, B. S. 1969. Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72: 323 - 327.
25. Fleiss, J. L., Levin, B. and Paik, M. C. 2003. Statistical methods for rates and proportions, 3rd ed. Hoboken, NJ: Wiley-Interscience.
26. Goodman, G. D. and Kruskal, W. H. 1959. Measures of association for cross classifications II: Further discussion and references. Journal of the American Statistical Association, 54: 123 - 163.
27. Hershey, J. C. and Schoemaker, P. J. H. 1980. Prospect theory's reflection hypothesis: A critical examination. Organizational Behavior and Human Performance, 25: 395 - 418.
28. Kraemer, H. C. 1979. Ramifications of a population model for κ as a confident of reliability. Psychometrika, 44: 461 - 472.
29. Landis, J. R. and Koch, G. G. 1977. The measurement of observer agreement for categorical data. Biometrics, 33: 159 - 174.
30. Lee, J. and Fung, K. P. 1993. Confidence interval of the kappa coefficient by bootstrap resampling. Psychiatry Research, 49: 97 - 98.
31. Maclure, M. and Willett, W. C. 1987. Misinterpretation and misuse of the kappa statistic. American Journal of Epidemiology, 126: 161 - 169.
32. Reichenheim, M. E. 2004. Confidence intervals for the kappa statistic. The Stata Journal, 4: 421 - 428.
33. Scott, W. 1955. Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19: 321 - 325.
34. Shoukri, M. M. 2003. Measures of interobserver agreement London, UK: Chapman & Hall.
35. Shrout, P. E., Spitzer, R. L. and Fleiss, J. L. 1987. Quantification of agreement in psychiatric diagnosis revisited. Archive of General Psychiatry, 44: 172 - 177.
36. Simon, P. 2006. Including omission mistakes in the calculation of Cohen's kappa and an analysis of the coefficient's paradox features. Educational and Psychological Measurement, 66: 765 - 777.
37. Vach, W. 2005. The dependence of Cohen's kappa on the prevalence does not matter. Journal of Clinical Epidemiology, 58: 655 - 661.
38. Vierkant, R. A SAS macro for calculating bootstrapped confidence intervals about a kappa coefficient. SAS Users Group International Online Proceedings. Retrieved from http://www2.sas.com/proceedings/sugi22/STATS/PAPER295.PDF
39. von Eye, A. and Mun, E. Y. 2005. Analyzing rater agreement: Manifest variable models Mahwah, NJ: Erlbaum.
40. von Eye, A. and von Eye, M. 2008. On the marginal dependency of Cohen's κ. European Psychologist, 13: 305 - 315.
41. Warrens, M. J. 2008a. On similarity coefficients for 2 × 2 tables and correction for chance. Psychometrika, 73: 487 - 502.
42. Warrens, M. J. 2008b. On association coefficients for 2 × 2 tables and properties that do not depend on the marginal distributions. Psychometrika, 73: 777 - 789.
43. Warrens, M. J. 2010a. Inequalities between kappa and kappa-like statistics for k × k tables. Psychometrika, 75: 176 - 185.
44. Warrens, M. J. 2010b. A formal proof of a paradox associated with Cohen's kappa. Journal of Classification, 27: 322 - 332.
45. Warrens, M. J. 2011. Cohen's kappa is a weighted average. Statistical Methodology, 8: 473 - 484.
46. Wilkinson, R., Schaalje, G. and Collings, B. 2000. Confidence intervals for the kappa parameter, with application to the semiconductor industry. Communications in Statistics: Simulation and Computation, 29: 647 - 665.
47. Zwick, R. 1988. Another look at interrater agreement. Psychological Bulletin, 103: 374 - 378.