Computing inter-rater reliability and its variance in the presence of high agreement

British Journal of Mathematical and Statistical Psychology

Volumn 61, Issue 1, 2008, Pages 29-48

  Gwet, Kilem Li ^a  

^a STATAXIS Consulting   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANALYSIS OF VARIANCE; ARTICLE; HUMAN; MONTE CARLO METHOD; OBSERVER VARIATION; PROBABILITY; PSYCHOLOGIC TEST; PSYCHOMETRY; REPRODUCIBILITY; STATISTICS;

ANALYSIS OF VARIANCE; HUMANS; MONTE CARLO METHOD; OBSERVER VARIATION; PROBABILITY; PSYCHOLOGICAL TESTS; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS;

EID: 44349101004     PISSN: 00071102     EISSN: None     Source Type: Journal    
DOI: 10.1348/000711006X126600     Document Type: Article

References (18)

1. Agresti, A. (2002). Categorical data analysis (2nd ed.). John Wiley & Sons, Inc., Hoboken, New Jersey, USA.
2. Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999). Beyond kappa: A review of interrater agreement measures. Canadian Journal of Statistics, 27, 3-23.
3. Bishop, Y. V. V., Fienberg, S. E., & Holland, P. W. (1975). Discrete multivariate analysis. Cambridge, MA: MIT Press.
4. Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43, 551-558.
5. Cochran, W. C. (1977). Sampling techniques (3rd ed.). New York: Wiley.
6. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
7. Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213-220.
8. Conger, A. J. (1980). Integration and generalization of kappas for multiple raters. Psychological Bulletin, 88, 322-328.
9. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382.
10. Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72, 323-327.
11. Holley, J. W., & Guilford, J. P. (1964). A note on the G index of agreement. Educational and Psychological Measurement, 24, 749-753.
12. Hubert, L. (1977). Kappa revisited. Psychological Bulletin, 84, 289-297.
13. Kraemer, H. C. (1980). Ramifications of a population model for κ as a coefficient of reliability. Psychometrika, 44, 461-472.
14. Landis, R. J., & Koch, G. G. (1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33, 363-374.
15. Light, R. J. (1971). Measures of response agreement for qualitative data: Some generalizations and alternatives. Psychological Bulletin, 76, 365-377.
16. Quenouille, M. H. (1949). Approximate tests of correlation in times series. Journal of the Royal Statistical Society, B, 11, 68-84.
17. Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, XIX, 321-325.
18. Tukey, J. W. (1958). Bias and confidence in not quite large samples. Annals of Mathematical Statistics, 29, 614.