Measuring inter-rater reliability for nominal data - Which coefficients and confidence intervals are appropriate?

BMC Medical Research Methodology

Volumn 16, Issue 1, 2016, Pages

  Zapf, Antonia ^a   Castell, Stefanie ^b   Morawietz, Lars ^c   Karch, André ^b,d  

^a UNIVERSITY MEDICAL CENTER GÖTTINGEN   (Germany)

^b HELMHOLTZ CENTRE FOR INFECTION RESEARCH   (Germany)

^c Ernst von Bergmann Hospital   (Germany)

^d German Center for Cardiovascular Research   (Germany)

Author keywords

Bootstrap; Confidence interval; Fleiss' K; Fleiss' kappa; Inter rater heterogeneity; Krippendorff's alpha

Indexed keywords

ALGORITHM; BREAST NEOPLASMS; CONFIDENCE INTERVAL; FEMALE; HUMAN; OBSERVER VARIATION; REPRODUCIBILITY; RETROSPECTIVE STUDY; STATISTICAL ANALYSIS;

ALGORITHMS; BREAST NEOPLASMS; CONFIDENCE INTERVALS; DATA INTERPRETATION, STATISTICAL; FEMALE; HUMANS; OBSERVER VARIATION; REPRODUCIBILITY OF RESULTS; RETROSPECTIVE STUDIES;

EID: 84982852143     PISSN: None     EISSN: 14712288     Source Type: Journal    
DOI: 10.1186/s12874-016-0200-9     Document Type: Article

References (38)

1. Gwet KL. Handbook of Inter-Rater Reliability. 3rd ed. USA: Advanced Analytics, LLC; 2012.
2. Michels KB. A renaissance for measurement error. Int J Epidemiol. 2001;30(3):421-2.
3. Roger VL, Boerwinkle E, Crapo JD, et al. Strategic transformation of population studies: recommendations of the working group on epidemiology and population sciences from the National Heart, Lung, and Blood Advisory Council and Board of External Experts. Am J Epidemiol. 2015;181(6):363-8.
4. Scott WA. Reliability of content analysis: the case of nominal scale coding. Public Opinion Quarterly. 1955;XIX:321-5.
5. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37-46.
6. Hallgren KA. Computing inter-rater reliability for observational data:-an overview and tutorial. Tutor Quant Methods Psychol. 2012;8(1):23-4.
7. Thompson WD, Walter SD. A reappraisal of the kappa coefficient. J Clin Epidemiol. 1988;41(10):949-58.
8. Kraemer HC, Bloch DA. Kappa coefficients in epidemiology: an appraisal of a reappraisal. J Clin Epidemiol. 1988;41(10):959-68.
9. Feinstein A, Cicchetti D. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43(6):543-9.
10. Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971;76:378-82.
11. Siegel S, Castellan Jr NJ. Nonparametric Statistics for the Behavioral Sciences. 2nd ed. New York: McGraw-Hill; 1988.
12. Krippendorff K. Estimating the reliability, systematic error, and random error of interval data. Educ Psychol Meas. 1970;30:61-70.
13. von Elm E, Altman D, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP, for the STROBE Initiative. The strenghtening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61:344-9.
14. Schulz KF, Altman DG, Moher D, for the CONSORT group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. J Clin Epidemiol. 2010;63:834-40.
15. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HCW, for the STARD steering group. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Br Med J. 2003;326:41-4.
16. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74.
17. Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychol Bull. 1969;72(5):323-7.
18. Fleiss JL, Nee JCM, Landis JR. Large sample variance of kappa in the case of different sets of raters. Psychol Bull. 1979;86(5):974-7.
19. Klar N, Lipsitz SR, Parzen M, Leong T. An exact bootstrap confidence interval for k in small samples. J Royal Stat Soc Series D (The Statistician). 2002;51(4):467-78.
20. Häußler S. Evaluation of reader heterogeneity in diagnostic trials. Master thesis at the University of Bremen, study course: Medical Biometry / Biostatistics; 2010.
21. Fleiss JL, Levin B, Paik MC. Statistical Methods for Rates and Proportions. 3rd ed. Hoboken: Wiley; 2003.
22. Rao CR. Linear Statistical Inference and its Applications. 2nd ed. New York: Wiley; 1973.
23. Efron B. Six questions raised by the bootstrap. Exploring the limits of bootstrap. In: LePage R, Billard L, editors. Technical Report No. 139. New York: Division of Biostatistics, Stanford University. Wiley; 1992.
24. Fleiss JL, Davies M. Jackknifing functions of multinomial frequencies, with an application to a measure of concordance. Am J Epidemiol. 1982;115:841-5.
25. McKenzie DP, Mackinnon AJ, Péladeau N, Onghena P, Bruce PC, Clarke DM, Harrigan S, McGorry PD. Comparing correlated kappas by resampling: is one level of agreement significantly different from another? J Psychiatr Res. 1996;30(6):483-92.
26. Vanbelle S, Albert A. A bootstrap method for comparing correlated kappa coefficients. J Stat Comput Simulation. 2008;78(11):1009-15.
27. Krippendorff. Content analysis, an introduction to its methodology. SAGE Publications, 2nd Edition; 2004.
28. cα. Version from 2015.06.17. Obtained by personal communication with Klaus Krippendorff; 2015.
29. Hayes A, Krippendorff K. Answering the call for a standard reliability measure for coding data. Comm Methods Measures. 2007;1(1):77-89.
30. Hayes A. http://www.afhayes.com/spss-sas-and-mplus-macros-and-code.html. Accessed 4 Aug 2016.
31. Gamer M, Lemon J, Singh P. Various coefficients of interrater reliability and agreement. Package 'irr', version 0.84; 2015.
32. MacPherson P, Choko AT, Webb EL, Thindwa D, Squire SB, Sambakunsi R, van Oosterhout JJ, Chunda T, Chavula K, Makombe SD, Lalloo DG, Corbett EL. Development and validation of a global positioning system-based "map book" system for categorizing cluster residency status of community members living in high-density urban slums in Blantyre. Malawi Am J Epidemiol. 2013;177(10):1143-7.
33. Devine A, Taylor SJ, Spencer A, Diaz-Ordaz K, Eldrige S, Underwood M. The agreement between proxy and self-completed EQ-5D for care home residents was better for index scores than individual domains. J Clin Epidemiol. 2014;67(9):1035-43.
34. Johnson SR, Tomlinson GA, Hawker GA, Granton JT, Grosbein HA, Feldmann BM. A valid and reliable belief elicitation method for Bayesian priors. J Clin Epidemiol. 2010;63(4):370-83.
35. Hoy D, Brooks P, Woolf A, Blyth F, March L, Bain C, Baker P, Smith E, Buchbinder R. Assessing risk of bias in prevalence studies: modification of an existing tool and evidence of interrater agreement. J Clin Epidemiol. 2012;65(9):934-9.
36. Krippendorff K. Commentars: A dissenting view on so-called paradoxes of reliability coefficients. In: Salmond CT, editor. Communication Yearbook 36. New York: Routledge; 2012. chapter 20: 481-499.
37. Schuster C. A mixture model approach to indexing rater agreement. Br J Math Stat Psychol. 2002;55(2):289-303.
38. Schuster C, Smith DA. Indexing systematic rater agreement with a latent-class model. Psychol Methods. 2002;7(3):384-95.