Interrater agreement statistics with skewed data: Evaluation of alternatives to Cohen's kappa

Journal of Consulting and Clinical Psychology

Volumn 82, Issue 6, 2014, Pages 1219-1227

  Xu, Shu ^a   Lorber, Michael F ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

Behavior observation; Diagnosis; Interrater agreement; Low base rate; Skew

Indexed keywords

ARTICLE; BEHAVIORAL SCIENCE; CLINICAL RESEARCH; COHEN KAPPA; HUMAN; INTERRATER AGREEMENT STATISTICS; INTERRATER RELIABILITY; KAPPA STATISTICS; MONTE CARLO METHOD; SIMULATION; OBSERVER VARIATION; REPRODUCIBILITY; STATISTICS;

HUMANS; OBSERVER VARIATION; REPRODUCIBILITY OF RESULTS; STATISTICS AS TOPIC;

EID: 84925639509     PISSN: 0022006X     EISSN: 19392117     Source Type: Journal    
DOI: 10.1037/a0037489     Document Type: Article

References (31)

1. Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis. Cambridge, United Kingdom: Cambridge University Press. doi:10.1017/CBO97805115
2. Bakeman, R., McArthur, D., Quera, V., & Robinson, B. F. (1997). Detecting sequential patterns and determining their reliability with fallible observers. Psychological Methods, 2, 357-370. doi:10.1037/1082-989X .2.4.357
3. Burton, A., Altman, D. G., Royston, P., & Holder, R. L. (2006). The design of simulation studies in medical statistics. Statistics in Medicine, 25, 4279-4292. doi:10.1002/sim.2673
4. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284-290. doi:10.1037/1040-3590.6.4.284
5. Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43, 551-558. doi:10.1016/0895-4356(90)90159-M
6. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46. doi:10.1177/ 00131644600
7. Cohen, J. (1968). Weighed kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213-220. doi:10.1037/h0026256
8. Feng, G. C. (2013). Factors affecting intercoder reliability: A Monte Carlo experiment. Quality & Quantity, 47, 2959-2982. doi:10.1007/s11135-012-9745-9
9. Few, L. R., Miller, J. D., Rothbaum, A. O., Meller, S., Maples, J., Terry, D. P., MacKillop, J. (2013). Examination of the Section III DSM-5
10. diagnostic system for personality disorders in an outpatient clinical sample. Journal of Abnormal Psychology, 122, 1057-1069. doi:10.1037/ a0034878
11. Gisev, N., Bell, J. S., & Chen, T. F. (2013). Interrater agreement and interrater reliability: Key concepts, approaches, and applications. Research in Social and Administrative Pharmacy, 9, 330-338. doi: 10.1016/j.sapharm.2012.04.004
12. Gwet, K. (2002). Kappa statistic is not satisfactory for assessing the extent of agreement between raters. Retrieved from http://www.agreestat.com/ research-papers/kappa-statistic-is-not-satisfactory.pdf
13. Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61, 29-48. doi:10.1348/000711006X1
14. Gwet, K. L. (2012). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among multiple raters (3rd ed.). Gaithersburg, MD: Advanced Analytics Press
15. Holley, J. W., & Guilford, J. P. (1964). A note on the G index of agreement. Educational and Psychological Measurement, 24, 749-753. doi:10.1177/00131644640
16. Karelitz, T. M., & Budescu, D. V. (2013). The effect of the raters' marginal distributions on their matched agreement: A rescaling framework for interpreting kappa. Multivariate Behavioral Research, 48, 923-952. doi:10.1080/00273171.2013.830064
17. Kazdin, A. E. (2003). Research design in clinical psychology (4th ed.). Boston, MA: Allyn & Bacon
18. Krippendorff, K. (2011). Agreement and information in the reliability of coding. Retrieved from http://repository.upenn.edu/asc-papers/278
19. Landis, J. R., & Koch, G. G. (1975a). A review of statistical methods in the analysis of data arising from observer reliability studies (Part I). Statistica Neerlandica, 29, 101-123. doi:10.1111/j.1467-9574.1975.tb00254.x
20. Landis, J. R., & Koch, G. G. (1975b). A review of statistical methods in the analysis of data arising from observer reliability studies (Part II). Statistica Neerlandica, 29, 151-161. doi:10.1111/j.1467-9574.1975.tb00259.x
21. Landis, J. R., & Koch, G. G. (1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33, 363-374. doi:10.2307/2529786
22. Lorber, M. F. (2006). Can minimally trained observers provide valid global ratings? Journal of Family Psychology, 20, 335-338. doi:10.1037/0893-3200.20.2.335
23. MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19-40. doi:10.1037/1082-989X.7.1.19
24. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46. doi:10.1037/1082-989X.1.1.30
25. Mitchell, S. (1979). Interobserver agreement, reliability, and generalizability of data collected in observational studies. Psychological Bulletin, 86, 376-390. doi:10.1037/0033-2909.86.2.376
26. Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19, 321-325. doi:10.1086/ 266577
27. Simon, P. (2006). Including omission mistakes in the calculation of Cohen's kappa and an analysis of the coefficient's paradox features. Educational and Psychological Measurement, 66, 765-777. doi:10.1177/ 00131644052
28. Slep, A. M. S., & Heyman, R. E. (2006). Creating and field-testing child maltreatment definitions: Improving the reliability of substantiation determinations. Child Maltreatment, 11, 217-236. doi:10.1177/ 10775595062
29. Spitznagel, E. L., & Helzer, J. E. (1985). A proposed solution to the base rate problem in the kappa statistic. Archives of General Psychiatry, 42, 725-728. doi:10.1001/archpsyc.1985.01790300093
30. Uebersax, J. S. (1987). Diversity of decision-making models and the measurement of interrater agreement. Psychological Bulletin, 101, 140-146. doi:10.1037/0033-2909.101.1.140
31. Vach, W. (2005). The dependence of Cohen's kappa on the prevalence does not matter. Journal of Clinical Epidemiology, 58, 655-661. doi: 10.1016/j.jclinepi.2004.02.021