Variance estimation of nominal-scale inter-rater reliability with random selection of raters

Psychometrika

Volumn 73, Issue 3, 2008, Pages 407-430

  Gwet, Kilem Li ^a  

^a STATAXIS Consulting   (United States)

Author keywords

AC 1 coefficient; Agreement coefficient; Inter rater reliability; Kappa statistic

Indexed keywords

EID: 52949085846     PISSN: 00333123     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11336-007-9054-8     Document Type: Article

References (36)

1. E. Bartfay A. Donner 2001 Statistical inferences for inter-observer agreement studies with nominal outcome data The Statistician 50 135 146
2. E.M. Bennet R. Alpert A.C. Goldstein 1954 Communications through limited response questioning Public Opinion Quarterly 18 303 308
3. K.J. Berry P.W. Mielke Jr. 1988 A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters Educational and Psychological Measurement 48 921 933
4. R.L. Brennan D.J. Prediger 1981 Coefficient kappa: Some uses, misuses, and alternatives Educational and Psychological Measurement 41 687 699
5. T. Byrt J. Bishop J.B. Carlin 1993 Bias, prevalence and kappa Journal of Clinical Epidemiology 46 423 429
6. D.V. Cicchetti A.R. Feinstein 1990 High agreement but low kappa: II. Resolving the paradoxes Journal of Clinical Epidemiology 43 551 558
7. Cochran, W.G. (1977). Sampling techniques (3rd ed.). New York: Wiley.
8. J. Cohen 1960 A coefficient of agreement for nominal scales Educational and Psychological Measurement 20 37 46
9. A.J. Conger 1980 Integration and generalization of kappas for multiple raters Psychological Bulletin 88 322 328
10. Cook, R.J. (1998). Kappa and its dependence on marginal rates. In P. Armitage & T. Colton (Eds.), Encyclopedia of biostatistics (pp. 2166-2168). New York: Wiley.
11. A. Donner M. Eliasziw 1992 A goodness-of-fit approach to inference procedures for the kappa statistic: Confidence interval construction, significance-testing and sample size estimation Statistics in Medicine 11 1511 1519
12. A.R. Feinstein D.V. Cicchetti 1990 High agreement but low kappa: I. The problems of two paradoxes Journal of Clinical Epidemiology 43 543 549
13. J.L. Fleiss 1971 Measuring nominal scale agreement among many raters Psychological Bulletin 76 378 382
14. Fuller, W.A., & Isaki, C.T. (1981). Survey design under superpopulation models. In D. Krewski, J.N.K. Rao, & R. Platek (Eds.), Current topics in survey sampling (pp. 199-226). New York: Academic Press.
15. L.A. Goodman W.H. Kruskal 1954 Measures of association for cross classifications Journal of the American Statistical Association 49 1732 1769
16. Gwet, K. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1).
17. J.W. Holley J.P. Guilford 1964 A note on the G index of agreement Educational and Psychological Measurement 24 749 753
18. C.T. Isaki W.A. Fuller 1982 Survey design under the regression superpopulation model Journal of the American Statistical Association 77 89 96
19. H. Janson U. Olsson 2001 A measure of agreement for interval or nominal multivariate observations Educational and Psychological Measurement 61 277 289
20. H. Janson U. Olsson 2004 A measure of agreement for interval or nominal multivariate observations by different sets of judges Educational and Psychological Measurement 64 62 70
21. S. Janson J. Vegelius 1979 On generalizations of the G index and the PHI coefficient to nominal scales Multivariate Behavioral Research 14 255 269
22. H.C. Kraemer V.S. Periyakoil A. Noda 2002 Kappa coefficients in medical research Statistics in Medicine 21 2109 2129
23. R.J. Landis G.G. Koch 1977 An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers Biometrics 33 363 374
24. R.J. Light 1971 Measures of response agreement for qualitative data: Some generalizations and alternatives Psychological Bulletin 76 365 377
25. A.E. Maxwell 1977 Coefficients of agreement between observers and their interpretation British Journal of Psychiatry 130 79 83
26. K.O. McGraw S.P. Wong 1996 Forming inferences about some intraclass correlation coefficients Psychological Methods 1 30 46
27. J.M. Nam 2000 Interval estimation of the kappa coefficient with binary classification and an equal marginal probability model Biometrics 56 583 585
28. Rao, C.R. (2002). Wiley series in probability and statistics. Linear statistical inference and its applications (2nd ed.).
29. C. Schuster 2004 A note on the interpretation of weighted kappa and its relations to other rater agreement statistics for metric scales Educational and Psychological Measurement 64 243 253
30. C. Schuster D.A. Smith 2006 Estimating with a latent class model the reliability of nominal judgments upon which two raters agree Educational and Psychological Measurement 66 739 747
31. W.A. Scott 1955 Reliability of content analysis: the case of nominal scale coding Public Opinion Quarterly XIX 321 325
32. P. Simon 2006 Including omission mistakes in the calculation of Cohen's kappa and an analysis of the coefficient's paradox features Educational and Psychological Measurement 66 765 777
33. J.S. Uebersax W.M. Grove 1990 Latent class analysis of diagnostic agreement Statistics in Medicine 9 559 572
34. J.S. Uebersax W.M. Grove 1993 A latent trait finite mixture analysis of rating agreement Biometrics 49 823 835
35. G. Zou N. Klar 2005 A non-iterative confidence interval estimating procedure for the intraclass kappa statistic with multinomial outcomes Biometrical Journal 5 682 690
36. R. Zwick 1988 Another look at interrater agreement Psychological Bulletin 103 374 378