On population-based measures of agreement for binary classifications

Canadian Journal of Statistics

Volumn 36, Issue 3, 2008, Pages 411-426

  Nelson, Kerrie P ^a   Edwards, Don ^b  

^a MASSACHUSETTS GENERAL HOSPITAL   (United States)

^b UNIVERSITY OF SOUTH CAROLINA   (United States)

Author keywords

Agreement; Cohen's kappa; Crossed random effects; Generalized linear mixed model; Model based kappa; Prevalence; Reliability; Scott's pi

Indexed keywords

EID: 59249106675     PISSN: 03195724     EISSN: None     Source Type: Journal    
DOI: 10.1002/cjs.5550360306     Document Type: Article

References (49)

1. A. Agresti (1988). A model for agreement between ratings on an ordinal scale. Biometrics, 44, 539-548.
2. W. C. Allsbrook, K. A. Mangold, M. H. Johnson, R. B. Lane, C. G. Lane & J. I. Epstein (2001). Interobserver reproducibility of Gleason grading of prostatic carcinoma: general pathologists. Human Pathology, 32, 81-88.
3. M. Banerjee, M. Capozzoli, L. McSweeney & D. Sinha (1999). Beyond kappa: a review of interrater agreement measures. The Canadian Journal of Statistics, 27, 3-23.
4. W. Barlow (1996). Measurement of interrater agreement with adjustment for covariates. Biometrics, 52, 695-702.
5. C. A. Beam, R M. Layde & D. C. Sullivan (1996). Variability in the interpretation of screening mammograms by US radiologists. Archives of Internal Medicine, 156, 209-213.
6. D. A. Bloch & H. C. Kraemer (1989). 2 × 2 kappa coefficients: Measurements of agreement and association. Biometrics, 45, 269-287.
7. R. L. Brennan & D. J. Prediger (1981). Coefficient kappa: some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687-699.
8. N. E. Breslow & D. G. Clayton (1993). Approximate inference in generalized linear mixed models. Journal of the American Statistical Association, 88, 9-25.
9. T. Byrt, J. Bishop & J. B. Carlin (1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46, 423-429.
10. V. M. Chincilli, J. K. Mattel, S. Kumanyika & T. Lloyd (1996). A weighted concordance correlation coefficient for repeated measurement designs. Biometrics, 52, 341-353.
11. J. Cohen (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
12. J. Cohen (1968). Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213-220.
13. S. S. Coughlin, L. W. Pickle, M. E. Goodman & L. R. Wilkens (1992). The logistic modeling of interobserver agreement. Journal of Clinical Epidemiology, 45, 1237-1241.
14. B. A. Coull & A. Agresti (2003). Generalized log-linear models with random effects, with application to smoothing contingency tables. Statistical Modelling, 3, 251-271.
15. A. P. Dawid & A. M. Skene (1979). Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28, 20-28.
16. J. G. Elmore, C. K. Wells, C. H. Lee, D. H. Howard & A. R. Feinstein (1994). Variability in radiologists' interpretations of mammograms. The New England Journal of Medicine, 331, 1493-1499.
17. A. R. Feinstein & D. V. Cicchetti (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43, 543-549.
18. J. L. Fleiss (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382.
19. L. A. Goodman (1979). Simple models for the analysis of association in cross classifications having ordered categories. Journal of the American Statistical Association, 74, 537-552.
20. P. Graham (1995). Modeling covariate effects in observer agreement studies: the case of nominal scale agreement. Statistics in Medicine, 14, 299-310.
21. D. F. R. Griffiths, J. Melia, L. J. McWilliam, R. Y. Ball, K. Grigor, P. Harnden, M. Jarmulowicz, R. Montironi, R. Moseley, M. Waller, S. Moss & M. C. Parkinson (2006). A study of Gleason score interpretation in different groups of UK pathologists; techniques for improving reproducibility. Histopathology, 48, 655-662.
22. N. D. Holmquist, C. A. McMahan & O. D. Williams (1967). Variability in classification of carcinoma in situ of the uterine cervix. Archives of Pathology, 84, 334-345.
23. N. Klar, S. R. Lipsitz & J. G. Ibrahim (2000). An estimating equations approach for modelling kappa. Biometrical Journal, 1, 45-58.
24. H. C. Kraemer (1979). Ramifications of a population model for k as a coefficient of reliability. Psychometrika, 44, 461-472.
25. H. C. Kraemer (1980). Extension of the kappa coefficient. Biometrics, 36, 207-216.
26. J. R. Landis & G. G. Koch (1977a). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.
27. J. R. Landis & G. G. Koch (1977b). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33, 363-374.
28. R. J. Light (1971). Measures of response agreement for qualitative data: some generalizations and alternatives. Psychological Bulletin, 76, 365-377.
29. S. R. Lipsitz & G. Fitzmaurice (1996). Estimating equations for measures of association between repeated binary responses. Biometrics, 52, 903-912.
30. S. R. Lipsitz, M. Parzen, G. M. Fitzmaurice & N. Klar (2003). A two-stage logistic regression model for analyzing inter-rater agreement. Psychometrika, 68, 289-298.
31. K.-J. Lui, W. G. Cumberland & L. Kuo (1996). An interval estimate for the intraclass correlation in beta-binomial sampling. Biometrics, 52, 412-425.
32. M. Maclure & W. C. Willett (1987). Misinterpretation and misuse of the Kappa statistic. American Journal of Epidemiology, 126, 161-169.
33. C. E. McCulloch (1997). Maximum likelihood algorithms for generalized linear mixed models. Journal of the American Statistical Association, 92, 162-170.
34. S. G. Meester & R. J. MacKay (1994). A parametric model for cluster correlated categorical data. Biometrics, 50, 954-963.
35. D. F. Moore (1987). Modelling the extraneous variance in the presence of extra-binomial variation. Applied Statistics, 36, 8-14.
36. J. C. Nelson & M. S. Pepe (2000). Statistical description of interrater variability in ordinal ratings. Statistical Methods in Medical Research, 9, 475-496.
37. J. M. Neuhaus (1992). Statistical methods for longitudinal and clustered designs with binary responses. Statistical Methods in Medical Research, 1, 249-273.
38. G. Rodriguez & I. Elo (2003). Intra-class correlation in random-effects models for binary data. The Stata Journal, 3, 32-46.
39. C. Schuster (2002). A mixture model approach to indexing rater agreement. British Journal of Mathematical and Statistical Psychology, 55, 289-303.
40. W. A. Scott (1955). Reliability of content analysis: the case of nominal scale coding. Public Opinion Quarterly, 19, 321-325.
41. M. M. Shoukri (2004). Measures of Interobserver Agreement. Chapman & Hall/CRC, London.
42. P. E. Shrout & J. L. Fleiss (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 2, 420-428.
43. M. A. Tanner & M. A. Young (1985). Modeling agreement among raters. Journal of the American Statistical Association, 80, 175-180.
44. W. D. Thompson & S. D. Walter (1988). A reappraisal of the kappa coefficient. Journal of Clinical Epidemiology, 41, 949-958.
45. J. S. Uebersax & W. M. Grove (1990). Latent class analysis of diagnostic agreement. Statistics in Medicine, 9, 559-572.
46. J. M. Williamson & A. K. Manatunga (1997). Assessing interrater agreement from dependent data. Biometrics, 54, 707-714.
47. J. M. Williamson. A. K. Manatunga & S. R. Lipsitz (2000). Modeling kappa for measuring dependent categorical agreement data. Biostatistics, 1, 191-202.
48. J. Yerushalmy (1956). The importance of observer error in the interpretation of photofluorograms and the value of multiple readings. International Tuberculosis Yearbook, 26, 110-124.
49. S. L. Zeger & K.-Y. Liang (1986). Longitudinal data analysis for discrete and continuous outcomes. Biometrics, 42, 121-130.