Agreement between two independent groups of raters

Psychometrika

Volumn 74, Issue 3, 2009, Pages 477-491

  Vanbelle, Sophie ^a   Albert, Adelin ^a  

^a UNIVERSITY OF LIÈGE   (Belgium)

Author keywords

Agreement; Kappa coefficient; Nominal scale

Indexed keywords

EID: 84893466166     PISSN: 00333123     EISSN: None     Source Type: Journal    
DOI: 10.1007/S11336-009-9116-1     Document Type: Article

References (21)

1. Barnhart, H.X., & Williamson, J.M. (2002). Weighted least squares approach for comparing correlated kappa. Biometrics, 58, 1012-1019.
2. Bland, A.C., Kreiter, C.D., & Gordon, J.A. (2005). The psychometric properties of five scoring methods applied to the Script Concordance Test. Academic Medicine, 80, 395-399.
3. Charlin, B., Gagnon, R., Sibert, L., & Van der Vleuten, C. (2002). Le test de concordance de script: un instrument d'évaluation du raisonnement clinique. Pédagogie Médicale, 3, 135-144.
4. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
5. Cohen, J. (1968). Weighted kappa: nominal scale agreement with provision for scaled disagreement of partial credit. Psychological Bulletin, 70, 213-220.
6. Efron, B., & Tibshirani, R.J. (1993). An introduction to the bootstrap. New York: Chapman and Hall.
7. Feigin, P.D., & Alvo, M. (1986). Intergroup diversity and concordance for ranking data: an approach via metrics for permutations. The Annals of Statistics, 14, 691-707.
8. Fleiss, J.L. (1981). Statistical methods for rates and proportions (2nd ed.) New York: Wiley.
9. Fleiss, J.L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measure of reliability. Educational and Psychological Measurement, 33, 613-619.
10. Hollander, M., & Sethuraman, J. (1978). Testing for agreement between two groups of judges. Biometrika, 65, 403-411.
11. Kraemer, H.C. (1979). Ramifications of a population model for κ as a coefficient of reliability. Psychometrika, 44, 461-472.
12. Kraemer, H.C. (1981). Intergroup concordance: definition and estimation. Biometrika, 68, 641-646.
13. Kraemer, H.C., Vyjeyanthi, S.P., & Noda, A. (2004). Agreement statistics. In D'Agostino, R.B. (Ed.), Tutorial in Bio-statistics (vol. 1, pp. 85-105). New York: Wiley.
14. Lipsitz, S.R., Williamson, J., Klar, N., Ibrahim, J., & Parzen, M. (2001). A simple method for estimating a regression model for κ between a pair of raters. Journal of the Royal Statistical Society Series A, 164, 449-465.
15. Raine, R., Sanderson, C., Hutchings, A., Carter, S., Larking, K., & Black, N. (2004). An experimental study of determinants of group judgments in clinical guideline development. Lancet, 364, 429-437.
16. Schouten, H.J.A. (1982). Measuring pairwise interobserver agreement when all subjects are judged by the same observers. Statistica Neerlandica, 36, 45-61.
17. Schucany, W.R., & Frawley, W.H. (1973). A rank test for two group concordance. Psychometrika, 38, 249-258.
18. van Hoeij, M.J., Haarhuis, J.C., Wierstra, R.F., & van Beukelen, P. (2004). Developing a classification tool based on Bloom's taxonomy to assess the cognitive level of short essay questions. Journal of Veterinary Medical Education, 31, 261-267.
19. Vanbelle, S., Massart, V., Giet, G., & Albert, A. (2007). Test de concordance de script: un nouveau mode d'établissement des scores limitant l'effet du hasard. Pédagogie Médicale, 8, 71-81.
20. Vanbelle, S., & Albert, A. (2009). Agreement between an isolated rater and a group of raters. Statistica Neerlandica, 63, 82-100.
21. Williamson, J.M., Lipsitz, S.R., & Manatunga, A.K. (2000). Modeling kappa for measuring dependent categorical agreement data. Biostatistics, 1, 191-202.