Interrater reliability in unbalanced designs: A comment on Scholte et al. (2009)

Bulletin of the Menninger Clinic

Volumn 74, Issue 3, 2010, Pages 238-244

  Polak, Marike ^a   Abraham, Robert E ^b   Van, Henricus L ^c   Ingenhoven, Theo ^d  

^a ERASMUS UNIVERSITY ROTTERDAM   (Netherlands)

^b LEIDEN UNIVERSITY   (Netherlands)

^c JellinekMentrum   (Netherlands)

^d Centers for Mental Healthcare   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

CORRELATION COEFFICIENT; INTERRATER RELIABILITY; INTERVIEW; NOTE; PSYCHODYNAMICS; SCORING SYSTEM; VIDEOTAPE;

HUMANS; INTERVIEW, PSYCHOLOGICAL; OBSERVER VARIATION; PERSONALITY ASSESSMENT; PSYCHOANALYTIC THERAPY; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS; VIDEO RECORDING;

EID: 77957834607     PISSN: 00259284     EISSN: 00259284     Source Type: Journal    
DOI: 10.1521/bumc.2010.74.3.238     Document Type: Note

References (14)

1. Altaye, M., Donner, A., & Eliasziw, M. (2001). A general goodness-of-fit approach for inference procedures concerning the kappa statistic. Statistics in Medicine, 20, 2479-2488.
2. Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43, 551-558. (Pubitemid 20178544)
3. Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology, 43, 543-549. (Pubitemid 20178543)
4. Flack, V. F., Afifi, A. A., Lachenbruch, P.A., & Schouten, H. J. A. (1988). Sample size determinations for the two rater kappa statistic. Psychometrika, 53, 321-325.
5. Fleiss, J. L., & Cohen, J. A. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613-619.
6. rd ed.). NJ: Wiley.
7. Hripcsak, G., & Heitjan, D. F. (2002). Measuring agreement in medical informatics reliability studies. Journal of Biomedical Informatics, 35, 99-110. (Pubitemid 35352172)
8. Kottner, J. (2009). Interrater reliability and the kappa statistic: A comment on Morris et al. (2008). International Journal of Nursing Studies, 46, 141-142.
9. Morris, R., MacNeela, P., Scott, A. Treacy, P. Hyde, A. O'Brien, J., et al. (2008). Ambiguities and conflicting results: The limitations of the kappa statistic in establishing the interrater reliability of the Irish nursing minimum data set for mental health: A discussion paper. International Journal of Nursing Studies, 45, 645-647.
10. Scholte, W.R., Tiemens, B.G., Verheul, R., Meerman, A., & Hutschemakers, G. (2009). The reliability of the Developmental Profile based on video rating. Bulletin of the Menninger Clinic, 73(3), 226-238.
11. Tammemagi, M. C., Frank, J. W., Leblanc, M., Artsob, H., & Streiner, D. L. (1995). Methodological issues in assessing reproducibility - A comparative study of various indices of reproducibility applied to repeat elisa serologic tests for Lyme-disease. Journal of Clinical Epidemiology, 48, 1123-1132.
12. Van, H. L., Ingenhoven, T. J. M., van Foeken I., van 't Spijker A., Spinhoven, P., & Abraham, R. E. (2000). The Developmental Profile: Preliminary results on interrater reliability and construct validity. Journal of Personality Disorders, 14(4), 360-365.
13. Van H. L., Polak, M. G., Abraham, R. E., Overeem-Seldenrijk, J., & van Keulen, A. (2005). De interbeoordelaarsbetrouwbaarheid van het Ontwikkelingsprofiel [The interrater reliability of the Developmental Profile]. In R.E. Abraham (Ed.), The Developmental Profile (pp. 267-274). Assen, The Netherlands: Van Gorcum.
14. Whitehurst, G. J. (1984). Interrater agreement for journal manuscript reviews. American Psychologist, 39, 22-28.