Rorschach interrater agreement estimates: An empirical evaluation

Scandinavian Journal of Psychology

Volumn 40, Issue 2, 1999, Pages 115-120

  Grønnerød, Cato ^a  

^a UNIVERSITY OF OSLO   (Norway)

Author keywords

Interrater agreement; Kappa; Percentage agreement; Rorschach

Indexed keywords

EID: 0000053598     PISSN: 00365564     EISSN: None     Source Type: Journal    
DOI: 10.1111/1467-9450.00107     Document Type: Article

References (38)

1. Abildsnes, A., Ringstad, A. J., Havik, O. E. & Nielsen, G. (1991). Fra kliniske tolkninger til avvik fra et gjennomsnitt - En utprøving av Killingmos Rorschach-system. En psykometrisk utprøving av en skåringsmanual for egoprofilen i Killingmos Rorschach-system. [From clinical interpretations to deviance from the mean - A testing of Killingmo's Rorschach System. A psychometric evaluation of a scoring μanual for the Ego Profile in the Killingmo Rorschach System.] Tidsskrift for Norsk Psykologforening, 28, 895-903.
2. Amundsen, C. M. & Dahle, B. (1993). En krysskulturell studie av Rorschach Comprehensive System. [A Crosscultural Study of the Rorschach Comprehensive System.] Master's thesis. Bergen: Faculty of Psychology, University of Bergen.
3. Burton, N. W. (1981). Estimating scorer agreement for nominal categorization systems. Educational and Psychological Measurement, 41, 953-962.
4. Cohen, J. (1960). A coefficient for agreement on nominal scales. Educational and Psychological Measurements, 20, 37-46.
5. Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 101, 140-146.
6. Coleman, M. J., Carpenter, J. T., Waternaux, C., Levy, D. L., Shenton, M. E., Perry, J., Medoff, D., Wong, H., Monoach, P., Meyer, P., O'Brian, C., Valentino, C., Robinson, D., Smith, M., Makowski, D. & Holzman, P. S. (1993). The Thought Disorder Index: A reliability study. Psychological Assessment, 5, 336-342.
7. Conger, A. J. (1980). Interation and generalization of kappa for multiple raters. Psychological Bulletin, 88, 322-328.
8. DeCato, C. M. (1994). Toward a training model for rorschach scoring revisited: a follow-up study on a training system for interscorer agreement. Perceptual and Motor Skills, 78, 3-10.
9. Exner, J. E. (1986). The Rorschach: A comprehensive system: Vol. 1. Basic Foundations. (2nd ed.) New York: John Wiley & Sons, Inc.
10. Exner, J. E. (1993). The Rorschach: A comprehensive system. (3rd ed.) New York: John Wiley & Sons, Inc.
11. Exner, J. E., Kinder, B. N. & Curtis, G. (1995). Reviewing basic design features. In John E. Exner (Ed.), Issues and Methods in Rorschach Research. Mahway, New Jersey: Lawrence Erlbaum Associates.
12. Ferguson, G. A. (1971). Statistical analysis in psychology and education. (3rd ed.) New York: McGraw-Hill Book Company.
13. Fleiss, J. L. (1981). Statistical methods for rates and proportions. (2nd ed.) New York: John Wiley & Sons, Inc.
14. Fleiss, J. L., Nee, J. C. M. & Landis, J. R. (1979). Large sample variance of kappa in the case of different sets of raters. Psychological Bulletin, 86, 974-977.
15. Fleiss, J. L. & Cohen, J. (1973). The equivalence of weighted kappa and intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurements, 33, 613-619.
16. Grove, W. M., Andreassen, N. C., McDonald-Scott, P., Keller, M. B. & Shapiro, R. W. (1981). Reliability studies of psychiatric diagnosis. Archives of General Psychiatry, 38, 408-413.
17. Grønnerød, C. & Krog, D. (1995). RO-95: Et revidert norsk Rorschach-skåringssystem. [RO-95: A Revised Norwegian Rorschach Scoring System.] Master's thesis. Oslo: Institute of Psychology, University of Oslo.
18. Heldal, J. (1997). Some considerations around reliability. Unpublished note, University of Oslo.
19. Hubert, L. (1977). Kappa revisited. Psychological Bulletin, 84, 289-297.
20. Huynh, H. (1978). Reliability of multiple classifications. Psychometrica, 43, 317-325.
21. Kelly, W. L. (1972). Reliability assessment of the Group Kahn Test of symbolic arrangement. International Journal of Symbology, 3(3), 22-34.
22. Kraemer, H. C. (1980). Extensions of the kappa coefficient. Biometrics, 36, 207-216.
23. Landis, J. R. & Koch, G. C. (1977a). The measurement of observer agreement on categorial data. Biometrics, 33, 159-174.
24. Landis, J. R. & Koch, G. C. (1977b). An application of kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33, 363-374.
25. Landis, J. R. & Koch, G. C. (1977c). A one-way components of variance of categorial data. Biometrics, 33, 671-679.
26. Light, R. J. (1971). Measures of response agreement for qualitative data: Some generalizations and alternatives. Psychological Bulletin, 76, 365-377.
27. McDowell, C. & Acklin, M. W. (1996). Standardizing procedures for calculating rorschach interrater reliability: Conceptual and empirical foundations. Journal of Personality Assessment, 66, 308-320.
28. Mezzich, J. E., Kraemer, H. C., Worthington, D. R. & Coffman, G. A. (1981). Assessment of agreement among several raters formulating multiple diagnoses. Journal of Psychiatric Research, 16, 29-39.
29. Nunnally, J. C. & Bernstein, I. H. (1994). Psychometric theory. (3rd ed.) New York: McGraw-Hill, Inc.
30. Sattler, J. M. (1988). Assessment of children. (3rd ed.) San Diego: Jerome M. Sattler, Publisher.
31. Shrout, P. E., Splitzer, R. L. & Fleiss, J. L. (1987). Quantification of agreement in psychiatric diagnoses revisited. Archives of General Psychiatry, 44, 172-177.
32. Shrout, P. E. & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
33. Skre, I., Onstad, S., Torgersen, S. & Kringlen, E. (1991). High interrater reliability for the Structured Clinical Interview for DSM-III-R Axis I (SCID-I). Acta Psychiatrica Scandinavica, 84, 167-173.
34. Uebersax, J. S. (1983). A design-independent method for measuring the reliability of psychiatric diagnosis. Journal of Psychiatric Research, 17, 335-342.
35. Uebersax, J. S. (1987). Diversity of decision-making models and the measurement of interrater agreement. Psychological Bulletin, 101, 140-146.
36. Weiner, I. B. (1991). Editor's Note: Interscorer agreement in Rorschach research. Journal of Personality Assessment, 56, 1.
37. Wood, J. M. & Nezworski, T. M. & Stejskal, W. J. (1996). The comprehensive system for the Rorschach: A critical examination. Psychological Science, 7, 3-10.
38. Zwick, R. (1988). Another look at interrater agreement. Psychological Bulletin, 103, 374-378.