A measure of agreement for interval or nominal multivariate observations by different sets of judges

Educational and Psychological Measurement

Volumn 64, Issue 1, 2004, Pages 62-70

  Janson, Harald ^a   Olsson, Ulf ^b  

^a UNIVERSITY OF OSLO   (Norway)

^b SWEDISH UNIVERSITY OF AGRICULTURAL SCIENCES   (Sweden)

Author keywords

Interrater reliability; Measurement; Multivariate analysis; Rating

Indexed keywords

EID: 1642578147     PISSN: 00131644     EISSN: None     Source Type: Journal    
DOI: 10.1177/0013164403260195     Document Type: Article

References (13)

1. Berry, K. J., & Mielke, P. W., Jr. (1988). A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters. Educational and Psychological Measurement, 48, 921-933.
2. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
3. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability of scores and profiles. New York: Wiley.
4. Efron, B., & Tibshirani, R. J. (1993). An introductions the bootstrap. London: Chapman & Hall.
5. Fabbris, L., & Gallo, F. (1993). Bivariate coefficients of agreement among any number of observers. Educational and Psychological Measurement, 53, 337-349.
6. Fleiss, J. L. (1966). Assessing the accuracy of multivariate observations. American Statistical Association Journal, 61, 403-412.
7. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382.
8. Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York: Wiley.
9. Harris, J. A. (1913). On the calculation of intraclass and interclass coefficients of correlation from class moments when the number of possible combinations is large. Biometrika, 9, 446-472.
10. Hubert, L. (1977). Kappa revisited. Psychological Bulletin, 84, 289-297.
11. Janson, H., & Olsson, U. (2001). A measure of agreement for interval or nominal multivariate observations. Educational and Psychological Measurement, 61, 277-289.
12. Landis, J. R., & Koch, G. G. (1977). A one-way components of variance model for categorical data. Biometrics, 33, 671-679.
13. Towstopiat, O. (1984). A review of reliability procedures for measuring observer agreement. Contemporary Educational Psychology, 9, 333-352.