Inference procedures for assessing interobserver agreement among multiple raters

Biometrics

Volumn 57, Issue 2, 2001, Pages 584-588

  Altaye, Mekibib ^a   Donner, Allan ^b   Klar, Neil ^c  

^a CHILDREN S HOSPITAL OF THE KING S DAUGHTERS   (United States)

^b UNIVERSITY OF WESTERN ONTARIO   (Canada)

^c DANA FARBER CANCER INSTITUTE   (United States)

Author keywords

Confidence interval; Interobserver agreement; Multiple rater; Sample size; Type I error

Indexed keywords

BINOMIAL MODELS; CHI-SQUARE GOODNESS-OF-FIT TEST; CONFIDENCE INTERVAL; GOODNESS OF FIT; INTEROBSERVER AGREEMENT; ITEM ANALYSIS; MULTIPLE RATER; PARAMETER COMBINATION; SAMPLE SIZES; TYPE-I ERROR;

ARTICLE; HUMAN; OBSERVER VARIATION; SAMPLE; STATISTICAL MODEL; VARIANCE;

EID: 0035001797     PISSN: 0006341X     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.0006-341X.2001.00584.x     Document Type: Article

References (14)

1. Bahadur, R. R. (1961). A representation of the joint distribution of responses to n dichotomous items. In Studies in Item Analysis and Prediction: Stanford Mathematical Studies in the Social Sciences VI, H. Solomon (ed.), 158-176. Stanford, California: Stanford University Press.
2. Bloch, D. A. and Kraemer, H. C. (1989). 2 × 2 Kappa coefficients: Measure of agreement or association. Biometrics 45, 269-287.
3. Donner, A. and Eliasziw, M. (1992). A goodness-of-fit approach to inference procedures for the kappa statistics: Confidence interval construction, significant testing and sample size estimation. Statistics in Medicine 11, 1511-1519.
4. Fisher, R. A. (1958). Statistical Methods for Research Workers, 13th edition. New York: Hafner.
5. Fleiss, J. L. (1981). Statistical Methods for Rates and Proportions, 2nd edition, New York: Wiley.
6. George, E. O. and Bowman, D. (1995). A full likelihood procedure for analyzing exchangeable binary data. Biometrics 51, 512-523.
7. Hale, C. A. and Fleiss, J. L. (1993). Interval estimation under two study designs for kappa with binary classifications. Biometrics 49, 523-534.
8. Kraemer, H. C. (1992). How many raters? Toward the most reliable diagnostic consensus. Statistics in Medicine 11, 317-331.
9. Landis, J. R. and Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33, 159-179.
10. Mak, T. K. (1988). Analysing intraclass correlation for dichotomous variables. Applied Statistics 37, 344-352.
11. Melia, B. M. and Diener-West, M. (1994). Modeling interrater agreement for pathologic features of choroidal melanoma. In Case Studies in Biometry, Lang et al. (eds), Chapter 16. New York: Wiley.
12. Molenberghs, G., Fitzmaurice, G. M., and Lipsitz, S. R. (1996). Efficient estimation of the intraclass correlation for a binary trait. Journal of Agricultural, Biological, and Environmental Statistics 1, 78-96.
13. Prentice, R. L. (1988). Correlated binary regression with covariates specific to each binary observation. Biometrics 44, 1033-1048.
14. Tanner, M. A. and Young, M. A. (1985). Modelling agreement among many raters. Journal of the American Statistical Association 80, 175-180.