Assessing inter-rater reliability when the raters are fixed: Two concepts and two estimates

Biometrical Journal

Volumn 53, Issue 3, 2011, Pages 477-490

  Rousson, Valentin ^a  

^a LAUSANNE UNIVERSITY HOSPITAL   (Switzerland)

Author keywords

Inter rater reliability; Intraclass correlation; Measurement error; Population of raters; Rater bias

Indexed keywords

BALANCED DESIGN; INTERRATER RELIABILITY; INTRACLASS CORRELATIONS; POPULATION OF RATER; RATER BIAS; RELIABILITY DATA; STATISTIC MODELING;

RELIABILITY;

BIOMETRY; COMPUTER SIMULATION; CONFIDENCE INTERVAL; HUMAN; PROCEDURES; REPRODUCIBILITY; STATISTICAL MODEL;

BIOMETRY; COMPUTER SIMULATION; CONFIDENCE INTERVALS; HUMANS; MODELS, STATISTICAL; REPRODUCIBILITY OF RESULTS;

EID: 79959614015     PISSN: 03233847     EISSN: 15214036     Source Type: Journal    
DOI: 10.1002/bimj.201000066     Document Type: Article

References (20)

1. Barnhart, H. X., Haber, M. J. and Lin, L. I. (2007). An overview on assessing agreement with continuous measurements. Journal of Biopharmaceutical Statistics 17, 529-569.
2. Bartko, J. J. (1966). The intraclass correlation as a measure of reliability. Psychological Reports 19, 3-11.
3. Campbell, M. J., Donner, A. and Klar, N. (2007). Developments in cluster randomized trials and Statistics in Medicine. Statistics in Medicine 26, 2-19.
4. Carrasco, J. L. and Jover, L. (2003). Estimating the generalized concordance correlation coefficient through variance components. Biometrics 59, 849-858.
5. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychychological Measurement 20, 37-46.
6. Cronbach, L. J (1951). Coefficient alpha and the internal structure of tests. Psychometrika 16, 297-334.
7. Dolezal, K. K., Burdick, R. K. and Birch, N. J. (1998). Analysis of a two-factors R & R study with fixed operators. Journal of Quality Technology 30, 163-170.
8. Donner, A. (1986). A review of inference procedures for the intraclass correlation coefficient in the one-way random effects model. International Statistical Review 54, 67-82.
9. Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver & Boyd, Edinburgh.
10. Fleiss, J. L. and Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement 33, 613-619.
11. Largo, R., Caflisch, J., Hug, F., Muggli, K. and Molinari, L. (2001). Neuromotor development from 5 to 18 years. Part II: Associated movements. Developmental Medicine and Child Neurology 43, 444-453.
12. Lee, J., Koh, D. and Ong, C. N. (1989). Statistical evaluation of agreement between two methods for measuring a quantitative variable. Computers in Biology and Medicine 19, 61-70.
13. Lin, L. I. K. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics 45, 255-268.
14. Lin, L. I. K., Hedayat, A. S. and Wu, W. (2007). A unified approach for assessing agreement for continuous and categorical data. Journal of Biopharmaceutical Statistics 17, 629-652.
15. Pearson, K. (1901). Mathematical distributions to the theory of evolution. Philosophical Transactions of the Royal Society of London, Series A 197, 385-497.
16. Rousson, V., Gasser, Th. and Seifert, B. (2002). Assessing intrarater, interrater and test-retest reliability of continuous measurements. Statistics in Medicine 21, 3431-3446.
17. Rousson, V., Gasser, Th. and Seifert, B. (2003). Confidence intervals for intraclass correlation in inter-rater reliability. Scandinavian Journal of Statistics 30, 617-624.
18. Shrout, P. E. and Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin 36, 420-428.
19. Tian, L. and Cappelleri, J. C. (2004). A new approach for interval estimation and hypothesis testing of a certain intraclass correlation: the generalized variable method. Statistics in Medicine 23, 2125-2135.
20. Ukoumunne, O. C. (2002). A comparison of confidence interval methods for the intraclass correlation coefficient in cluster randomized trials. Statistics in Medicine 21, 3757-3774.