Interrater reliability for multiple raters in clinical trials of ordinal scale

Drug Information Journal

Volumn 41, Issue 5, 2007, Pages 595-605

  Sawa, Jungo ^a,c   Morikawa, Toshihiko ^b  

^a R and D   (Japan)

^b KURUME UNIVERSITY   (Japan)

^c Shinjuku Park Tower ^*  (Japan)

Author keywords

Intraclass correlation coefficients; Multiple raters; Ordinal scale; Reliability; Weighted kappa statistics

Indexed keywords

ARTICLE; COMPARATIVE STUDY; CORRELATION COEFFICIENT; DIAGNOSTIC IMAGING; EVALUATION; INTERRATER RELIABILITY; KAPPA STATISTICS; MATHEMATICAL ANALYSIS; NUCLEAR MAGNETIC RESONANCE IMAGING; PRIORITY JOURNAL; PROBABILITY; STATISTICAL ANALYSIS; STATISTICS;

EID: 35148822891     PISSN: 00928615     EISSN: None     Source Type: Journal    
DOI: 10.1177/009286150704100506     Document Type: Article

References (17)

1. Food and Drug Administration. Guidance for Industry: Developing Medical Imaging Drugs and Biological Products; Washington, DC: Food Drug Administration; 2000.
2. Statistical Principles for Clinical Trials. ICH Harmonized Tripartite Guideline. International conference on harmonization of technical requirements for registration of pharmaceuticals for human use; 1998.
3. Davies M, Fleiss JL. Measurement agreement for multinomial data. Biometrics. 1982;38:1047-1051.
4. Schouten HJA. Measuring pairwise inter-observer agreement when all subjects are judged by the same observers. Stat Neerlandica. 1982;36:45-61.
5. Morikawa T, Sekine N. Statistical issues in applying reliability measures. Manuscript for the 7th DIA Biostatistical Workshop; 2000; Tokyo, Japan.
6. Gwet K. Handbook of Inter-rater Reliability: How to Estimate the Level of Agreement Between Two or Multiple Raters. Stataxis Publishing Company, Gaithersburg, MD; 2001.
7. Fleiss JL, Cohen J. The equivalence of weighted kappa and the intra-class correlation coefficient as measures of reliability. Educ Psychol Meas. 1973;33:613-619.
8. Shoukri MM, Pause CA. Statistical Methods for Health Sciences. 2nd ed. New York: CRC Press; 1999.
9. Bartko JJ. The intra-class correlation coefficient as a measure of reliability. Psychol Rep. 1966;19:3-11.
10. Fleiss JL. The Design and Analysis of Clinical Experiments. New York: John Wiley & Sons; 1986.
11. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70:213-220.
12. Fleiss JL, Levin B, Paik MC. Statistical Methods for Rates and Proportions. 3rd ed. New York: John Wiley & Sons; 2003.
13. SAS Institute Inc. SAS/STAT. Users Guide. Version 9.2; 2005.
14. O'Brien PC. Procedure for comparing samples with multiple endpoints. Biometrics. 1984;40:1079-1087.
15. Brown H, Prescott R. Applied Mixed Models in Medicine. New York: John Wiley & Sons; 1999.
16. Friedman M. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc. 1936;32:675-701.
17. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174.