Assessing the inter-rater agreement for ordinal data through weighted indexes

Statistical Methods in Medical Research

Volumn 25, Issue 6, 2016, Pages 2611-2633

  Marasini, Donata ^a   Quatto, Piero ^a   Ripamonti, Enrico ^a  

^a UNIVERSITY OF MILANO BICOCCA   (Italy)

Author keywords

Fleiss' kappa; inter rater agreement; multiple observers; ordinal variables; weighted indexes

Indexed keywords

ARTICLE; BOOTSTRAPPING; CLASSIFICATION; CONFIDENCE INTERVAL; DATA ANALYSIS; HUMAN; HYPOTHESIS; INTERRATER RELIABILITY; KAPPA STATISTICS; MONTE CARLO METHOD; STATISTICAL ANALYSIS; STATISTICAL DISTRIBUTION; UTERINE CERVIX CARCINOMA IN SITU; CARCINOMA IN SITU; FEMALE; OBSERVER VARIATION; PATHOLOGY; REPRODUCIBILITY; UTERINE CERVIX TUMOR;

CARCINOMA IN SITU; FEMALE; HUMANS; MONTE CARLO METHOD; OBSERVER VARIATION; REPRODUCIBILITY OF RESULTS; UTERINE CERVICAL NEOPLASMS;

EID: 84995812845     PISSN: 09622802     EISSN: 14770334     Source Type: Journal    
DOI: 10.1177/0962280214529560     Document Type: Article

References (45)

1. Nelson JC, Pepe MS,. Statistical description of interrater variability in ordinal ratings. Stat Meth Med Res 2000; 9: 475-496.
2. Lawton MP, Brody EM,. Assessment of older people: self-maintaining and instrumental activities of daily living. Gerontologist 1969; 9: 179-186.
3. Van, Swieten JC, Koudstaal PJ, Visser MC, et al. Interobserver agreement for the assessment of handicap in stroke patients. Stroke 1988; 19: 604-607.
4. Lin KL,. A concordance correlation coefficient to evaluate reproducibility. Biometrics 1989; 45: 255-268.
5. Chinchilli VM, Martel JK, Kumanyika S, et al. A weighted concordance correlation coefficient for repeated measurement designs. Biometrics 1996; 52: 341-353.
6. Fleiss JL, Levin B, Paik MC,. Statistical methods for rates and proportions, 3rd ed. Hoboken, NJ: John Wiley & Sons, 2003.
7. Kraemer HC,. Measurement of reliability for categorical data in medical research. Stat Meth Med Res 1992; 1: 183-199.
8. Lin LH, Hedayat AS, Wu W,. Statistical tools for measuring agreement, New York: Springer, 2012.
9. Shrout PE,. Measurement reliability and agreement in psychiatry. Stat Meth Med Res 1998; 7: 301-317.
10. Gwet KL,. Handbook of inter-rater reliability: the definitive guide to measuring the extent of agreement among multiple raters, Gaithersburg, MD: Advanced Analytics Press, 2012.
11. Agresti A,. An agreement model with kappa as parameter. Stat Prob Lett 1988; 7: 271-273.
12. Feinstein AR, Cicchetti DV,. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990; 43: 543-549.
13. Landis JR, Koch GG,. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159-174.
14. Holmquist ND, McMahan CA, Williams OD, et al. Variability in classification of carcinoma in situ of the uterine cervix. Arch Pathol 1967; 84: 334-345.
15. Goodman LA, Kruskal WH,. Measures of association for cross classifications. J Am Stat Assoc 1954; 49: 732-764.
16. Cohen J,. A coefficient of agreement for nominal scales. Educ Psychol Measure 1960; 20: 37-46.
17. Banerjee M, Capozzoli M, McSweeney L, et al. Beyond kappa: A review of interrater agreement measures. Can J Stat 1999; 27: 3-23.
18. Scott WA,. Reliability of content analysis: the case of nominal scale coding. Public Opin Q 1955; 19: 321-325.
19. Everitt BS,. Moments of the statistics kappa and weighted kappa. Br J Math Stat Psychol 1968; 21: 97-103.
20. Fleiss JL, Cohen J, Everitt BS,. Large sample standard errors of kappa and weighted kappa. Psychol Bull 1969; 72: 323-327.
21. Fleiss JL,. Measuring nominal scale agreement among many raters. Psychol Bull 1971; 76: 378-382.
22. Light RJ,. Measures of response agreement for qualitative data: some generalizations and alternatives. Psychol Bull 1971; 76: 365-377.
23. Krippendorff K,. Bivariate agreement coefficients for reliability of data. Sociol Methodol 1970; 2: 139-150.
24. Fabbris L, Gallo F,. Bivariate coefficients of agreement among any number of observers. Educ Psychol Measure 1993; 53: 337-349.
25. Landis JR, Koch GG,. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics 1977; 33: 363-374.
26. Davies M, Fleiss JL,. Measuring agreement for multinomial data. Biometrics 1982; 38: 1047-1051.
27. O'Connell DL, Dobson AJ,. General observer-agreement measures on individual subjects and groups of subjects. Biometrics 1984; 40: 973-983.
28. Cohen J,. Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol Bull 1968; 70: 213-220.
29. Berry KJ, Mielke PW,. A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters. Educ Psychol Measure 1988; 48: 921-933.
30. Janson H, Olsson U,. A measure of agreement for interval or nominal multivariate observations by different sets of judges. Educ Psychol Measure 2004; 64: 62-70.
31. Maclure M, Willett WC,. Misinterpretation and misuse of the kappa statistic. Am J Epidemiol 1987; 126: 161-169.
32. Falotico R, Quatto P,. On avoiding paradoxes in assessing inter-rater agreement. Stat Appl-Ital J Appl Stat 2010; 22: 151-160.
33. Abraira V, de Vargas A,. Generalization of the Kappa coefficient for ordinal categorical data, multiple observers and incomplete design. Questiò 1999; 23: 561-571.
34. Cicchetti DV, Allison T,. A new procedure for assessing reliability of scoring EEG sleep recordings. Am J EEG Technol 1971; 11: 101-109.
35. Jakobsson U, Westergren A,. Statistical methods for assessing agreement for ordinal data. Scand J Caring Sci 2005; 19: 427-431.
36. Vanbelle S, Albert A,. A note on the linearly weighted kappa coefficient for ordinal scales. Stat Method 2009; 6: 157-163.
37. Brenner H, Kliebsch U,. Dependence of weighted kappa coefficients on the number of categories. Epidemiology 1996; 7: 199-202.
38. Krippendorff K,. Estimating the reliability, systematic error and random error of interval data. Educ Psychol Measure 1970; 30: 61-70.
39. Fleiss JL, Cohen J,. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Measure 1973; 33: 613-619.
40. Guo Y, Manatunga AK,. Measuring agreement of multivariate discrete survival times using a modified weighted kappa coefficient. Biometrics 2009; 65: 125-134.
41. DiCiccio TJ, Efron B,. Bootstrap confidence intervals. Stat Sci 1996; 11: 189-212.
42. Efron B,. Bootstrap methods: another look at the jackknife. Ann Stat 1979; 7: 1-26.
43. Schuster C,. A note on the interpretation of weighted kappa and its relations to other rater agreement statistics for metric scales. Educ Psychol Measure 2004; 64: 243-253.
44. Lehmann EL,. Elements of large-sample theory, New York: Springer, 1999.
45. Dow M,. Explicit inverses of Toeplitz and associated matrices. ANZIAM J 2008; 44: E185-E215.