Agreement Statistics: Kappa Coefficients in Medical Research

Tutorials in Biostatistics, Statistical Methods in Clinical Studies

Volumn 1, Issue , 2005, Pages 85-105

  Kraemer, Helena Chmura ^a   Periyakoil, Vyjeyanthi S ^b   Noda, Art ^a  

^a STANFORD UNIVERSITY   (United States)

^b VA PALO ALTO HEALTH CARE SYSTEM   (United States)

Author keywords

Consensus; Kappa; Reliability; Validity

Indexed keywords

CATEGORICAL VARIABLES; CONSENSUS; INTRA CLASS; KAPPA; KAPPA COEFFICIENT; MEDICAL RESEARCH; MISLEADING INFORMATIONS; VALIDITY;

RELIABILITY;

EID: 84879606023     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/0470023678.ch1c     Document Type: Chapter

References (66)

1. Fleiss JL, Nee JCM, Landis JR. Large sample variance of kappa in the case of different sets of raters. Psychological Bulletin 1979; 86:974-977.
2. Scott WA. Reliability of content analysis: the case of nominal scale coding. Public Opinion Quarterly 1955; 321-325.
3. Cohen J. A coeficient of agreement for nominal scales. Educational and Psychological Measurement 1960; 20:37-46.
4. Feinstein AR. A bibliography of publications on observer variability. Journal of Chronic Diseases 1985; 38: 619-632.
5. Banerjee M, Capozzoli M, McSweeney L, Sinha D. Beyond kappa: a review of interrater agreement measures. Canadian Journal of Statistics 1999; 27:3-23.
6. Bartko JJ, Carpenter WT. On the methods and theory of reliability. Journal of Nervous and Mental Disease 1976; 163:307-317.
7. Brennan RL, Prediger DJ. Coeficient kappa; some uses, misuses, and alternatives. Educational and Psychological Measurement 1981; 41:687-699.
8. Fleiss JL. Measuring agreement between two judges on the presence or absence of a trait. Biometrics 1975; 31:651- 659.
9. Green SB. A comparison of three indexes of agreement between observers: proportion of agreement, G-index, and kappa. Educational and Psychological Measurement 1981; 41:1069 -1072.
10. Kraemer HC, Bloch DA. Kappa coeficients in epidemiology: an appraisal of a reappaisal. Journal of Clinical Epidemiology 1988; 41:959-968.
11. Landis JR, Koch GG. A review of statistical methods in the analysis of data arising from observer reliability studies (Part I). Statistica Neerlandica 1975; 29:101-123.
12. Light RJ. Measures of response agreement for qualitative data: some generalizations and alternatives. Psychological Bulletin 1971; 76:365-377.
13. Maclure M, Willett WC. Misinterpretation and misuse of the Kappa statistic. American Journal of Epidemiology 1987; 126:161-169.
14. Shrout PE. Measurement reliability and agreement in psychiatry. Statistical Methods in Medical Research 1998; 7:301-317.
15. Thompson WD, Walter SD. A reappraisal of the kappa coeficient. Journal of Clinical Epidemiology 1988; 41:949-958.
16. Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. Journal of Clinical Epidemiology 1993; 46:423-429.
17. Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology 1990; 43:543 -549.
18. Hoehler FK. Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity. Journal of Clinical Epidemiology 2000; 53:499 -503.
19. Lantz CA, Nebenzahl E. Behavior and interpretation of the k statistic: resolution of the two paradoxes. Journal of Clinical Epidemiology 1996; 49:431- 434.
20. May SM. Modelling observer agreement-an alternative to kappa. Journal of Clinical Epidemiology 1994; 47:1315 -1324.
21. Fleiss JL. Measuring nominal scale agreement among many raters. Psychological Bulletin 1971; 76:378-382.
22. Kraemer HC. Extensions of the kappa coeficient. Biometrics 1980; 36:207 -216.
23. Kupper LL. On assessing interrater agreement for multiple attribute responses. Biometrics 1989; 45:957-967.
24. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159-174.
25. Janes CL. Agreement measurement and the judgment process. Journal of Nervous and Mental Disease 1979; 167:343-347.
26. Spitznagel EL, Helzer JE. A proposed solution to the base rate problem in the kappa statistic. Archives of General Psychiatry 1985; 42:725-728.
27. Cicchetti DV, Heavens RJ. A computer program for determining the significance of the difference betweeen pairs of independently derived values of kappa or weighted kappa. Educational and Psychological Measurement 1981; 41:189 -193.
28. Donner A, Eliasziw M. Sample size requirements for reliability studies. Statistics in Medicine 1987; 6: 441-448.
29. Donner A, Eliasziw M, Klar N. Testing the homogeneity of kappa statistics. Biometrics 1996; 52:176 -183.
30. Donner A. Sample size requirements for the comparison of two or more coeficients of inter-observer agreement. Statistics in Medicine 1998; 17:1157-1168.
31. Kraemer HC. Ramifications of a population model for k as a coeficient of reliability. Psychometrika 1979; 44:461- 472.
32. Aickin M. Maximum likelihood estimation of agreeement in the constant predictive probability model, and its relation to Cohen's kappa. Biometrics 1990; 46:293-302.
33. Maxwell AE. Coeficients of agreement between observers and their interpretations. British Journal of Psychiatry 1977; 130:79-83.
34. Kraemer HC. Estimating false alarms and missed events from interobserver agreement: comment on Kaye. Psychological Bulletin 1982; 92:749-754.
35. Cronbach LJ, Gleser GC, Nanda H, Rajaratnam J. The Dependability of Behavioral Measurements. Wiley: New York, 1972.
36. Lord FM, Novick MR. Statistical Theories of Mental Test Scores. Addison-Wesley: Reading, MA, 1968.
37. Kraemer HC. Measurement of reliability for categorical data in medical research. Statistical Methods in Medical Research 1992; 1:183 -199.
38. Carey G, Gottesman II. Reliability and validity in binary ratings. Archives of General Psychiatry 1978; 35: 1454-1459.
39. Huynh H. Reliability of multiple classifications. Psychometrika 1978; 43:317-325.
40. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 1968; 70:213-229.
41. Darroch JN, McCloud PI. Category distinguishability and observer agreement. Australian Journal of Statistics 1986; 28:371- 388.
42. Donner A, Eliasziw MA. A hierarchical approach to inferences concerning interobserver agreement for multinomial data. Statistics in Medicine 1997; 16:1097 -1106.
43. Fleiss JL. Statistical Methods For Rates and Proportions. Wiley: New York, 1981.
44. Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation coeficient as measures of reliability. Educational and Psychological Measurement 1973; 33:613-619.
45. Fleiss JL, Davies M. Jackknifing functions of multinomial frequencies, with an application to a measure of concordance. American Journal of Epidemiology 1982; 115:841- 845.
46. Donner A, Eliasziw M. A goodness-of-fit approach to inference procedures for the kappa statistics: confidence interval construction, significance-testing and sample size estimation. Statistics in Medicine 1992; 11:1511-1519.
47. Bloch DA, Kraemer HC. 2x2 kappa coeficients: measures of agreement or association. Biometrics 1989; 45:269 -287.
48. Landis JR, Koch GG. A one-way components of variance model for categorical data. Biometrics 1977; 33: 671-679.
49. Hanley JA. Standard error of the kappa statistic. Psychological Bulletin 1987; 102:315-321.
50. Fisher RA. Statistical Methods for Research Workers, 2nd edn. Oliver & Boyd: London, 1928.
51. Efron B. Bootstrap methods: another look at the jackknife. Annals of Statistics 1979; 7:1-26.
52. Flack VF. Confidence intervals for the interrater agreement measure kappa. Communications in Statistics- Theory and Methods 1987; 16:953-968.
53. Fleiss JL, Cicchetti DV. Inference about weighted kappa in the non-null case. Applied Psychological Measurement 1978; 2:113 -117.
54. Blackman NJ-N, Koval JJ. Interval estimation for Cohen's kappa as a measure of agreement. Statistics in Medicine 2000; 19:723-741.
55. Borenstein M. Hypothesis testing and effect size estimation in clinical trials. Annals of Allergy, Asthma, and Immunology 1997; 78:5 -16.
56. Kraemer HC. Evaluating Medical Tests: Objective and Quantitative Guidelines. Sage Publications: Newbury Park, CA, 1992.
57. Collis GM. Kappa, measures of marginal symmetry and intraclass correlations. Educational and Psychological Measurement 1985; 45:55-62.
58. Kraemer HC, Kazdin AE, Offord DR, Kessler RC, Jensen PS, Kupfer DJ. Measuring the potency of a risk factor for clinical or policy significance. Psychological Methods 1999; 4:257-271.
59. Ross DC. Testing patterned hypotheses in multi-way contingency tables using weighted kappa and weighted chi square. Educational and Psychological Measurement 1977; 37:291- 307.
60. Cohen J. The cost of dichotomization. Applied Psychological Measurement 1983; 7:249 -253.
61. Breiman L, Friedman JH, Olshen RA, Stone CJ. Classification and Regression Trees. Wadsworth & Brooks/Cole Advanced Books & Software: Monterey, CA, 1984.
62. Noda AM, Kraemer HC, Yesavage JA, Periyakoil VS. How many raters are needed for a reliable diagnosis? International Journal of Methods in Psychiatric Research 2001; 10:119 -125.
63. Kraemer HC. How many raters? Toward the most reliable diagnostic consensus. Statistics in Medicine 1992; 11:317-331.
64. Kraemer HC. What is the 'right' statistical measure of twin concordance (or diagnostic reliability and validity)? Archives of General Psychiatry 1997; 54:1121-1124.
65. Kraemer HC. Statistical issues in assessing comorbidity. Statistics in Medicine 1995; 14:721-733.
66. Tanner MA, Young MA. Modeling agreement among raters. Journal of the American Statistical Association 1985; 80:175-180.