Resampling probability values for weighted kappa with multiple raters

Psychological Reports

Volumn 102, Issue 2, 2008, Pages 606-613

  Mielke Jr , Paul W ^a   Berry, Kenneth J ^a   Johnston, Janis E ^b  

^a Veterinary Dentistry and Oral Surgery   (United States)

^b US EPA Homeland Security Research Center

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; COMPUTER ANALYSIS; CONFIDENCE INTERVAL; EPIDEMIOLOGY; HUMAN; METHODOLOGY; PEER REVIEW; PROBABILITY; PUBLICATION; SAMPLE SIZE; STATISTICAL MODEL; STATISTICS;

COMPUTING METHODOLOGIES; CONFIDENCE INTERVALS; HUMANS; MODELS, STATISTICAL; PEER REVIEW; PERIODICALS AS TOPIC; PROBABILITY; SAMPLE SIZE; SAMPLING STUDIES; STATISTICS AS TOPIC;

EID: 45149095415     PISSN: 00332941     EISSN: 1558691X     Source Type: Journal    
DOI: 10.2466/PR0.102.2.606-613     Document Type: Article

References (51)

1. AGRESTI, A., & GHOSH, A. (1995) Raking kappa: describing potential impact of marginal distributions of measures of agreement. Biomedical Journal, 37, 811-820.
2. ANDRÉS, A. M., & MARZO, P. F. (2004) Delta: a new measure of agreement between two raters. British Journal of Mathematical and Statistical Psychology, 57, 1-19.
3. ANDRÉS, A. M., & MARZO, P. F. (2005) Chance-corrected measures of reliability and validity in K x K tables. Statistical Methods in Medical Research, 14, 473-492.
4. BANERJEE, M., CAPOZZOLI, M., MCS WEENEY, L., & SINHA, D. (1999) Beyond kappa: a review of interrater agreement measures. The Canadian Journal of Statistics, 27, 3-23.
5. BENNETT, E. M., ALPERT, R., & GOLDSTEIN, A. C. (1954) Communications through limited response questioning. Public Opinion Quarterly, 18, 303-308.
6. BERRY, K. J., JOHNSTON, J. E., & MIELKE, P. W. (2005) Exact and resampling probability values for weighted kappa. Psychological Reports, 96, 243-252.
7. BERRY, K. J., & MIELKE, P. W. (1988) A generalization of Cohen's kappa agreement measure to interval measurement and multiple raters. Educational and Psychological Measurement, 48, 921-933.
8. BRENNAN, R. L., & PREDIGER, D. (1981) Coefficient kappa: some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687-699.
9. CAREY, G., & GOTTESMAN, I. I. (1978) Reliability and validity in binary ratings: areas of common misunderstanding in diagnosis and symptom ratings. Archives of General Psychiatry, 35, 1454-1459.
10. CICCHETTI, D. V. (1981) Testing the normal approximation and minimal sample size requirements of weighted kappa when the number of categories is large. Applied Psychological Measurement, 5, 101-104.
11. CICCHETTI, D. V., & FLEISS, J. L. (1977) Comparison of the null distribution of weighted kappa and the C ordinal statistic. Applied Psychological Measurement, 1, 195-201.
12. COHEN, J. (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
13. COHEN, J. (1968) Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213-220.
14. CONGER, A. J. (1980) Integration and generalization of kappa for multiple raters. Psychological Bulletin, 88, 322-328.
15. DE MAST, J. (2007) Agreement and kappa-type indices. The American Statistician, 61, 148-153.
16. EPSTEIN, D. M., DALINKA, M. K., KAPLAN, F. S., ARONCHICK, J. M., MARINELLI, D. L., & KUNDEL, H. L. (1986) Observer variation in the detection of osteopenia. Skeletal Radiology, 15, 347-349.
17. FLEISS, J. L. (1971) Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382.
18. FLEISS, J. L., COHEN, J., & EVERITT, B. S. (1969) Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72, 323-327.
19. GRAHAM, P., & JACKSON, R. (1993) The analysis of ordinal agreement data: beyond weighted kappa. Journal of Clinical Epidemiology, 46, 1055-1062.
20. HERMAN, P. G., KHAN, A., KALLMAN, C. E., ROJAS, K. A., CARMODY, D. P., & BODENHEIMER, M. M. (1990) Limited correlation of left ventricular end-diastolic pressure with radiographic assessment of pulmonary hemodynamics. Radiology, 174, 721-724.
21. HOLLEY, W., & GUILFORD, J. P. (1964) A note on the G-index of agreement. Educational and Psychological Measurement, 24, 749-753.
22. n, Cohen's kappa, Scott's π, and Aickin's α. Understanding Statistics, 2, 205-219.
23. HUBERT, L. (1977) Kappa revisited. Psychological Bulletin, 84, 289-297.
24. HUTCHINSON, T. P. (1993) Kappa muddles together two sources of disagreement: tetrachoric correlation is preferable. Research in Nursing & Health, 16, 313-315.
25. JANSON, S., & VEGELIUS, J. (1979) On generalizations of the G index and the phi coefficient to nominal scales. Multivariate Behavioral Research, 14, 255-269.
26. JOHNSTON, J. E., BERRY, K. J., & MIELKE, P. W. (2007) Permutation tests: precision in estimating probability values. Perceptual and Motor Skills, 105, 915-920.
27. KER, M. (1991) Issues in the use of kappa. Investigative Radiology, 26, 78-83.
28. KRAMER, M. S., & FEINSTEIN, A. R. (1981) Clinical biostatistics: LIV. The biostatistics of concordance. Clinical Pharmacology & Therapeutics, 29, 111-123.
29. KUNDEL, H. L., & POLANSKY, M. (2003) Measurement of observer agreement. Radiology, 228, 303-308.
30. LIGHT, R. J. (1971) Measures of response agreement for qualitative data: some generalizations and alternatives. Psychological Bulletin, 76, 365-377.
31. LUDBROOK, J. (2002) Statistical techniques for comparing measurers and methods of measurement: a critical review. Clinical and Experimental Pharmacology and Physiology, 29, 527-536.
32. MACLURE, M., & WILLETT, W. C. (1987) Misinterpretation and misuse of the kappa statistic. American Journal of Epidemiology, 126, 161-169.
33. MAXWELL, A. E. (1979) Comparing the classification of subjects by two independent judges. British Journal of Psychiatry, 116, 651-655.
34. MEYER, G. J. (1997) Assessing reliability: critical corrections for a critical examination of the Rorschach comprehensive system. Psychological Assessment, 9, 480-489.
35. MIELKE, P. W., & BERRY, K. J. (1988) Cumulant methods for analyzing independence of r-way contingency tables and goodness-of-fit frequency data. Biometrika, 75, 790-793.
36. MIELKE, P. W., BERRY, K. J., & JOHNSTON, J. E. (2007a) Resampling programs for multiway contingency tables with fixed marginal frequency totals. Psychological Reports, 101, 18-24.
37. MIELKE, P. W., BERRY, K. J., & JOHNSTON, J. E. (2007b) The exact variance of weighted kappa with multiple raters. Psychological Reports, 101, 655-660.
38. NELSON, J. C., & PEPE, M. S. (2000) Statistical description of interrater variability in ordinal ratings. Statistical Methods in Medical Research, 9, 475-496.
39. SCHOUTEN, H. J. A. (1980) Measuring pairwise agreement among many observers. Biometrical Journal, 22, 497-504.
40. SCHOUTEN, H. J. A. (1982a) Measuring pairwise agreement among many observers: II. Some improvements and additions. Biometrical Journal, 24, 431-435.
41. SCHOUTEN, H. J. A. (1982b) Measuring pairwise interobserver agreement when all subjects are judged by the same observers. Statistica Neerlandica, 36, 45-61.
42. SCHUSTER, C. (2004) A note on the interpretation of weighted kappa and its relations to other rater agreement statistics for metric scales. Educational and Psychological Measurement, 64, 243-253.
43. SCHUSTER, C., & SMITH, D. A. (2005) Dispersion-weighted kappa: an integrative framework for metric and nominal scale agreement coefficients. Psychometrika, 70, 135-146.
44. SEIGEL, D. G. PODGOR, M. J., & REMALEY, N. A. (1992) Acceptable values of kappa for comparison of two groups. American Journal of Epidemiology, 135, 571-578.
45. SHESKIN, D. J. (2004) Handbook of parametric and nonparametric statistical procedures. (3rd ed.) Boca Raton, FL: Chapman & Hall/CRC.
46. SHROUT, P. E., SPITZER, R. L., & FLEISS, J. L. (1987) Quantification of agreement in psychiatric diagnosis revisited. Archives of General Psychiatry, 44, 172-177.
47. SOEKEN, K. L., & PRESCOTT, P. A. (1986) Issues in the use of kappa to estimate reliability. Medical Care, 24, 733-741.
48. SPITZNAGEL, E. L., & HELZER, J. E. (1985) A proposed solution to the base rate problem in the kappa statistic. Archives of General Psychiatry, 42, 725-728.
49. TAPLIN, S. H., RUTTER, C. M., ELMORE, J. G., SEGER, D., WHITE, D., & BRENNER, R. J. (2000) Accuracy of screening mammography using single versus independent double interpretation. American Journal of Roentgenology, 174, 1257-1262.
50. THOMPSON, W. D., & WALTER, S. D. (1988) A reappraisal of the kappa coefficient. Journal of Clinical Epidemiology, 41, 949-958.
51. ZWICK, R. (1988) Another look at interrater agreement. Psychological Bulletin, 103, 374-378.