Assumptions behind intercoder reliability indices

Communication Yearbook 36

Volumn , Issue , 2012, Pages 419-480

  Zhao, Xinshu ^a,b   Liu, Jun S ^c   Deng, Ke ^c  

^a FUDAN UNIVERSITY   (China)

^b HONG KONG BAPTIST UNIVERSITY   (Hong Kong)

^c HARVARD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85178051193     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Chapter

References (51)

1. Benini, R. (1901). Principii di Demongraphia: Manuali Barbera Di Scienze Giuridiche Sociali e Politiche (No. 29) [Principles of demographics (Barbera Manuals of Jurisprudence and Social Policy)]. Firenze, Italy: G. Barbera.
2. Bennett, E. M., Alpert, R., & Goldstein, A. C. (1954). Communication through limited response questioning. Public Opinion Quarterly, 18, 303-308.
3. Brennan, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687-699.
4. Byrt, T., Bishop, J., & Carlin, J. B. (1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46, 423-429.
5. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
6. Cohen, J. (1968). Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213-220.
7. Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43, 543-549.
8. Fleiss, J. L. (1975). Measuring agreement between two judges on the presence or absence of a trait. Biometrics, 31, 651-659.
9. Glander, T. (2000). Origins of mass communications research during the American cold war: Educational effects and contemporary implications. Mahwah, NJ: Erlbaum.
10. Goodman, L. A., & Kruskal, W. H. (1954). Measures of association for cross classification. Journal of the American Statistical Association, 49, 732-764.
11. Grove, W. M., Andreasen, N. C., McDonald-Scott, P., Keller, M. B., & Shapiro, R. W. (1981). Reliability studies of psychiatric diagnosis: Theory and practice. Archives of General Psychiatry, 38, 408-413.
12. Guilford, J. P. (1961, November). Preparation of item scores for correlation between individuals in a Q factor analysis. Paper presented at the annual convention of the Society of Multivariate Experimental Psychologists.
13. Guttman, L. (1946). The test-retest reliability of qualitative data. Psychometrika, 11, 81-95.
14. Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61, 29-48.
15. Gwet, K. L. (2010). Handbook of inter-rater reliability: The defi nitive guide to measuring the extent of agreement among multiple raters (2nd ed.). Gaithersburg, MD: Advanced Analytics.
16. Hayes, A. F. (2009). Beyond Baron and Kenny: Statistical mediation analysis in the new millennium. Communication Monographs, 76, 408-420.
17. Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1, 77-89.
18. Holley, W., & Guilford, J. P. (1964). A note on the G-index of agreement. Educational and Psychological Measurement, 24, 749-753.
19. Holsti, O. R. (1969). Content analysis for the social sciences and humanities. Reading, MA: Addison-Wesley.
20. Jason, S., & Vegelius, J. (1979). On generalizations of the G index and the phi coefficient to nominal scales. Multivariate Behavioral Research, 14, 255-269.
21. Kraemer, H. C. (1979). Ramifications of a population model for kappa as a coefficient of reliability. Psychometrika, 44, 461-472.
22. Krippendorff, K. (1970). Bivariate agreement coefficients for reliability of data. Sociological Methodology, 2, 139-150.
23. Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Newbury Park, CA: Sage.
24. Krippendorff, K. (2004a). Content analysis: An introduction to its methodology (2nd ed.). Thousand Oaks, CA: Sage.
25. Krippendorff, K. (2004b). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30, 411-433.
26. Krippendorff, K. (2007). Computing Krippendorff's alpha reliability. Philadelphia, PA: University of Pennsylvania Scholarly Commons. Retrieved from http://reposi- tory.upenn.edu/asc_papers/43
27. Lasswell, H. D. (1948). The structure and function of communication in society. In L. Bryson (Ed.), The communication of ideas (pp. 215-228). New York: The Institute for Religious and Social Studies.
28. Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002). Content analysis in mass communication research: An assessment and reporting of intercoder reliability. Human Communication Research, 28, 587-604.
29. Maxwell, A. E. (1977). Coefficients of agreement between observers and their interpretation. British Journal of Psychiatry, 130, 79-83.
30. Neuendorf, K. (2002). The content analysis guidebook. Thousand Oaks, CA: Sage.
31. Osgood, C. E. (1959). The representational model and relevant research methods. In I. de Sola Pool (Ed.), Trends in content analysis (pp. 33-88). Urbana: University of Illinois Press.
32. Perreault, W. D., & Leigh, L. E. (1989). Reliability of nominal data based on qualitative judgments. Journal of Marketing Research, 26, 135-148.
33. Popping, R. (1988). On agreement indices for nominal data. In W. E. Saris & I. N. Gallhofer (Eds.), Sociometric research: Vol. 1. Data collection and scaling (pp. 90-105). New York: St. Martin's.
34. Potter, W. J., & Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis. Journal of Applied Communication Research, 27, 258-284.
35. R Development Core Team (2011). R: A language and environment for statistical computing. Vienna, Austria. Retrieved from http://www.R-project.org
36. Riffe, D., Lacy, S., & Fico, F. G. (1998). Analyzing media messages: Using quantitative content analysis in research. Mahwah, NJ: Erlbaum.
37. Riffe, D., Lacy, S., & Fico, F. G. (2005). Analyzing media messages: Using quantitative content analysis in research (2nd ed.). Mahwah, NJ: Erlbaum.
38. Rogot, E., & Goldberg I. D. (1966). A proposed index for measuring agreement in test- retest studies. Journal of Chronic Diseases, 19(9), 991-1006
39. Schramm, W. L. (1973). Men, messages, and media: A look at human communication. New York: Harper & Row.
40. Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19, 321-325.
41. Shrout, P. E., Spitzer, R. L., & Fleiss, J. L. (1987). Quantification of agreement in psychiatric diagnosis revisited. Archives of General Psychiatry, 44, 172-177.
42. Siegel, S., & Castellan, N. J. (1988). Nonparametric statistics for the behavioral sciences. New York: McGraw-Hill.
43. Spitznagel, E. L., & Helzer, J. E. (1985). A proposed solution to the base rate problem in the kappa statistic. Archives of General Psychiatry, 42, 725-728.
44. Tankard, J. W., Jr. (1988). Wilbur Schramm: Definer of a field. Journalism Educator, 43(3), 11-16.
45. Tinsley, H. E. A., & Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology, 22, 358-376.
46. Tinsley, H. E. A., & Weiss, D. J. (2000). Interrater reliability and agreement. In H. E. A. Tinsley & S. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 95-124). San Diego, CA: Academic Press.
47. Zhao, X. (2011a). When to use Cohen's k, if ever? Paper presented at the 61st annual conference of International Communication Association, Boston.
48. Zhao, X. (2011b). When to use Scott's p or Krippendorff's a, if ever? Paper presented at the annual conference of the Association for Education in Journalism and Mass Communication, St. Louis.
49. Zhao, X., Deng, K., Feng, G., Zhu, L., & Chan, V. K. C. (2012, May). Liberalconservative hierarchies for indices of inter-coder reliability. Paper presented at the 62nd annual conference of International Communication Association, Phoenix, Arizona.
50. Zhao, X, Lynch, J. G., & Chen, Q. (2010). Reconsidering Baron and Kenny: Myths and truths about mediation analysis. Journal of Consumer Research, 37, 197-206.
51. Zwick, R. (1988). Another look at inter-rater agreement. Psychological Bulletin, 103, 374-378.