Content Analysis in Mass Communication: Assessment and Reporting of Intercoder Reliability

Human Communication Research

Volumn 28, Issue 4, 2002, Pages 587-604

  Lombard, Matthew ^a   Snyder Duch, Jennifer ^b   Bracken, Cheryl Campanella ^c  

^a TEMPLE UNIVERSITY   (United States)

^b Carlow College   (United States)

^c CLEVELAND STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0036409443     PISSN: 03603989     EISSN: None     Source Type: Journal    
DOI: 10.1093/hcr/28.4.587     Document Type: Review

References (39)

1. Bakeman, R. (2000). Behavioral observation and coding. In H. T. Reis & C. M. Judge (Eds.), Handbook of research methods in social and personality psychology (pp. 138-159). New York: Cambridge University Press.
2. Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999). Beyond kappa: A review of interrater agreement measures. The Canadian Journal of Statistics, 27, 3-23.
3. Bennett, E. M., Alpert, R., & Goldstein, A. C. (1954). Communications through limited response questioning. Public Opinion Quarterly, 18, 303-308.
4. Berelson, B. (1952). Content analysis in communication research. Glencoe, IL: Free Press.
5. Berry, K. J., & Mielke, P. W., Jr. (1997). Measuring the joint agreement between multiple raters and a standard. Educational and Psychological Measurement, 57, 527-530.
6. Brennan, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687-699.
7. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
8. Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement of partial credit. Psychological Bulletin, 70, 213-220.
9. Craig, R. T. (1981). Generalization of Scott's index of intercoder agreement. Public Opinion Quarterly, 45, 260-264.
10. Dewey, M. E. (1983). Coefficients of agreement. British Journal of Psychiatry, 143, 487-489.
11. Ellis, L. (1994). Research methods in the social sciences. Madison, WI: Brown & Benchmark.
12. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382.
13. Frey, L. R., Botan, C. H., & Kreps, G. L. (2000). Investigating communication: An introduction to research methods (2nd ed.). Boston: Allyn & Bacon.
14. Holsti, O. R. (1969). Content analysis for the social sciences and humanities. Reading, MA: Addison-Wesley.
15. Hughes, M. A., & Garrett, D. E. (1990). Intercoder reliability estimation - approaches in marketing: A generalizability theory framework for quantitative data. Journal of Marketing Research, 27, 185-195.
16. Kang, N., Kara, A., Laskey, H. A., & Seaton, F. B. (1993). A SAS macro for calculating intercoder agreement in content analysis. Journal of Advertising, 23, 17-28.
17. Kolbe, R. H., & Burnett, M. S. (1991). Content-analysis research: An examination of applications with directives for improving research reliability and objectivity. Journal of Consumer Research, 18, 243-250.
18. Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Beverly Hills, CA: Sage.
19. Krippendorff, K. (2001, August 22). Content #724. Message posted to the Content electronic mailing list, available from http://www.content-analysis.de/contpub.htm
20. Lacy, S., & Riffe, D. (1996). Sampling error and selecting intercoder reliability samples for nominal content categories: Sins of omission and commission in mass communication quantitative research. Journalism & Mass Communication Quarterly, 73, 969-973.
21. Lawlis, G. F., & Lu, E. (1972). Judgment of counseling process: Reliability, agreement, and error. Psychological Bulletin, 78, 17-20.
22. Lichter, S. R., Lichter, L. S., & Amundsom, D. (1997). Does Hollywood hate business or money? Journal of Communication, 47, 68-84.
23. Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45, 255-268.
24. Neuendorf, K. A. (2002). The content analysis guidebook. Thousand Oaks, CA: Sage.
25. Pasadeos, Y., Huhman, B., Standley, T., & Wilson, G. (1995, May). Applications of content analysis in news research: A critical examination. Paper presented at the annual conference of the Association for Education in Journalism and Mass Communication, Washington, D.C.
26. Perreault, W. D., & Leigh, L. E. (1989). Reliability of nominal data based on qualitative judgments. Journal of Marketing Research, 26, 135-148.
27. Popping, R. (1984). AGREE, a package for computing nominal scale agreement. Computational Statistics and Data Analysis, 2, 182-185.
28. Popping, R. (1988). On agreement indices for nominal data. In W. E. Saris & I. N. Gallhofer (Eds.), Sociometric research: Data collection and scaling (Vol. 1, pp. 90-105). New York: St. Martin's Press.
29. Potter, W. J., & Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis. Journal of Applied Communication Research, 27, 258-284.
30. ProGAMMA. (2002, July 7). AGREE [Computer software]. Available from http://www.gamma.rug.nl/
31. Riffe, D., & Freitag, A. A. (1997). A content analysis of content analyses: Twenty-five years of Journalism Quarterly. Journalism & Mass Communication Quarterly, 74, 873-882.
32. Riffe, D., Lacy, S., & Fico, F. G. (1998). Analyzing media messages: Using quantitative content analysis in research. Mahwah, NJ: Erlbaum.
33. Rust, R., & Cooil, B. (1994). Reliability measures for qualitative data: Theory and implications. Journal of Marketing Research, 31, 1-14.
34. Scott, W. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 17, 321-325.
35. Seun, H. K., & Lee, P. S. C. (1985). Effects of the use of percentage agreement on behavioral observation reliabilities: A reassessment. Journal of Psychopathology and Behavioral Assessment, 7, 221-234.
36. Singletary, M. W. (1993). Mass communication research: Contemporary methods and applications. Boston: Addison-Wesley.
37. Skymeg Software (2002, July 7). Program for reliability assessment with multiple coders (PRAM) [Computer software]. Available from http://www.geocities.com/skymegsoftware/pram.html
38. Tinsley, H. E. A., & Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgements. Journal of Counseling Psychology, 22, 358-376.
39. Tinsley, H. E. A., & Weiss, D. J. (2000). Interrater reliability and agreement. In H. E. A. Tinsley & S. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 95-124). San Diego, CA: Academic Press.