Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research

Field Methods

Volumn 16, Issue 3, 2004, Pages 307-331

  Hruschka, Daniel J ^a   Schwartz, Deborah ^b   St john, Daphne Cobb ^b   Picone Decaro, Erin ^b   Jenkins, Richard A ^b   Carey, James W ^b  

^a EMORY UNIVERSITY   (United States)

^b CENTERS FOR DISEASE CONTROL AND PREVENTION   (United States)

Author keywords

data; intercoder agreement; interrater agreement; open ended; qualitative; reliability

Indexed keywords

EID: 33444476650     PISSN: 1525822X     EISSN: None     Source Type: Journal    
DOI: 10.1177/1525822X04266540     Document Type: Article

References (63)

1. Altheide, D. L., and J. M. Johnson. 1994. Criteria for assessing interpretive validity in qualitative research. In Handbook of qualitative research, edited by N. K. Denzin and Y. Lincoln, 485–499. Thousand Oaks, CA: Sage.
2. Appleton, J. V. 1995. Analysing qualitative interview data: Addressing issues of validity and reliability. Journal of Advanced Nursing 22: 993–997.
3. Armstrong, D., A. Gosling, J. Weinman, and T. Marteau. 1997. The place of inter-rater reliability in qualitative research: An empirical study. Sociology 31: 597–606.
4. Banerjee, M., M. Capozzoli, L. McSweeney, and D. Sinha. 1999. Beyond kappa: A review of interrater agreement measures. The Canadian Journal of Statistics 27: 3–23.
5. Bell, P. 2001. Semiotics and the content analysis of visual images. Journal of Information, Communication and Library Sciences 7: 81–100.
6. Bernard, H. R. 2002. Research methods in anthropology: Qualitative and quantitative approaches. 3rd ed. Walnut Creek, CA: Altamira.
7. Boyatzis, R. E. 1998. Transforming qualitative information: Thematic analysis and code development. Thousand Oaks, CA: Sage.
8. Brennan, R. L., and D. J. Prediger. 1981. Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement 41: 687–699.
9. Byrt, T., J. Biship, and J. B. Carlin. 1993. Bias, prevalence and kappa. Journal of Clinical Epidemiology 46 (5): 423–429.
10. Carey, J., P. H. Wenzel, C. Reilly, J. Sheridan, and J. M. Steinberg. 1998. CDC EZ-TEXT: Software for management and analysis of semistructured qualitative data sets. Cultural Anthropology Methods Journal 10: 14–20.
11. Carey, J. W., M. Morgan, and M. Oxtoby. 1996. Inter-coder agreement in analysis of responses to open-ended interview questions: Examples From tuberculosis research. Cultural Anthropology Methods 8: 1–5.
12. Cicchetti, D. V. 1994. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment 6: 284–290.
13. Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychosocial Measurement 20: 37–46.
14. Conger, A. J. 1980. Integration and generalization of kappas for multiple raters. Psychological Bulletin 88: 322–328.
15. Cronbach, L. J. 1990. Essentials of psychological testing. New York: Harper Collins.
16. Denzin, N. K., and Y. Lincoln. 1994. Handbook of qualitative research. Thousand Oaks, CA: Sage.
17. Drisko, J. 1997. Strengthening qualitative studies and reports: Standards to promote academic integrity. Journal of Social Work Education 33: 185–197.
18. Field, A. P., and J. M. Morse. 1985. Nursing research: The application of qualitative approaches. London: Croom Helm.
19. Fitzpatrick, R., and M. Boulton. 1996. Qualitative research in health care I: The scope and validity of methods. Journal of Evaluation in Clinical Practice 2: 123–130.
20. Fleiss, J. L. 1971. Measuring nominal scale agreement among many raters. Psychological Bulleting 76 (5): 378–382.
21. Fleiss, J. L. 1981. Statistical methods for rates and proportions. 2nd ed. New York: Wiley.
22. Fleiss, J. L., J. C. M. Nee, and J. R. Landis. 1979. Large sample variance of kappa in the cases of different sets of raters. Psychological Bulletin 86 (5): 974–977.
23. General Accounting Office (GAO). 1991. Using structured interview techniques. Washington DC: U.S. GAO.
24. Googenmoos-Holzmann, Irene. 1993. How reliable are chance-corrected measures of agreement? Statistics in Medicine 12: 2191–2205.
25. Gorden, R. 1992. Basic interviewing skills. Itasca, IL: F.E. Peacock.
26. Guba, E. G., and Y. S. Lincoln. 1994. Competing paradigms in qualitative research. In Handbook of qualitative research, edited by N. K. Denzin and Y. Lincoln, 105–117. Thousand Oaks, CA: Sage.
27. Hagelin, E. 1999. Coding data from child health records: The relationship between interrater agreement and interpretive burden. Journal of Pediatric Nursing 14: 313–321.
28. Hak, T., and T. Bernts. 1996. Coder training: Theoretical training or practical socialization? Qualitative Sociology 19: 235–257.
29. Heyman, R. E., B. R. Chaudhry, D. Treboux, J. Crowell, C. Lord, D. Vivian, and E. B. Waters. 2001. How much observational data is enough? An empirical test using marital interaction coding. Behavior Therapy 32: 107–122.
30. Hohnloser, J. H., P. Kadlec, and F. Puerner. 1995. Experiments in coding clinical information: An analysis of clinicians using a computerized coding tool. Computers and Biomedical Research 28: 393–401.
31. Hruschka, D., B. Cummings, D. Cobb St. John, J. Moore, G. Khumalo-Sakutukwa, and J. W. Carey. 2004. Fixed-choice and open-ended response formats: A comparison from HIV Prevention Research in Zimbabwe. Field Methods 16: 184–202.
32. Hutchins, E. 1995. Cognition in the wild. Cambridge, MA. MIT Press.
33. Kahneman, D., P. Slovic, and A. Tversky. 1982. Judgment under uncertainty: Heuristics and biases. New York: Cambridge University Press.
34. Klahr, D., and K. Kotovsky. 1989. Complex information processing: The impact of Herbert A. Simon. Hillsdale, NJ: Lawrence Erlbaum.
35. Kolbe, R. H., and M. S. Burnett. 1991. Content-analysis research: An examination of applications with directions for improving research reliability and objectivity. Journal of Consumer Research 18: 243–250.
36. Krippendorff, K. 1980. Content analysis: An introduction to its methodology. Beverly Hills, CA: Sage.
37. Krippendorff, K. 1995. On the reliability of unitizing continuous data. Sociological Methodology 25: 47–76.
38. Landis, J. R., and G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33: 159–174.
39. Liebetrau, A. M. 1983. Measures of association. Newbury Park, CA: Sage.
40. Lombard, M., J. Snyder-Duch, and C. C. Campanella. 2002. Content analysis in mass communication assessment and reporting of inter-coder reliability. Human Communication Research 28 (4): 587–604.
41. MacQueen, K. M., E. McLelland, K. Kay, and B. Milstein. 1998. Codebook development for team-based qualitative analysis. Cultural Anthropology Methods 10: 31–36.
42. Madill, A., A. Jordan, and C. Shirley. 2000. Objectivity and reliability in qualitative analysis: Realist, contextualist and radical constructionist epistemologies. British Journal of Psychology 91: 1–20.
43. Mantell, J. E., A. T. DiVittis, and M. I. Auerbach. 1997. Evaluating HIV prevention interventions. New York: Plenum.
44. Mays, N., & Pope, C. 1995. Rigour and qualitative research. British Medical Journal 311: 109–112.
45. Mays, N., & Pope, C. 2000. Assessing quality in qualitative research. British Medical Journal 320: 50–52.
46. Miles, M. B., and A. M. Huberman. 1994. Qualitative data analysis. Thousand Oaks, CA: Sage.
47. Morse, J. M. 1997. Perfectly healthy but dead: The myth of inter-rater reliability. Qualitative Health Research 7: 445–447.
48. Murphy, K. R., R. De Shon. 2000. Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology 53: 873–900.
49. Mutchler, M. G., T. Bingham, M. Chion, R. A. Jenkins, L. E. Klosinski, and G. Secura. In press. Comparing sexual behavioral patterns between two bathhouses: Implications for HIV prevention intervention policy. Journal of Homosexuality.
50. Neumark-Sztainer, D., and M. Story. 1997. Recommendations from overweight youth regarding school-based weight control programs. Journal of School Health 67: 428–433.
51. Nunnally, J. C., and I. H. Bernstein. 1994. Psychometric theory. New York: McGraw-Hill.
52. O’Leary, A., J. Moore, G. Sakatukwa, L. Loeb, D. Hruschka, R. Khan, and N. Padian. 2003. Association of negotiation strategies to consistent use of male condoms by women receiving an HIV prevention intervention in Zimbabwe. AIDS 17: 1705-1707.
53. Parsons, J. T., H. C. Huszti, S. O. Crudder, B. Gage, D. Jarvis, J. Mendoza, and K. L. Parish. 1998. Determinants of HIV risk reduction behaviors among female partners of men with hemophilia and HIV infection. AIDS and Behavior 2: 1–11.
54. Potter, W.J., and D. Levine-Donnerstein. 1999. Rethinking validity and reliability in content analysis. Journal of Applied Communication Research 27: 258–284.
55. Rose, K., and C. Webb. 1998. Analyzing data: Maintaining rigor in a qualitative study. Qualitative Health Research 8: 556–562.
56. Ryan, G. W., and H. R. Bernard. (2003). Techniques to identify themes. Field Methods 15 (1): 85–109.
57. Simon, H. A. 1981. The sciences of the artificial. 2nd ed. Cambridge, MA: MIT Press.
58. Strotman, R., E. McLelland, K. M. MacQueen, and B. Milstein. 2002. AnSWR: Analysis software for Word-based records, version 6.2. Atlanta: Centers for Disease Control and Prevention.
59. Sykes, W. 1991. Taking stock: Issues from the literature on validity and reliability in qualitative research. Journal of the Market Research Society 33: 3–12.
60. Wang, S. S. L., L.-C. Lin, and B. Ing-Tau Kuo. 1997. The health care needs of hospitalized patients with AIDS in Taiwan. AIDS Patient Care and STDS 11: 179–188.
61. Weber, R. P. 1990. Basic content analysis. Newbury Park, CA: Sage.
62. Weinberger, M., J. A. Ferguson, G. Westmoreland, L. A. Mamlin, D. S. Segar, G. J. Eckert, J. Y. Greene, D. K. Martin, and W. Tierney. 1998. Can raters consistently evaluate the content of focus groups? Social Science and Medicine 46: 929–933.
63. Willms, D. G., J. A. Best, D. W. Taylor, J. R. Gilbert, D. M. C. Wilson, E. A. Lindsay, and J. Singer. 1990. A systematic approach for using qualitative methods in primary prevention research. Medical Anthropology Quarterly 4: 391–409.