Assessment of interjudge reliability in the open-ended questions coding process

Quality and Quantity

Volumn 40, Issue 4, 2006, Pages 519-537

  Leiva, Francisco Muñoz ^a   Ríos, Francisco Javier Montoro ^a   Martínez, Teodoro Luque ^a  

^a UNIVERSITY OF GRANADA   (Spain)

Author keywords

Coding process; Concordance coefficients; Interjudge reliability; Open ended questions

Indexed keywords

EID: 33747042528     PISSN: 00335177     EISSN: 15737845     Source Type: Journal    
DOI: 10.1007/s11135-005-1093-6     Document Type: Article

References (42)

1. Abernethy, A. M. & Franke, G. R. (1996). The information content of advertising: A meta-analysis. Journal of Advertising 25(2): 1-17.
2. Bardin, L. (1977). L'analyse de contenu. (Paris: PUF).
3. Bennett, E. M. & Goldstein, A. C. (1954). Communications through limited response questioning. Public Opinion Quaterly 18: 303-308.
4. Berelsen, B. (1952). Content Analysis in Communications Research. Glencoe (Ill.): Free Press.
5. Brenner, H. & Kliebsch, U. (1996). Dependence of weighted kappa coefficients on the number of categories. Epidemiology 7(2): 199-202.
6. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20 (Winter): 37-46.
7. Benzin, N. K. & Lincoln, Y. S. (1994). Handbook of Qualitative Research. Thousand Oaks (CA): Sage Publications.
8. ESOMAR (2003). Annual Study of the Market Research Industry 2002. (Amsterdam: ESO-MAR).
9. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin 76: 378-382.
10. Fontana, A. & Frey, J. H. (1994). Interviewing. The art of science. In: N. K. Denzin & Y. S. Licoln (eds.),Handbook of Qualitative Research. Thousand Oaks (CA): Sage Publications.
11. Glaser, B. G. & Strauss, A. L. (1999). The Discovery of Grounded Theory: Strategies for Qualitative Research. New York: Aldine de Gruyter.
12. Grayson, K. & Rust, R. (2001). Interrater reliability. Journal of Consumer Psychology 10 (1&2): 71-73.
13. Herbert, L. (1977). Kappa revisited. Psychological Bulletin: 84(2): 289-297.
14. Holbrook, M. B. (1977). More on content analysis in consumer research. Journal of Consumer Research: 4(3): 176-177.
15. Holsti, O. R. (1969). Content Analysis for the Social Sciences and Humanities. Reading (MA): Addison-Wesley.
16. Hsu & Fied (2003). Interrater agreement measures: Comments on Kappa, Cohen's Kappa, Scott's π and Aicking α. Understanding statistics 2(3): 205-219.
17. Hughes, M. A. & Garret, D. E. (1990). Intercoder reliability estimation approaches in marketing: A Generability Theory framework for quantitative data. Journal of Marketing Research 27(2): 185-95.
18. Kang, N., Kara, A., Laskey, H. A. & Seaton, F. B. (1993). A SAS MACRO for calculating intercoder agreement in content analysis. Journal of Advertising 22(2): 17-28.
19. Kassarjian, H. H. (1977). Content analysis in consumer research. Journal of Consumer Research 4(2): 8-18.
20. Kolbe, R.H. & Burnett, M.S. (1991). Content-analysis research: An examination of applications with directives for improving research reliability and objectivity. Journal of Consumer Research 18(September): 243-250.
21. Krippendorff, K. (1971). Reliability of recording instructions: Multivariate agreement for nominal data. Behavioral Science 16(3): 228-235.
22. Krippendorff, K. (1980). Content Analysis, an Introduction to Its Methodology. Thousand Oaks (CA): Sage Publications.
23. Landis, J. R. & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33: 159-174.
24. Lehmann, R. L., Gupta, S. & Steckel, J. H. (1998). Marketing Research. Reading (MA): Addison-Wesley Educational Publishers Inc.
25. Light, R. J. (1971). Measures of response agreement for qualitative data: some generalizations and alternatives. Psychological Bulleting 76(5): 365-377.
26. Lincoln, Y. S. & Guba, E. G. (1985). Naturalistic Inquiry. Beverly Hills (CA): Sage Publications.
27. Lombard, M., ZINDER-duch, J. & Campanela, C. (2002). Content analysis in mass communication: Assessment and reporting of intercoder reliability. Human Communication Research 28(4): 587-604.
28. López-Aranguren (1989). El análisis de contenido. In: M. García et al. (Coord.): El Análisis de la Realidad Social: Métodos y Técnicas de Investigación. Madrid: Alianza Editorial.
29. Luque, T. (1997). Investigación de Marketing. Barcelona: Ed. Ariel.
30. Mackensen, K. & Wille, U. (1999). Qualitative text analysis supported by conceptual data systems. Quality & Quantity 33: 135-156.
31. Miles, M. B. & Huberman, A. M. (1994). Quality Data Analysis. An expanded Sourcebook. Thousand Oaks (CA): Sage Publications.
32. Miller, W. L. & Crabtree, B. F. (1994). Clinical research. In: N. K. Denzin & Y. S. Licoln (eds.), Handbook of Qualitative Research. Thousand Oaks (CA): Sage Publications.
33. Perreault, W. D. & LEIGH E. L. (1989). Reliability of nominal data based on quantitative judgments. Journal of Marketing Research 23(May): 130-43.
34. Richards & Richards (1994). Using computers in qualitative research. In: N. K. Denzin & Y. S. Licoln (eds.), Handbook of Qualitative Research. Thousand Oaks (CA): Sage Publications.
35. Riffe, D. & Freitag, A. A. (1997). A content analysis of content analysis: Twenty-five years of Journalism Quarterly. Journalism & Mass Communication Quarterly 74: 873-882.
36. Roberts, C. W. (2000). A conceptual framework for quantitative text analysis: On joining probabilities and substantive inferences about text. Quality & Quantity 34: 259-274.
37. Rust, R. T. & Cooil, B. (1994). Reliability measures for qualitative data: Theory and implications. Journal of Marketing Research 31: 1-14.
38. Spiggle, S. (1994). Analysis and interpretation of qualitative data in consumer research. Journal of Marketing Research 21 (December): 491-503.
39. Spitznagel, E. L. & Helzer, J. E. (1985). A proposed solution to the base rate problem in the kappa statistic. Archives of General Psychiatry 42(7): 725-28.
40. Strauss, A. & Corbin, J. (1990). Basic of qualitative research: Grounded theory procedures and techniques. Newbury Park (CA): Sage Publications.
41. Weber, R. P. (1985). Basic. Content Analysis. Newbury Park: Ed. Sage Publications.
42. Weitzman, E. A. & Miles, M. B. (1995). Computer Programs for Qualitative Data Analysis: A Software Sourcebook. Thousand Oaks (CA): Sage Publications.