Sources of difference in reliability: Identifying sources of difference in reliability in content analysis of online asynchronous discussions

International Review of Research in Open and Distance Learning

Volumn 6, Issue 2, 2005, Pages

  Murphy, Elizabeth ^a   Ciszewska Carr, Justyna ^a  

^a MEMORIAL UNIVERSITY OF NEWFOUNDLAND   (Canada)

Author keywords

Coding; Cohen's kappa; Content analysis; Online discussions; Reliability; Sources of difference

Indexed keywords

EID: 23844451736     PISSN: 14923831     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (25)

1. Berelson, B. (1952). Content analysis in communication research. New York: Free Press.
2. Brennan, R. L., and Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687-699.
3. Capozzoli, M., McSweeney, L., and Sinha, D. (1999). Beyond kappa: A review of interrater agreement measures. The Canadian Journal of Statistics, 27(1), 3-23.
4. Carletta, J. (1996). Assessing Agreement on Classification Tasks: The kappa statistic. Computational Linguistics, 22(2), 49-54.
5. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
6. Crocker, R., and Schulz, H. (2001). Design of a generalizability study for SAIP assessments. Report submitted to the Council of Ministers of Education, Canada.
7. Garrison, D. R., Anderson, T., and Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7-23.
8. Hillman, D. C. A. (1999). A new method for analyzing patterns of interaction. The American Journal of Distance Education, 13(2), 37-47.
9. Kanuka, H., and Anderson, T. (1998). Online social interchange, discourse, and knowledge construction. Journal of Distance Education, 13(1), 57-74.
10. Kolbe, R. H., and Burnett, M. S. (1991). Content Analysis Research: An examination of applications with directives for improving research reliability and objectivity. Journal of Consumer Research, 18, 243-250.
11. Krippendorff, K. (1980). Content Analysis: An introduction to its methodology. Beverly Hills, CA: Sage.
12. Kvalseth, T. O. (1989). Note on Cohen's kappa. Psychological Reports, 65, 223-226.
13. Lombard, M., Snyder-Duch, J., and Bracken, C. C. (2002). Content Analysis in Mass Communication: Assessment and reporting of intercoder reliability. Human Communication Research, 28, 587-604.
14. Murphy, E. (2004a). Identifying and measuring problem formulation and resolution in online asynchronous discussions. Canadian Journal of Learning and Technology, 30(1), 5-20.
15. Murphy, E. (2004b). Promoting construct validity in instruments for the analysis of transcripts of online asynchronous discussions. Educational Media International, 41(4), 346-354.
16. Neuendorf, K. A. (2002). The Content Analysis Guidebook. Thousand Oaks, CA: Sage Publications.
17. Perreault, W. D. Jr., Leigh, L. E. (1989). Reliability of nominal data based on qualitative judgments. Journal of Marketing Research, 26, 135-148.
18. Popping, R. (1988). On agreement indices for nominal data. In W. E. Saris & I. N. Gallhofer (Eds.), Sociometric Research: Volume 1, data collection and scaling (pp. 90-105). New York: St. Martin's Press.
19. Potter, W. J., and Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis. Journal of Applied Communication Research, 27(3), 258-284.
20. Riffe, D., Lacy, S., and Fico, F. G. (1998). Analyzing Media Messages: Using quantitative content analysis in research. Mahwah, NJ: Lawrence Erlbaum.
21. Rourke, L., and Anderson, T. (2004). Validity in quantitative content analysis. Educational Technology Research and Development, 52(1), 5-18.
22. Rourke, L., Anderson, T., Garrison, D. R., and Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education, 12(1), 8-22. Retrieved February 11, 2005 from: http://communitiesofinquiry.com/ documents/MethPaperFinal.pdf
23. Singletary, M. (1994). Mass Communication Research: Contemporary methods, and applications. White Plains, NY: Longman.
24. Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9(4). Retrieved April 7, 2005 from: http://PAREonline.net/getvn.asp? v=9&n=4
25. Tinsley, H. E., and Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology, 22, 358-376.