Interrater reliability estimators commonly used in scoring language assessments: A Monte Carlo investigation of estimator accuracy

Language Assessment Quarterly

Volumn 11, Issue 3, 2014, Pages 304-324

  Morgan, Grant B ^a   Zhu, Min ^b   Johnson, Robert L ^c   Hodge, Kari J ^a  

^a BAYLOR UNIVERSITY   (United States)

^b EXCELSIOR COLLEGE   (United States)

^c UNIVERSITY OF SOUTH CAROLINA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84906857063     PISSN: 15434303     EISSN: 15434311     Source Type: Journal    
DOI: 10.1080/15434303.2014.937486     Document Type: Article

References (57)

1. Babakus, E., Ferguson, C. E., & Jöreskog, K. G. (1987). The sensitivity of confirmatory maximum likelihood factor analysis to violations of measurement scale and distributional assumptions. Journal of Marketing Research, 24(2), 222-229.
2. Bandalos, D. L., & Leite, W. (2013). Use of Monte Carlo studies in structural equation modeling research. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 625-666). Charlotte, NC: Information Age.
3. Barkaoui, K. (2010). Variability in ESL essay rating processes: The role of the rating scale and rater experience. Language Assessment Quarterly, 7(1), 54-74.
4. Bollen, K. A. (1989). Structural equations with latent variables. New York, NY: John Wiley & Sons, Inc.
5. Bollen, K. A., & Barb, K. H. (1981). Pearson's r and coarseley categorized data. American Sociological Review, 46(2), 232-239.
6. Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer-Verlag.
7. Cheng, L., & Qi, L. (2006). Description and examination of the national matriculation English test. Language Assessment Quarterly, 3(1), 53-70.
8. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
9. Common Core State Standards Initiative. (2012). In the states. Retrieved from http://www.corestandards.org/in-the-states
10. Coote, L. (1998). A review and recommended approach for estimating conditional structural equation models. Paper presented at the annual conference of the Australia and New Zealand Marketing Academy, University of Otago, Dunedin.
11. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
12. Cureton, E. E. (1958). The average Spearman rank criterion when ties are present. Psychometrika, 23(3), 271-272.
13. East, M. (2009). Evaluating the reliability of a detailed analytic scoring rubric for foreign language writing. Assessing Writing, 14(2), 88-115.
14. Finney, S. J., & DiStefano, C. (2013). Nonnormal and categorical data in structural equation modeling. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 439-492). Charlotte, NC: Information Age.
15. Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis. Psychological Methods, 9(4), 466-491.
16. Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research, & Evaluation, 17(3). Retrieved from http://pareonline.net/getvn.asp?v=17&n=3
17. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65-110). Westport, CT: American Council on Education and Praeger Publishers.
18. Hamp-Lyons, L., & Henning, G. (1991). Communicative writing profiles: An investigation of the transferability of a multiple-trait scoring instrument across ESL writing assessment contexts. Language Learning, 41(3), 337-373.
19. Harman, H. H. (1967). Modern factor analysis. Chicago, IL: University of Chicago Press.
20. Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77-89.
21. Herman, J., Aschbacher, P., & Winters, L. (1992). A practical guide to alternative assessment. Alexandria, VA: Association for Supervision and Curriculum Development.
22. Isaacs, T., & Thomson, R. (2013). Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly, 10(2), 135-159.
23. Johnson, E., Habing, B., & Johnson, R. L. (2010). The impact of rubric changes on item properties and test reliability. Paper presented at the annual meeting of the American Educational Research Association, Denver, CO.
24. Johnson, R. L., McDaniel, F., &Willeke, M. (2000). Using portfolios in program evaluation: An investigation of interrater reliability. The American Journal of Evaluation, 21(1), 65-80.
25. Johnson, R. L., Penny, J. A.,&Gordon, B. (2000). The relation between score resolution methods and interrater reliability: An empirical study of an analytic scoring rubric. Applied Measurement in Education, 13(2), 121-138.
26. Johnson, R. L., Penny, J. A., & Gordon, B. (2009). Assessing performance: Developing, scoring, and validating performance tasks. New York, NY: Guilford Publications.
27. Jöreskog, K. G. (1994). On the estimation of polychoric correlations and their asymptotic covariance matrix. Psychometrika, 59(3), 381-389.
28. Kaplan, R., & Saccuzzo, D. (1982). Psychological testing: Principles, applications, and issues. Monterey, CA: Brooks/Cole.
29. Koretz, D., Stecher, B., Klein, S., & McCaffrey, D. (1994). The Vermont portfolio assessment program: Findings and implications. Educational Measurement: Issues and Practice, 13(3), 5-16.
30. Kunnan, A. J. (2004). Test fairness. In M. Milanovic, C. Weir, & S. Bolton (Eds.), Europe language testing in a global context: Selected papers from the ALTE conference in Barcelona (pp. 27-48). Cambridge, England: Cambridge University Press.
31. Linacre, J. M. (1994). Constructing measurement with a many-facet Rasch model. In M. Wilson (Ed.), Objective measurement: Theory in practice: Vol. 2. (pp. 356-442). Newark, NJ: Ablex.
32. Linacre, J. M. (2002). Judge ratings with forced agreement. Rasch Measurement Transactions, 16(1), 857-858.
33. McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
34. Milanovic, M., Saville, N., Pollitt, A., & Cook, A. (1996). Developing rating scales for CASE: Theoretical concerns and analyses. In M. Milanovic, N. Saville, & A. Pollitt (Eds.), Validation in language testing (pp. 15-38). Clevedon, United Kingdom: Multilingual Matters.
35. Moskal, B. M., Leydens, J. A. (2000). Scoring rubric development: Validity and reliability. Practical Assessment, Research, and Evaluation, 7(10). Retrieved from http://PAREonline.net/getvn.asp?v=7&n=10
36. National Center for Education Statistics. (2012). The Nation's Report Card: Writing 2011 (NCES 2012 470).Washington, DC: Institute of Education Sciences, U.S. Department of Education.
37. Nie, Y., Yeo, S., & Lau, S. (2007). Application of generalizability theory in the investigation of the quality of journal writing in mathematics. Studies in Educational Evaluation, 33(3?4), 371-383.
38. Nunnally, J. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.
39. Ockey, G., Koyama, D., & Setoguchi, E. (2013). Stakeholder input and test design: A case study on changing the interlocutor familiarity facet of the group oral discussion test. Language Assessment Quarterly, 10(3), 292-308.
40. Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44(4), 443-460.
41. Penny, J. A., Johnson, R. L., & Gordon, B. (2000). The effect of score augmentation on the interrater reliability: An empirical study of a holistic rubric. Assessing Writing, 7(2), 142-164.
42. Penny, J. A., & Surles, J. (2002). SAS code for simulating the reliability for two raters and breaking those ratings into different scale points. Jamestown, NC: Author.
43. Phillips, S. (2000). Legal corner: GI forum v TEA. NCME Newsletter, 8(2).
44. SAS Institute.(2010). SAS 9.2 help and documentation. Cary, NC: SAS Institute, Inc.
45. Shavelson, R. J., & Webb, N. (1991). Generalizability theory: A primer. Thousand Oaks, CA: Sage.
46. Sheskin, D. J. (2007). Handbook of parametric and nonparametric statistical procedures (4th ed.). Boca Raton, FL: Chapman and Hal/CRCl.
47. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlation: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428.
48. Smarter Balanced Assessment Consortium. (2012). Sample items and performance tasks. Retrieved from http://www.smarterbalanced.org/sample-items- and-performance-tasks/
49. Stanley, J. C. (1971). Reliability. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 356-442).Washington, DC: American Council on Education.
50. Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research, and Evaluation, 9(4). Retrieved from http://PAREonline. net/getvn.asp?v=9&n=4
51. Stemler, S. E. (2007). Interrater reliability. In N. Salkind (Ed.), Encyclopedia of measurement and statistics (pp. 484-486). Thousand Oaks, CA: Sage.
52. Struthers, L., Lapadat, J., & MacMillan, P. (2013). Assessing cohesion in children's writing: Development of a checklist. Assessing Writing, 18(3), 187-201.
53. Supovitz, J., MacGowan, A., & Slattery, J. (1997). Assessing agreement: An examination of the interrater reliability of portfolio assessment in Rochester, New York. Educational Assessment, 4(3), 237-259.
54. Thorndike, R. M. (2005). Measurement and evaluation in psychology and education. Upper Saddle River, NJ: Prentice Hall.
55. Uebersax, J. S. (2010). The tetrachoric and polychoric correlation coefficients. Retrieved from http://john-uebersax.com/stat/tetra.htm
56. Yao, Y., Thomas, N., M., N., Downing, J., Burkett, R., & Lamson, S. (2008). Validity evidence of an electronic portfolio for preservice teachers. Educational Measurement: Issues and Practice, 27(1), 10-24.
57. Zhang, Y., & Elder, C. (2009). Measuring the speaking proficiency of advanced EFL learners in China: The CET-SET solution. Language Assessment Quarterly, 6(4), 298-314.