Rater effects on essay scoring: A multilevel analysis of severity drift, central tendency, and rater experience

Journal of Educational Measurement

Volumn 48, Issue 4, 2011, Pages 399-418

  Leckie, George ^a   Baird, Jo Anne ^a  

^a UNIVERSITY OF BRISTOL   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84055198279     PISSN: 00220655     EISSN: 17453984     Source Type: Journal    
DOI: 10.1111/j.1745-3984.2011.00152.x     Document Type: Article

References (35)

1. Brown, A. (1995). The effect of rater variables in the development of an occupation-specific language performance test. Language Testing, 12, 1-15.
2. Congdon, P. J., & McQueen, J. (2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37, 163-178.
3. Cumming, A. (1990). Expertise in evaluating second language compositions. Language Testing, 7, 31-51.
4. Eckes, T. (2005). Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly, 2, 197-221.
5. Eckes, T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25, 155-185.
6. Engelhard, G. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93-112.
7. Goldstein, H. (2010). Multilevel statistical models (4th ed.). New York, NY: Wiley.
8. Harik, P., Clauser, B. E., Grabovsky, I., Nungester, R. J., Swanson, D., & Nandakumar, R. (2009). An examination of rater drift within a generalizability theory framework. Journal of Educational Measurement, 46, 43-58.
9. Hedeker, D. R., & Gibbons, R. D. (2006). Longitudinal data analysis. New York, NY: Wiley.
10. Hoskens, M., & Wilson, M. (2001). Real-time feedback on rater drift in constructed-response items: An example from the Golden State examination. Journal of Educational Measurement, 38, 121-145.
11. Kingsbury, F. A. (1922). Analyzing ratings and training raters. Journal of Personnel Research, 1, 377-383.
12. Knoch, U., Read, J., & von Randow, J. (2007). Re-training writing raters online: How does it compare with face-to-face training? Assessing Writing, 12, 26-43.
13. Lamprianou, I. (2006). The stability of marker characteristics across tests of the same subject and across subjects. Journal of Applied Measurement, 7, 192-205.
14. Leckie, G., & Charlton, C. (2011). runmlwin: Stata module for fitting multilevel models in the MLwiN software package [Computer software]. Bristol, England: Centre for Multilevel Modeling, University of Bristol.
15. Leckie, G., & Goldstein, H. (2009). The limitations of using school league tables to inform school choice. Journal of the Royal Statistical Society Series a-Statistics in Society, 172, 835-851.
16. Leckie, G., & Goldstein, H. (2011). Understanding uncertainty in school league tables. Fiscal Studies, 32, 207-224.
17. Lumley, T., Lynch, B. K., & McNamara, T. F. (1994). A new approach to standard-setting in language assessment. Melbourne Papers in Language Testing, 3, 19-40.
18. Lunz, M. E., & Stahl, J. A. (1990). Judge consistency and severity across grading periods. Evaluation & the Health Professions, 13, 425-444.
19. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
20. Myford, C. M., & Wolfe, E. W. (2009). Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale category use. Journal of Educational Measurement, 46, 371-389.
21. Pinot de Moira, A., Massey, C., Baird, J., & Morrissey, M. (2002). Marking consistency over time. Research in Education, 67, 79-87.
22. Powers, D., & Kubota, M. (1998). Qualifying essay readers for an online scoring network (OSN) (Research Report). Princeton, NJ: Educational Testing Service.
23. Rasbash, J., Charlton, C., Browne, W. J., Healy, M., & Cameron, B. (2009). MLwiN Version 2.1 [Computer software]. Bristol, England: Centre for Multilevel Modeling, University of Bristol.
24. Rasbash, J., & Goldstein, H. (1994). Efficient analysis of mixed hierarchical and cross-classified random structures using a multilevel model. Journal of Educational and Behavioral Statistics, 19, 337-350.
25. Raudenbush, S. W. (1993). A crossed random effects model for unbalanced data with applications in cross-sectional and longitudinal research. Journal of Educational and Behavioral Statistics, 18, 321-349.
26. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Newbury Park, CA: Sage.
27. Royal-Dawson, L., & Baird, J. (2009). Is teaching experience necessary for reliable scoring of extended English questions? Educational Measurement: Issues and Practice, 28(2), 2-8.
28. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
29. Shohamy, E., Gordon, C. M., & Kraemer, R. (1992). The effect of raters' background and training on the reliability of direct writing tests. The Modern Language Journal, 76, 27-33.
30. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York, NY: Oxford University Press.
31. Thorndike, E. L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 4, 25-29.
32. Weigle, S. C. (1999). Investigating rater/prompt interactions writing assessment: Quantitative and qualitative approaches. Assessing Writing, 6(2), 145-178.
33. Wolfe, E. W. (2004). Identifying rater effects using latent trait models. Psychology Science, 46, 35-51.
34. Wolfe, E. W., Kao, C. W., & Ranney, M. (1998). Cognitive differences in proficient and nonproficient essay scorers. Written Communication, 15, 465-492.
35. ® English Literature and Composition Examination using benchmark essays (Research Report 2007-2). New York, NY: The College Board (accessed 27 April 2010).