Effects of marking method and rater experience on ESL essay scores and rater performance

Assessment in Education: Principles, Policy and Practice

Volumn 18, Issue 3, 2011, Pages 279-293

  Barkaoui, Khaled ^a  

^a YORK UNIVERSITY   (Canada)

Author keywords

Analytic marking; ESL essay rating; Holistic marking; Marking method; Multi faceted rasch models; Rater experience

Indexed keywords

EID: 79961074394     PISSN: 0969594X     EISSN: 1465329X     Source Type: Journal    
DOI: 10.1080/0969594X.2010.526585     Document Type: Article

References (32)

1. Bacha, N. 2001. Writing evaluation: What can analytic versus holistic essay scoring tell us? System 29, no. 3: 371-383.
2. Barkaoui, K. 2007a. Participants, texts, and processes in second language writing assessment: A narrative review of the literature. The Canadian Modern Language Review 64, no. 1: 97-132.
3. Barkaoui, K. 2007b. Rating scale impact on EFL essay marking: A mixed-method study. Assessing Writing 12, no. 2: 86-107.
4. Barkaoui, K. 2008. Effects of scoring method and rater experience on ESL essay rating processes and outcomes. Unpublished PhD thesis, University of Toronto.
5. Barkaoui, K. 2010a. Variability in ESL essay rating processes: The role of rating scale and rater experience. Language Assessment Quarterly 7, no. 1: 54-74.
6. Barkaoui, K. 2010b. Explaining ESL essay holistic scores: A multilevel modeling approach. Language Testing 27, no. 4: 515-435.
7. Bond, T.G., and C.M. Fox. 2001. Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.
8. Cumming, A. 1990. Expertise in evaluating second language compositions. Language Testi ng7, no. 1: 31-511.
9. Field, A. 2005. Discovering statistics using SPSS. 2nd Thousand Oaks, CA: Sage.
10. Freedman, S.W. 1984. The registers of student and professional expository writing: Influences on teachers' responses. In New directions in composition research, ed. R. Beach and S. Bridwell, 334-347. New York: Guilford Press.
11. Goulden, N.R. 1992. Theory and vocabulary for communication assessments. Communication Education 41, no. 3: 258-269.
12. Goulden, N.R. 1994. Relationship of analytic and holistic methods to raters' scores for speeches. The Journal of Research and Development in Education 27, no. 2: 73-82.
13. Hamp-Lyons, L. 1991. Scoring procedures for ESL contexts. In Assessing second language writing in academic contexts, ed. L. Hamp-Lyons, 241-76. Norwood, NJ: Ablex.
14. Hamp-Lyons, L. 1995. Rating non-native writing: The trouble with holistic scoring. TESOL Quarterly 29, no. 4: 759-762.
15. Hamp-Lyons, L., and B. Kroll. 1997. TOEFL 2000-writing: Composition, community and assessment. TOEFL Monograph Series No. 5. Princeton, NJ: Educational Testing Service.
16. Huot, B.A. 1993. The influence of holistic scoring procedures on reading and rating student essays. In Validating holistic scoring for writing assessment: Theoretical and empirical foundations, ed. M.M. Williamson and B.A. Huot, 206-2366. Creskill, NJ: Hampton Press.
17. Linacre, J.M. 2009. A user's guide to FACETS: Rasch model computer program. http://www.winsteps.com.
18. Lumley, T. 2005. Assessing second language writing: The rater's perspective. New York: Peter Lang.
19. McNamara, T. 1996. Measuring second language performance. London: Longman.
20. Myford, C.M., and E.W. Wolf. 2000. Strengthening the ties that bind: Improving the linking network in sparsely connected rating designs. TOEFL Technical Report, No. 15. Princeton, NJ: Educational Testing Service.
21. O'Laughlin, K. 1994. The assessment of writing by English and ESL teachers. Australian Review of Applied Linguistics 17, no. 1: 23-44.
22. Perkins, K. 1983. On the use of composition scoring techniques, objective measures, and objective tests to evaluate ESL writing ability. TESOL Quarterly 17, no. 4: 651-6711.
23. Royal-Dawson, L., and J. Baird. 2009. Is teaching experience necessary for reliable scoring of extended English questions? Educational Measurement: Issues and Practice 28, no. 2: 2-8.
24. Schoonen, R. 2005. Generalisability of writing scores: An application of structural equation modeling. Language Testing 22, no. 1: 1-30.
25. Schoonen, R., M. Vergeer, and M. Eiting. 1997. The assessment of writing ability: Expert readers versus lay readers. Language Testing 14, no. 2: 157-184.
26. Song, C.B., and I. Caruso. 1996. Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students? Journal of Second Language Writing 5, no. 2: 163-182.
27. Sweedler-Brown, C.O. 1985. The influence of training and experience on holistic essay evaluation. English Journal 74, no. 5: 49-55.
28. Weigle, S.C. 1994. Effects of training on raters of ESL compositions. Language Testing 11, no. 2: 197-223.
29. Weigle, S.C. 1998. Using FACETS to model rater training effects. Language Testing 15, no. 2: 263-287.
30. Weigle, S.C. 2002. Assessing writing. Cambridge: Cambridge University Press.
31. Wolfe, E.W. 2006. Uncovering rater's cognitive processing and focus using think-aloud protocols. Journal of Writing Assessment 2: 37-56.
32. Wolfe, E.W., C. Kao, and M. Ranney. 1998. Cognitive differences in proficient and non-proficient essay scorers. Written Communication 15, no. 4: 465-492.