Variability in ESL essay rating processes: The role of the rating scale and rater experience

Language Assessment Quarterly

Volumn 7, Issue 1, 2010, Pages 54-74

  Barkaoui, Khaled ^a  

^a YORK UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77949999471     PISSN: 15434303     EISSN: 15434311     Source Type: Journal    
DOI: 10.1080/15434300903464418     Document Type: Article

References (38)

1. Bacha, N. (2001). Writing evaluation: What can analytic versus holistic essay scoring tell us? System, 29, 371-383
2. Barkaoui, K. (2007a). Participants, texts, and processes in second language writing assessment: A narrative review of the literature. The Canadian Modern Language Review, 64, 97-132
3. Barkaoui, K. (2007b). Rating scale impact on EFL essay marking: A mixed-method study. Assessing Writing, 12, 86-107
4. Barkaoui, K. (2008). Effects of scoring method and rater experience on ESL essay rating processes and outcomes. Unpublished Ph.D. Thesis, University of Toronto
5. Barkaoui, K. (forthcoming). Think-aloud protocols in research on essay rating: An empirical study of their veridicality and reactivity. Language Testing
6. Broad, B. (2003). What we really value: Rubrics in teaching and assessing writing. Logan: Utah State University Press
7. Cumming, A. (1990). Expertise in evaluating second language compositions. Lang-uage Testing, 7, 31-51
8. Cumming, A., Kantor, R., & Powers, D. (2001). Scoring TOEFL essays and TOEFL 2000 prototype writing tasks: An investigation into raters' decision making and development of a preliminary analytic framework (TOEFL Monograph Series N 22). Princeton, NJ: Educational Testing Service
9. Cumming, A., Kantor, R., & Powers, D. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal, 86, 67-96
10. Davison, C. (2004). The contradictory culture of teacher-based assessment: ESL teacher assessment practices in Australia and Hong Kong secondary schools. Language Testing, 21, 305-334
11. Delaruelle, S. (1997). Text type and rater decision-making in the writing module. In G. Brindley & G. Wigglesworth (Eds.), Access: Issues in English language test design and delivery (pp. 215-242). Sydney, Australia: National Center for English Language Teaching and Research, Macquarie University
12. Erdosy, M. U. (2004). Exploring variability in judging writing ability in a second language: A study of four experienced raters of ESL compositions (TOEFL Research Report No. RR-03-17). Princeton, NJ: Educational Testing Service
13. Freedman, S. W., & Calfee, R. C. (1983). Holistic assessment of writing: Experimental design and cognitive theory. In P. Mosenthal, L. Tamor, & S. A. Walmsley (Eds.), Research on writing: Principles and methods (pp. 75-98). New York: Longman
14. Goulden, N. R. (1992). Theory and vocabulary for communication assessments. Communication Education, 41, 258-269
15. Goulden, N. R. (1994). Relationship of analytic and holistic methods to raters' scores for speeches. The Journal of Research and Development in Education, 27, 73-82
16. Hamp-Lyons, L. (1991). Scoring procedures for ESL contexts. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 241-276). Norwood, NJ: Ablex
17. Hamp-Lyons, L., & Kroll, B. (1997). TOEFL 2000-writing: Composition, community and assessment (TOEFL Monograph Series N 5). Princeton, NJ: Educational Testing Service
18. Homburg, T. J. (1984). Holistic evaluation of ESL compositions: Can it be validated objectively? TESOL Quarterly, 18, 87-107
19. Huot, B. A. (1993). The influence of holistic scoring procedures on reading and rating student essays. In M. M. Williamson & B. A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and Empirical foundations (pp. 206-236). Cresskill, NJ: Hampton
20. Lumley, T. (2005). Assessing second language writing: The rater's perspective. New York: Peter Lang
21. Lumley, T., & McNamara, T. F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12, 54-71
22. Milanovic, M., Saville, N., & Shuhong, S. (1996). A study of the decision-making behaviour of composition markers. In M. Milanovic & N. Saville (Eds.), Performance testing, cognition and assessment: Selected papers from the 15th Language Testing Colloquium (LTRC), Cambridge and Arnhem (pp. 92-114) Cambridge, UK: Cambridge University Press
23. Noldus Information Technology b.v. (2003). Observer (Version 5.0). [Computer software]. Wageningen, the Netherlands: Author
24. O'Loughlin, K. (1994). The assessment of writing by English and ESL teachers. Australian Review of Applied Linguistics, 17, 23-44
25. Ruth, L., & Murphy, S. (1988). Designing writing tasks for the assessment of writing. Norwood, NJ: Ablex
26. Sakyi, A. A. (2003). A study of the holistic scoring behaviors of experienced and novice ESL instructors. Unpublished doctoral dissertation, University of Toronto, Toronto, Canada
27. Schoonen, R. (2005). Generalizability of writing scores: An application of structural equation modeling. Language Testing, 22, 1-30
28. Schoonen, R., Vergeer, M., & Eiting, M. (1997). The assessment of writing ability: Expert readers versus lay readers. Language Testing, 14, 157-184
29. Smith, D. (2000). Rater judgments in the direct assessment of competency-based second language writing ability. In G. Brindley (Ed.), Studies in immigrant English language assessment, Volume 1 (pp. 159-189). Sydney, Australia: Macquarie University
30. Song, C. B., & Caruso, I. (1996). Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students? Journal of Second Language Writing, 5, 163-182
31. Torrance, H. (1998). Learning from research in assessment: A response to writing assessment-raters' elaboration of the rating task. Assessing Writing 5, 31-37
32. Vaughan, C. (1991). Holistic assessment: What goes on in the rater's mind? In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 111-125). Norwood, NJ: Ablex
33. Weigle, S. C. (1994). Effects of training on raters of ESL compositions. Language Testing, 11, 197-223
34. Weigle, S. C. (1998). Using FACETS to model rater training effects. Language Testing, 15, 263-287
35. Weigle, S. C. (1999). Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing, 6, 145-178
36. Weigle, S. C. (2002). Assessing writing. Cambridge, UK: Cambridge University Press
37. Wolfe, E. W. (2006). Uncovering rater's cognitive processing and focus using think-aloud protocols. Journal of Writing Assessment, 2, 37-56
38. Wolfe, E. W., Kao, C., & Ranney, M. (1998). Cognitive differences in proficient and non-proficient essay scorers. Written Communication, 15, 465-492