Explaining ESL essay holistic scores: A multilevel modeling approach

Language Testing

Volumn 27, Issue 4, 2010, Pages 515-535

  Barkaoui, Khaled ^a  

^a YORK UNIVERSITY   (Canada)

Author keywords

ESL essay rating; essay features; hierarchical data; holistic scoring; multilevel modeling; nested data; rating experience

Indexed keywords

EID: 78049452840     PISSN: 02655322     EISSN: 14770946     Source Type: Journal    
DOI: 10.1177/0265532210368717     Document Type: Article

References (37)

1. Barkaoui K. (2007). Participants, texts, and processes in second language writing assessment: A narrative review of the literature. The Canadian Modern Language Review, 64, 97-132.
2. Barkaoui K. (2008). Effects of scoring method and rater experience on ESL essay rating processes and outcomes. Unpublished PhD thesis, University of Toronto.
3. Bickel R. (2007). Multilevel analysis for applied research: It's just regression. New York: Guilford Press.
4. Breland H., Lee Y., and Muraki E. (2004). Comparability of TOEFL CBT writing prompts: Response mode analyses. TOEFL Research Report RR-04-23. Princeton, NJ: Educational Testing Service.
5. Breland H., Lee Y., Najarian M., and Muraki E. (2004). An analysis of TOEFL CBT writing prompt difficulty and comparability for different gender groups. TOEFL Research Report RR-04-05. Princeton, NJ: Educational Testing Service.
6. Cumming A. (1990). Expertise in evaluating second language compositions. Language Testing, 7, 31-51.
7. Cumming A., Kantor R., and Powers D. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal, 86, 67-96.
8. Eckes T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing 25, 155-185.
9. Frase LT, Faletti J., Ginther A., and Grant L. (1999). Computer analysis of the TOEFL test of written English (TOEFL Research Report N 64). Princeton, NJ: Educational Testing Service.
10. Freedman SW, Calfee RC (1983). Holistic assessment of writing: Experimental design and cognitive theory. In P Mosenthal, L Tamor, and SA Walmsley (Eds), Research on writing: Principles and methods (pp. 75-98). New York: Longman.
11. Goulden NR (1994). Relationship of analytic and holistic methods to raters' scores for speeches. The Journal of Research and Development in Education, 27, 73-82.
12. Grant L., Ginther A. (2000). Using computer-tagged linguistic features to describe L2 writing differences. Journal of Second Language Writing, 9, 123-145.
13. Hamp-Lyons L. (1991). Scoring procedures for ESL contexts. In L Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 241-276). Norwood, NJ: Ablex.
14. Heck RH, Thomas SL (2000). An introduction to multilevel modeling techniques. Mahwah, NJ: Lawrence Erlbaum.
15. Homburg TJ (1984). Holistic evaluation of ESL compositions: Can it be validated objectively? TESOL Quarterly, 18, 87-107.
16. Hox JJ (2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum.
17. Jarvis S., Grant L., Bikowskia D., and Ferris D. (2003). Exploring multiple profiles of highly rated learner compositions. Journal of Second Language Writing, 12, 377-403.
18. Kreft I., de Leeuw J. (1998). Introducing multilevel modeling. Thousand Oaks, CA: Sage.
19. Luke D. (2004). Multilevel modeling. Thousand Oaks, CA: Sage.
20. Lumley T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19, 246-276.
21. Lumley T. (2005). Assessing second language writing: The rater's perspective. New York: Peter Lang.
22. Mendelsohn D., Cumming A. (1987). Professors' ratings of language use and rhetorical organization in ESL compositions. TESL Canada Journal, 5, 9-26.
23. O'Loughlin K. (1994). The assessment of writing by English and ESL teachers. Australian Review of Applied Linguistics, 17, 23-44.
24. Porter D., O'Sullivan B. (1999). The effect of audience age on measured written performance. System, 27, 65-77.
25. Raudenbush SW, Bryk AS (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
26. Raudenbush SW, Bryk AS, Cheong YF and Congdon R. (2004). HLM6: Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International.
27. Rinnert C., Kobayashi H. (2001). Differing perceptions of EFL writing among readers in Japan. The Modern Language Journal, 85, 189-209.
28. Roux AVD (2003). Potentialities and limitations of multilevel analysis in public health and epidemology. In D Courgeau (Ed.), Methodology and epistemology of multilevel analysis: Approaches from different social sciences (pp. 93-119). London: Kluwer Academic.
29. Singer JD, Willett JB (2003). Applied longitudinal data analysis: Modeling change and event occurrence. Oxford: Oxford University Press.
30. Song CB, Caruso I. (1996). Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students? Journal of Second Language Writing, 5, 163-182.
31. Sweedler-Brown CO (1985). The influence of training and experience on holistic essay evaluation. English Journal, 74, 49-55.
32. Sweedler-Brown CO (1993). ESL essay evaluation: The influence of sentence-level and rhetorical features. Journal of second Language Writing, 2, 3-17.
33. Tedick DJ, Mathison MA (1995). Holistic scoring in ESL writing assessment: What does an analysis of rhetorical features reveal? In D Belcher and G Braine (Eds), Academic writing in a second language: Essays on research and pedagogy (pp. 205-230). Norwood, NJ: Ablex.
34. Vacc NN (1989). Writing evaluation: Examining four teachers' holistic and analytic scores. The Elementary School Journal, 90, 87-95.
35. Weigle SC (1994). Effects of training on raters of ESL compositions. Language Testing, 11, 197-223.
36. Weigle SC (1998). Using FACETS to model rater training effects. Language Testing, 15, 263-287.
37. Weigle SC (2002). Assessing writing. Cambridge: Cambridge University Press.