Do ESL essay raters' evaluation criteria change with experience? A mixed-methods, cross-sectional study

TESOL Quarterly

Volumn 44, Issue 1, 2010, Pages 31-57

  Barkaoui, Khaled ^a  

^a YORK UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77950634873     PISSN: 00398322     EISSN: 15457249     Source Type: Journal    
DOI: 10.5054/tq.2010.214047     Document Type: Article

References (38)

1. Barkaoui, K. (2008). Effects of scoring method and rater experience on ESL essay rating processes and outcomes (Unpublished doctoral thesis). University of Toronto, Canada.
2. Breland, H. M., & Jones, R. J.(1984). Perceptions of writing skills. Written Communication, 1, 101-119.
3. Broad, B. (2003). What we really value: Rubrics in teaching and assessing writing. Logan, UT: Utah State University Press.
4. Brown, J. D. (1991). Do English and ESL faculties rate writing samples differently? TESOL Quarterly, 25, 587-603.
5. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York, NY: Academic Press.
6. Connor-Linton, J. (1995). Crosscultural comparison of writing standards: American ESL and Japanese EFL. World Englishes, 14, 99-115.
7. Cumming, A. (1990). Expertise in evaluating second language compositions. Language Testing, 7, 31-51.
8. Cumming, A., Kantor, R., & Powers, D. (2002). Decision making while rating ESL/ EFL writing tasks: A descriptive framework. Modern Language Journal, 86, 67-96.
9. Davison, C. (2004). The contradictory culture of teacher-based assessment: ESL teacher assessment practices in Australian and Hong Kong secondary schools. Language Testing, 21, 305-334.
10. Delaruelle, S. (1997). Text type and rater decision-making in the writing module. In G. Brindley & G. Wigglesworth (Eds.), Access: Issues in English language test design and delivery (pp. 215-242). Sydney, Australia: National Center for English Language Teaching and Research, Macquarie University.
11. Erdosy, M. U. (2004). Exploring variability in judging writing ability in a second language: A study of four experienced raters of ESL compositions (TOEFL Research Report RR-03-17). Princeton, NJ: Educational Testing Service.
12. Field, A. (2005). Discovering statistics using SPSS (2nd ed.). Thousand Oaks, CA: Sage.
13. Frase, L. T., Faletti, J., Ginther, A., & Grant, L. (1999). Computer analysis of the TOEFL test of written English (TOEFL Research Report No. 64). Princeton, NJ: Educational Testing Service.
14. Goulden, N. R. (1994). Relationship of analytic and holistic methods to raters' scores for speeches. The Journal of Research and Development in Education, 27, 73-82.
15. Hamp-Lyons, L. (1989). Raters respond to rhetoric in writing. In H. W. Dechert & M. Raupach (Eds.), Interlingual processes (pp. 229-244). Tubingen, Germany: Gunter Narr Verlag.
16. Hamp-Lyons, L. (1991). Scoring procedures for ESL contexts. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp.241-276). Norwood, NJ: Ablex.
17. Harris, W. (1977). Teacher response to student writing: A study of the response patterns of high-school English teachers to determine the basis for teacher judgment of student writing. Research in the Teaching of English, 11, 175-185.
18. Homburg, T. J. (1984). Holistic evaluation of ESL compositions: Can it be validated objectively? TESOL Quarterly, 18, 87-107.
19. Hox, J. J.(2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum.
20. Kobayashi, H., & Rinnert, C. (1996). Factors affecting composition evaluation in an EFL context: Cultural rhetorical pattern and readers' background. Language Learning, 46, 397-437.
21. Luke, D. (2004). Multilevel modeling. Thousand Oaks, CA: Sage.
22. Lumley, T. (2005). Assessing second language writing: The rater's perspective. New York, NY: Peter Lang.
23. Lumley, T., & McNamara, T. F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12, 54-71.
24. McNamara, T. (1996). Measuring second language performance. London, UK: Longman.
25. Mendelsohn, D., & Cumming, A. (1987). Professors' ratings of language use and rhetorical organization in ESL compositions. TESL Canada Journal, 5, 9-26.
26. Milanovic, M., Saville, N., & Shuhong, S. (1996). A study of the decision-making behaviour of composition markers. In M. Milanovic & N. Saville (Eds.), Performance testing, cognition and assessment: Selected papers from the 15th Language Testing Colloquium (LTRC), Cambridge and Arnhem (pp. 92-114). Cambridge, England: Cambridge University Press.
27. O'Laughlin, K. (1994). The assessment of writing by English and ESL teachers. Australian Review of Applied Linguistics, 17, 23-44.
28. Raudenbush, S. W., Bryk, A. S., Cheong, Y. F., & Congdon, R. (2004). HLM6: Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International.
29. Richards, L.(1999). Using NVivo in qualitative research. Melbourne, Australia: Qualitative Solutions and Research.
30. Rinnert, C., & Kobayashi, H. (2001). Differing perceptions of EFL writing among readers in Japan. The Modern Language Journal, 85, 189-209.
31. Sakyi, A. A. (2003). A study of the holistic scoring behaviors of experienced and novice ESL instructors (Unpublished doctoral dissertation). University of Toronto, Canada.
32. Santos, T. (1988). Professors' reactions to the academic writing of non-native-speaking students. TESOL Quarterly, 22, 69-90.
33. Shi, L. (2001). Native- and nonnative-speaking EFL teachers' evaluation of Chinese students' English writing. Language Testing, 18, 303-325.
34. Song, C. B., & Caruso, I. (1996). Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students? Journal of Second Language Writing, 5, 163-182.
35. Sweedler-Brown, C. O. (1985). The influence of training and experience on holistic essay evaluation. English Journal, 74, 49-55.
36. Tedick, D. J., & Mathison, M. A. (1995). Holistic scoring in ESL writing assessment: What does an analysis of rhetorical features reveal? In D. Belcher & G. Braine (Eds.), Academic writing in a second language: Essays on research and pedagogy (pp. 205-230). Norwood, NJ: Ablex.
37. Weigle, S. C. (1999). Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing, 6, 145-78.
38. Weigle, S. C. (2002). Assessing writing. Cambridge, England: Cambridge University Press.