Rating scale impact on EFL essay marking: A mixed-method study

Assessing Writing

Volumn 12, Issue 2, 2007, Pages 86-107

  Barkaoui, Khaled ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

EFL essay rating; Generalizability theory; Holistic rating; Mixed method research; Multiple trait rating; Think aloud protocols; Writing assessment

Indexed keywords

EID: 36348972265     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.asw.2007.07.001     Document Type: Article

References (40)

1. Bacha N. Writing evaluation: What can analytic versus holistic essay scoring tell us?. System 29 4 (2001) 371-383
2. Bachman, L. F. (1997). Generalizability theory. In: D. Corson (Series Ed.) & C. Clapham (Vol. Ed.), Encyclopaedia of language and education: Vol. 7. Language assessment (pp. 255-262). Dordrecht, Netherlands: Kluwer.
3. Brennan R.L. Generalizability theory (2001), Springer, New York
4. Brown J.D., and Bailey K.M. A categorical instrument for scoring second language writing skills. Language Learning 34 4 (1984) 21-42
5. Carr N. A comparison of the effects of analytic and holistic composition in the context of composition tests. Issues in Applied Linguistics 11 2 (2000) 207-241
6. Cooper C.R. Holistic evaluation of writing. In: Cooper C.R., and Odell L. (Eds). Evaluating writing: Describing, measuring, judging (1977), NCTE, Urbana, IL 3-31
7. Crick J.E., and Brennan R.L. GENOVA: A general-purpose analysis of variance system. Version 2.2 (1984), American College Testing Program, Iowa City, IA
8. Cumming A. Expertise in evaluating second language compositions. Language Testing 7 1 (1990) 31-51
9. Cumming A., Kantor R., and Powers D. Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal 86 1 (2002) 67-96
10. Davidson F. Statistical support for training in ESL composition rating. In: Hamp-Lyons L. (Ed). Assessing second language writing in academic contexts (1991), Ablex, Norwood, NJ 155-164
11. Educational Testing Service. TOEFL: Information bulletin for computer-based testing (2000), Educational Testing Service, Princeton, NJ
12. Erdosy, M. U. (2004). Exploring variability in judging writing ability in a second language: A study of four experienced raters of ESL compositions (TOEFL Research Report RR-03-17). Princeton, NJ: Educational Testing Service.
13. Freedman S.W. How characteristics of essay influence teachers' evaluation. Journal of Educational Psychology 71 3 (1979) 328-338
14. Goulden N.R. Theory and vocabulary for communication assessments. Communication Education 41 3 (1992) 258-269
15. Goulden N.R. Relationship of analytic and holistic methods to rater's scores for speeches. The Journal of Research and Development in Education 27 (1994) 73-82
16. Hamp-Lyons L. Second language writing: Assessment issues. In: Kroll B. (Ed). Second language writing: Research insights for the classroom (1990), Cambridge University Press, Cambridge 69-87
17. Hamp-Lyons L. Scoring procedures for ESL contexts. In: Hamp-Lyons L. (Ed). Assessing second language writing in academic contexts (1991), Ablex, Norwood, NJ 241-276
18. Hamp-Lyons L. Rating nonnative writing: The trouble with holistic scoring. TESOL Quarterly 29 (1995) 759-762
19. Hamp-Lyons L., and Henning G. Communicative writing profiles: An investigation of the transferability of a multiple-trait scoring instrument across ESL writing assessment contexts. Language Learning 41 3 (1991) 337-373
20. Hamp-Lyons, L., & Kroll, B. (1997). TOEFL 2000-writing: Composition, community and assessment (TOEFL Monograph Series No 5). Princeton, NJ: Educational Testing Service.
21. Hughes A. Testing for language teachers (1989), Cambridge University Press, Cambridge
22. Jacobs H.L., Zinkgraf S.A., Wormuth D.R., Hartfiel V.F., and Hughhey J.B. Testing ESL composition: A practical approach (1981), Newbury House, Rowley, MS
23. Lee, Y., & Kantor, R. (2005). Dependability of new ESL writing scores: Evaluating prototype tasks and alternative rating schemes (TOEFL Research Report RR-05-14). Princeton, NJ: Educational Testing Service.
24. Lumley T. Assessment criteria in a large-scale writing test: what do they really mean to the raters?. Language Testing 19 3 (2002) 246-276
25. Lumley T. Assessing second language writing: The rater's perspective (2005), Peter Lang, Frankfurt
26. Lynch B.K. Language program evaluation: Theory and practice (1996), Cambridge University Press, Cambridge
27. Lynch B.K., and McNamara T.F. Using G-theory and Many-Facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing 15 2 (1998) 158-180
28. Milanovic M., Saville N., and Shuhong S. A study of the decision-making behaviour of composition markers. In: Milanovic M., and Saville N. (Eds). Performance testing, cognition and assessment: Selected papers from the 15th Language Testing Colloquium, Cambridge and Arnhem (1996), Cambridge University Press, Cambridge 92-114
29. Perkins K. On the use of composition scoring techniques, objective measures, and objective tests to evaluate ESL writing ability. TESOL Quarterly 17 (1983) 651-671
30. Schaeffer, G., Briel, J. B., & Fowles, M. E. (2001). Psychometric evaluation of the new GRE writing assessment (GRE Research Report RR-01-08). Princeton, NJ: Educational Testing Service.
31. Schoonen R. Generalizability of writing scores: An application of structural equation modeling. Language Testing 22 1 (2005) 1-30
32. Schoonen R., Vergeer M., and Eiting M. The assessment of writing ability: Expert readers versus lay readers. Language Testing 14 2 (1997) 157-184
33. Shavelson R.J., and Webb N.M. Generalizability theory: A primer (1991), Sage, Newbury Park, CA
34. Song C.B., and Caruso I. Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students?. Journal of Second Language Writing 5 2 (1996) 163-182
35. Swartz C.W., Hooper S.R., Montgomery J.W., Wakely M.B., De Kruif R.E.L., Reed M., et al. Using generalizability theory to estimate the reliability of writing scores derived from holistic and analytical scoring methods. Educational and Psychological Measurement 59 (1999) 492-506
36. Sweedler-Brown C.O. The influence of training and experience on holistic essay evaluation. English Journal 74 5 (1985) 49-55
37. Tyndall B., and Kenyon D.M. Validation of a new holistic rating scale using Rasch multifaceted analysis. In: Cumming A., and Berwick R. (Eds). Validation in language testing (1996), Multilingual Matters, Clevedon 39-57
38. Weigle S.C. Assessing writing (2002), Cambridge University Press, Cambridge
39. Weir C.J. Understanding and developing language tests (1993), Prentice Hall, London
40. Wolfe E.W., Kao C., and Ranney M. Cognitive differences in proficient and non-proficient essay scorers. Written Communication 15 (1998) 465-492