Evaluating the reliability of a detailed analytic scoring rubric for foreign language writing

Assessing Writing

Volumn 14, Issue 2, 2009, Pages 88-115

  East, Martin ^a  

^a UNIVERSITY OF AUCKLAND   (New Zealand)

Author keywords

Rating; Reliability; Scoring; Timed writing tests; Writing assessment

Indexed keywords

EID: 67349144030     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.asw.2009.04.001     Document Type: Article

References (46)

1. Arnold, V., Legas, J., Obler, S., Pacheco, M. A., Russell, C., & Umbdenstock, L. (1990). Do students get higher scores on their word-processed papers? A study of bias in scoring hand-written versus word-processed papers. Unpublished manuscript, Rio Hondo College, Whittier, CA.
2. Bachman L.F. Fundamental considerations in language testing (1990), Oxford University Press, Oxford, UK
3. Barkaoui K. Rating scale impact on EFL essay marking: A mixed-method study. Assessing Writing 12 2 (2007) 86-107
4. Brennan R.L. Generalizability theory (2001), Springer-Verlag, New York
5. Brown, A. (2001). The impact of handwriting on the scoring of essays. Paper presented at the 'European language testing in a global context' conference, Association of Language Testers in Europe (ALTE), 5th-7th July, Barcelona, Spain.
6. Canale M. On some dimensions of language proficiency. In: Oller J.W.J. (Ed). Issues in language testing research (1983), Newbury House, Rowley, MA 333-342
7. Canale M., and Swain M. Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics 1 (1980) 1-47
8. Charney D. The validity of using holistic scoring to evaluate writing: A critical overview. Research in the Teaching of English 18 (1984) 65-81
9. Cherry R., and Meyer P. Reliability issues in holistic assessment. In: Williamson M., and Huot B. (Eds). Validating holistic scoring for writing assessment: Theoretical and empirical foundations (1993), Hampton Press, Cresskill, NJ 109-141
10. Davies W. Essay-writing in German: A students' guide (1997), Manchester University Press, Manchester, UK
11. East M. The impact of bilingual dictionaries on lexical sophistication and lexical accuracy in tests of L2 writing proficiency: A quantitative analysis. Assessing Writing 11 3 (2006) 179-197
12. East M. Bilingual dictionaries in tests of L2 writing proficiency: Do they make a difference?. Language Testing 24 3 (2007) 331-353
13. East M. Dictionary use in foreign language writing exams: Impact and implications (2008), John Benjamins, Amsterdam/Philadelphia, PA
14. Engber C.A. The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing 4 2 (1995) 139-155
15. Edexcel. Specification-Edexcel AS/A GCE in modern foreign languages (2000), Edexcel, London
16. Feldt L.S., and Brennan R.L. Reliability. In: Linn R.L. (Ed). Educational measurement. 3rd ed. (1989), The American Council on Education and the National Council on Measurement in Education, Washington, DC 105-146
17. Grabe W., and Kaplan R. Theory and practice of writing: An applied linguistic perspective (1996), Longman, New York
18. Grobe C. Syntactic maturity, mechanics, and vocabulary as predictors of quality ratings. Research in the Teaching of English 15 (1981) 75-85
19. Hamp-Lyons L. Second language writing: Assessment issues. In: Kroll B. (Ed). Second language writing: Research insights for the classroom (1990), Cambridge University Press, New York 69-87
20. Hamp-Lyons L. Scoring procedures for ESL contexts. In: Hamp-Lyons L. (Ed). Assessing second language writing in academic contexts (1991), Ablex, Norwood, NJ 241-276
21. Hamp-Lyons L. Worrying about rating. Assessing Writing 12 1 (2007) 1-9
22. Hatch E., and Lazaraton A. The research manual: Design and statistics for applied linguistics (1991), Heinle and Heinle, Boston, MA
23. Hurman J., and Tall G. The use of dictionaries in GCSE modern foreign languages written examinations [French] (1998), University of Birmingham School of Education, Birmingham, UK
24. Jacobs H.L., Zinkgraf S.A., Wormuth D.R., Hartfiel V.F., and Hughey J.B. Testing ESL composition: A practical approach (1981), Newbury House, Rowley, MA
25. Knoch U. 'Little coherence, considerable strain for reader': A comparison between two rating scales for the assessment of coherence. Assessing Writing 12 2 (2006) 108-128
26. Laufer B. Lexical frequency profiles: From Monte Carlo to the real world. Applied Linguistics 26 4 (2005) 582-588
27. Laufer B., and Nation P. Vocabulary size and use: Lexical richness in L2 writing. Applied Linguistics 16 3 (1995) 307-322
28. Lumley T. Assessment criteria in a large-scale writing test: What do they really mean to the raters?. Language Testing 19 3 (2002) 246-276
29. Lumley T. Assessing second language writing: The rater's perspective (2005), Peter Lang, Frankfurt am Main, Germany
30. Lumley T., and McNamara T. Rater characteristics and rater bias: Implications for training. Language Testing 12 1 (1995) 54-71
31. Ministry of Education. German in the New Zealand curriculum (2002), Learning Media, Wellington, NZ
32. Moss P. Can there be validity without reliability?. Educational Researcher 23 2 (1994) 5-12
33. Oxford University Language Centre. Online tests (2003). http://www.lang.ox.ac.uk/placement.html Retrieved 3rd February, 2003, from http://www.lang.ox.ac.uk/placement.html
34. Powers D.E., Fowles M.E., Farnum M., and Ramsey P. Will they think less of my handwritten essay if others word process theirs? Effects on essay scores of intermingling handwritten and word-processed essays. Journal of Educational Measurement 31 3 (1994) 220-233
35. Reichelt M. Toward a more comprehensive view of L2 writing: Foreign language writing in the U.S. Journal of Second Language Writing 8 2 (1999) 181-204
36. Reichelt M. A critical review of foreign language writing research on pedagogical approaches. The Modern Language Journal 85 (2001) 578-598
37. Sakyi A.A. Validation of holistic scoring for ESL writing assessment: How raters evaluate compositions. In: Kunnan A.J. (Ed). 19th Language testing research colloquium. Fairness and validation in language assessment (2001), Cambridge University Press, Cambridge, UK 131-152
38. Sasaki M., and Hirose K. Development of an analytic rating scale for Japanese L1 writing. Language Testing 16 4 (1999) 457-478
39. Silva T., and Brice C. Research in teaching writing. Annual Review of Applied Linguistics 24 (2004) 70-106
40. Stemler S.E. A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation 9 (2004) 4. http://pareonline.net/getvn.asp?v=9 Retrieved 6th February, 2006, from http://PAREonline.net/getvn.asp?v=9&n=4
41. Stewart M., and Grobe C. Syntactic maturity, mechanics of writing and teachers' quality ratings. Research in the Teaching of English 13 (1979) 207-215
42. Swain M. Second language testing and second language acquisition: Is there a conflict with traditional psychometrics?. Language Testing 10 (1993) 193-207
43. Vaughan C. Holistic assessment: What goes on in the raters' minds?. In: Hamp-Lyons L. (Ed). Assessing second language writing in academic contexts (1991), Ablex, Norwood, NJ 111-125
44. Weigle S.C. Assessing writing (2002), Cambridge University Press, Cambridge, UK
45. White E.M. Teaching and assessing writing (1985), Jossey-Bass, San Fransisco, CA
46. Wray A. Formulaic sequences in second language teaching: Principle and practice. Applied Linguistics 21 4 (2000) 463-489