'Little coherence, considerable strain for reader': A comparison between two rating scales for the assessment of coherence

Assessing Writing

Volumn 12, Issue 2, 2007, Pages 108-128

  Knoch, Ute ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Coherence; Multi faceted Rasch analysis; Rating scale validation; Rating scales; Writing assessment

Indexed keywords

EID: 36348939234     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.asw.2007.07.002     Document Type: Article

References (47)

1. Alderson C. Diagnosing foreign language proficiency. The interface between learning and assessment (2005), Continuum, London
2. Bamberg B. Assessing coherence: A reanalysis of essays written for National Assessment of Education Progress. Research in the Teaching of English 18 3 (1984) 305-319
3. Bond T.G., and Fox C.M. Applying the Rasch model: Fundamental measurement in the human sciences (2001), Lawrence Erlbaum, Mahwah, NJ
4. Burneikaité N., and Zabiliúté J. Information structuring in learner texts: A possible relationship between the topical structure and the holistic evaluation of learner essays. Studies about Language 4 (2003) 1-11
5. Cason G.J., and Cason C.L. A deterministic theory of clinical performance rating. Evaluation and the Health Professions 7 (1984) 221-247
6. Cheng X., and Steffensen M.S. Metadiscourse: A technique for improving student writing. Research in the Teaching of English 30 2 (1996) 149-181
7. Connor U., and Farmer F. The teaching of topical structure analysis as a revision strategy for ESL writers. In: Kroll B. (Ed). Second language writing: Research insights for the classroom (1990), Cambridge University Press, Cambridge
8. Crismore A., Markkanen R., and Steffensen M.S. Metadiscourse in persuasive writing: A study of texts written by American and Finnish university students. Written Communication 10 (1993) 39-71
9. Elder C. The DELNA initiative at the University of Auckland. TESOLANZ Newsletter 12 1 (2003) 15-16
10. Elder C., Barkhuizen G., Knoch U., and von Randow J. Evaluating rater responses to an online rater training program. Language Testing 24 1 (2007) 37-64
11. Elder, C., & Erlam, R. (2001). Development and validation of the diagnostic English language needs assessment (DELNA): Final Report. Auckland: University of Auckland, Department of Applied Language Studies and Linguistics.
12. Elder C., Knoch U., Barkhuizen G., and von Randow J. Individual feedback to enhance rater training: Does it work?. Language Assessment Quarterly 2 3 (2005) 175-196
13. Elder, C., & von Randow, J., (2002). Report on the 2002 Pilot of DELNA at the University of Auckland. Auckland: University of Auckland, Department of Applied Language Studies and Linguistics.
14. Fulcher G. Tests of oral performance: The need for data-based criteria. ELT Journal 41 4 (1987) 287-291
15. Fulcher G. Does thick description lead to smart tests? A data-based approach to rating scale construction. Language Testing 13 2 (1996) 208-238
16. Fulcher G. Testing second language speaking (2003), Pearson Longman, London
17. Hamp-Lyons L. Scoring procedures for ESL contexts. In: Hamp-Lyons L. (Ed). Assessing second language writing in academic contexts (1991), Ablex Publishing Corporation, Norwood, NJ
18. Hoenisch, S. (1996). The theory and method of topical structure analysis. Retrieved 30 April 2007, from http://www.criticism.com/da/tsa-method.php
19. Hoey M. Patterns of lexis in text (1991), Oxford University Press, Oxford
20. Intaraprawat P., and Steffensen M.S. The use of metadiscourse in good and poor ESL essays. Journal of Second Language Writing 4 3 (1995) 253-272
21. Jacobs H., Zinkgraf S., Wormuth D., Hartfiel V., and Hughey J. Testing ESL composition: A practical approach (1981), Newbury House, Rowley, MA
22. Knoch U., Read J., and von Randow J. Re-training writing raters online: How does it compare with face-to-face training?. Assessing Writing 12 1 (2007) 26-43
23. Lautamatti L. Observations on the development of the topic of simplified discourse. In: Connor U., and Kaplan R.B. (Eds). Writing across languages: Analysis of L2 text (1987), Addison-Wesley, Reading, MA 87-114
24. Lee I. Teaching coherence to ESL students: a classroom inquiry. Journal of Second Language Writing 11 (2002) 135-159
25. Linacre J.M. Investigating rating scale category utility. Journal of Outcome Measurement 3 2 (1999) 103-122
26. Linacre J.M. Facets Rasch measurement computer program (2006), Winsteps, Chicago
27. Lumley T. Assessment criteria in a large-scale writing test: What do they really mean to the raters?. Language Testing 19 3 (2002) 246-276
28. McIntyre, P. N. (1993). The importance and effectiveness of moderation training on the reliability of teachers' assessments of ESL writing samples. Unpublished master's thesis, University of Melbourne, Australia.
29. Myford C.M., and Wolfe E.W. Detecting and measuring rater effects using many-facet rasch measurement: Part I. Journal of Applied Measurement 4 4 (2003) 386-422
30. Myford C.M., and Wolfe E.W. Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of Applied Measurement 5 2 (2004) 189-227
31. North B. The development of a common framework scale of descriptors of language proficiency based on a theory of measurement. System 23 4 (1995) 445-465
32. North, B. (2003). Scales for rating language performance: Descriptive models, formulation styles, and presentation formats. TOEFL Research Paper. Princeton, NJ: Educational Testing Service.
33. North B., and Schneider G. Scaling descriptors for language proficiency scales. Language Testing 15 2 (1998) 217-263
34. Schneider M., and Connor U. Analyzing topical structure in ESL essays. Studies in Second Language Acquisition 12 4 (1990) 411-427
35. Tribble C. Writing (1996), Oxford University Press, Oxford
36. Turner C.E., and Upshur J.A. Rating scales derived from student samples: Effects of the scale maker and the student sample on scale content and student scores. TESOL Quarterly 36 1 (2002) 49-70
37. Upshur J.A., and Turner C.E. Constructing rating scales for second language tests. ELT Journal 49 1 (1995) 3-12
38. Upshur J.A., and Turner C.E. Systematic effects in the rating of second-language speaking ability: Test method and learner discourse. Language Testing 16 1 (1999) 82-111
39. Watson Todd R. Topic-based analysis of classroom discourse. System 26 (1998) 303-318
40. Watson Todd R., Thienpermpool P., and Keyuravong S. Measuring the coherence of writing using topic-based analysis. Assessing Writing 9 (2004) 85-104
41. Weigle S.C. Effects of training on raters of English as a second language compositions: Quantitative and qualitative approaches (1994), Unpublished doctoral dissertation, University of California, Los Angeles
42. Weigle S.C. Effects of training on raters of ESL compositions. Language Testing 11 2 (1994) 197-223
43. Weigle S.C. Using FACETS to model rater training effects. Language Testing 15 2 (1998) 263-287
44. Witte S. Topical structure analysis and revision: An exploratory study. College Composition and Communication 34 3 (1983) 313-341
45. Witte S. Topical structure and writing quality: Some possible text-based explanations of readers' judgments of students' writing. Visible Language 17 (1983) 177-205
46. Wright B.D., and Masters G.N. Rating scale analysis (1982), MESA Press, Chicago
47. Wu J. Topical structure analysis of English as a second language (ESL) texts written by college Southeast Asian refugee students (1997), Unpublished doctoral dissertation, University of Minnesota