Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches

Assessing Writing

Volumn 6, Issue 2, 1999, Pages 145-178

  Weigle, Sara Cushing ^a  

^a GEORGIA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0043206862     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/S1075-2935(00)00010-6     Document Type: Article

References (60)

1. Bachman, L. F., Lynch, B. K., & Mason, M. (1995). Investigating variability in tasks and rater judgements in a performance test of foreign language speaking. Language Testing, 12 (2), 238-57.
2. Brossell, G. (1983). Rhetorical specification in essay topics. College English, 45, 165-73.
3. Brown, J. D. (1991). Do English and ESL faculties rate writing samples differently? TESOL Quarterly, 25 (4), 587-603.
4. Connor, U., & Carrell, P. (1993). The interpretation of tasks by writers and readers in holistically rated direct assessment of writing. In J. G. Carson and I. Leki, Reading in the composition classroom: Second language perspectives (pp. 141-160). Boston, MA: Heinle & Heinle.
5. Charney, D. (1984). The validity of using holistic scoring to evaluate writing. Research in the teaching of English, 18, 65-81.
6. Cohen, A. D. (1984). On taking language tests: What the students report. Language Testing, 1 (1), 70-81.
7. Connor-Linton, J. (1995). Looking behind the curtain: What do L2 composition ratings really mean? TESOL Quarterly, 29, 762-65.
8. Crowhurst, M. (1980). Syntactic complexity and teachers' ratings of narrations and arguments. Research in the Teaching of English, 13, 223-31.
9. Cumming, A. (1990). Expertise in evaluating second language compositions. Language Testing, 7, 31-51.
10. Diederich, P. B. (1974). Measuring growth in English. Urbana, IL: National Council of Teachers of English.
11. Du, Y., Wright, B., & Brown, W. L. (1997). Raters and single prompt-to-prompt equating using the Facets model in a writing performance assessment. Paper presented at the International Objective Measurement Conference, Chicago, IL. Eric Document #ED410291.
12. Engelhard, G. (1992). The measurement of writing with a many-faceted Rasch model. Applied Measurement in Education, 5 (3), 71-91.
13. Engelhard, G. (1996). Evaluating rater accuracy in performance assessments. Journal of Educational Measurement, 33 (1), 56-70.
14. Ericsson, K. A., & Simon, H. (1980). Verbal reports as data. Psychological Review, 87, 215-251.
15. Ericsson, K. A., & Simon, H. (1984) Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
16. Flower, L. S., & Hayes, J. R. (1980). The dynamics of composing: Making plans and juggling constraints. In L. W. Gregg & E. R. Steinberg, (Eds.), Cognitive processes in writing, (pp. 31-50). Hillsdale, NJ: Lawrence Erlbaum Associates.
17. Freedman, S. W., & Calfee, R. C. (1983). Holistic assessment of writing: Experimental design and cognitive theory. In P. Mosenthal, L. Tamor, & S. Walmsley (Eds.), Research on writing: Principles and methods (pp. 75-98). New York, NY: Longman.
18. Hake, R. (1986). How do we judge what they write. In K. L. Greenberg, H. S. Wiener, & R. A. Donovan (Eds.), Writing assessment: Issues and strategies (pp. 153-167). New York: Longman.
19. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage Publications.
20. Hamp-Lyons, L. (1991). Scoring procedures for ESL contexts. In Hamp-Lyons, L. (Ed.), Assessing second language writing in academic contexts (pp. 241-276). Norwood, NJ: Ablex Publishing Corp.
21. Hamp-Lyons, L., & Mathias, S. P. (1994). Examining experts' judgements of task difficulty of essay tests. Journal of Second Language Writing, 3, 49-68.
22. Hatch, E., & Lazaraton, A. (1991). The research manual: Design and statistics for applied linguistics. New York: Newbury House Publishers.
23. Hayes, J. R., & Flower, L. S. (1980). Identifying the organization of writing processes. In L. W. Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 3-30). Hillsdale, NJ: Lawrence Erlbaum Associates.
24. Hoetker, J. (1982). Essay examination topics and student writing. College Composition and Communication, 33, 377-92.
25. Huot, B. (1988). The validity of holistic scoring: A comparison of the talk-aloud protocols of expert and novice holistic raters. Unpublished manuscript.
26. Huot, B. (1990a). The literature of direct writing assessment: Major concerns and prevailing trends. Review of Educational Research, 60, 237-63.
27. Huot, B. (1990b). Reliability, validity, and holistic scoring: What we know and what we need to know. College Composition and Communication, 41, 201-13.
28. Kroll, B., & Reid, J. (1994). Guidelines for designing writing prompts: clarifications, caveats and cautions. Journal of Second Language Writing, 3 (3), 231-55.
29. Linacre, J. M. (1989). Many-faceted Rasch measurement. Chicago, IL: MESA Press.
30. Linacre, J.M. & Wright, B.D. (1992). A user's guide to FACETS (Version 2.6). Chicago, IL: MESA Press.
31. Linacre, J. M., & Wright, B. D. (1993). A user's guide to FACETS (Version 2.7). Chicago, IL: MESA Press.
32. Lloyd-Jones, R. (1977). Primary trait scoring. In C.R. Cooper & L. Odell (eds.), Evaluating writing: Describing, measuring, judging (pp. 33-66). Urbana, IL: NCTE.
33. Lunz, M. E., & Stahl, J. A. (1990). Judge consistency and severity across grading periods. Evaluation and the Health Professions, 13, 425-44.
34. Lunz, M. E., Stahl, J. A., Wright, B. D., & Linacre, J. M. (1989). Variation among examiners and protocols on oral examinations. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
35. Lunz, M. E., Wright, B. D., & Linacre, J. M. (1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3 (4), 331-45.
36. McNamara, T. F. (1996). Measuring second language performance. New York: Addison Wesley Longman Limited.
37. McNamara, T. F., & Adams, R. J. (1991). Exploring rater behavior with Rasch techniques. Paper presented at the Language Testing Research Colloquium, Princeton, NJ.
38. Mendelsohn, D., & Cumming, A. (1987). Professors' ratings of language use and rhetorical organization in ESL compositions. TESL Canada Journal, 5(1), 9-26.
39. Miles, M., & Huberman, M. (1983). Qualitative data analysis. Beverly Hills, CA: Sage Publications.
40. Myford, C. M. (1991a). Assessment of acting ability. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
41. Myford, C. M. (1991b). Judging acting ability: the transition from the novice to expert. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
42. Nisbett, R. E., & Wilson, W. T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-59.
43. Polio, C., & Glew, M. (1996). ESL writing assessment prompts: How students choose. Journal of Second Language Writing, 5 (1), 35-49.
44. Powers, D., Fowles, M., Farnum, M., & Ramsey, P. (1994). Will they think less of my handwritten essay if others word process theirs? Effects on essay scores of intermingling handwritten and word processed essays. Journal of Educational Measurement, 31 (3), 220-33.
45. Pula, J. J., & Huot, B. A. (1993). A model of background influences on holistic raters. In M. M. Williamson and B. A. Huot (Eds.), Validating holistic scoring for writing assessment. Theoretical and empirical foundations (pp. 237-65). Cresskill, NJ: Hampton Press, Inc.
46. Purves, A. (1992). Reflection on research and assessment in written composition. Research in the Teaching of English, 26, 108-22.
47. Quellmalz, E. S., Capell, F., & Chou, C. P. (1982). Effects of discourse and response mode on the measurement of writing competence. Journal of Educational Measurement, 19, 241-58.
48. Reid, J. (1990). Responding to different topic types: A quantitative analysis from a contrastive rhetoric perspective. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 191-210). Cambridge: Cambridge University Press.
49. Santos, T. (1988). Professors' reactions to the academic writing of nonnative-speaking students. TESOL Quarterly, 22 (1), 69-90.
50. Shohamy, E., Gordon, C., & Kramer, R. (1992). The effect of raters' background and training on the reliability of direct writing tests. Modern Language Journal, 76 (1), 27-33.
51. Smagorinsky, P. (1989). The reliability and validity of protocol analysis. Written Communication, 6 (4), 463-79.
52. Smith, W.L., Hull, G.A., Land, R.E., Moore, M.T., Ball, C., Dunham, D.E., Hickey, L.S., & Ruzich, C.W. (1985). Some effects of varying the structure of a topic on college students' writing. Written Communication, 2, 73-89.
53. Stansfield, C. W., & Kenyon, D. M. (1992). Comparing the scaling of speaking tasks by language teachers and by ACTFL guidelines. Paper presented ta the Language Testing Research Colloquium, Vancouver, B.C.
54. Stock, P. L., & Robinson, J. L. (1987). Taking on testing: Teachers as researchers. English Education, 19, 93-121.
55. Strauss, A. (1987). Qualitative analysis for social scientists. Cambridge: Cambridge University Press.
56. Sweedler-Brown, C. (1993). ESL essay evaluation: The influence of sentence-level and rhetorical features. Journal of Second Language Writing, 2 (1), 3-17.
57. Vaughan, C. (1992). Holistic assessment: What goes on in the rater's mind? In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 111-26). Norwood, NJ: Ablex.
58. Weigle, S. (1994a). Effects of training on raters of ESL compositions: Quantitative and qualitative approaches. PhD dissertation, University of California, Los Angeles.
59. Weigle, S. (1994b). Effects of training on raters of ESL compositions. Language Testing, 11, 197-223.
60. Weigle, S. (1998). Using FACETS to model rater training effects. Language Testing, 15 (2), 263-87.