Participants, texts, and processes in ESL/EFL essay tests: A narrative review of the literature

Canadian Modern Language Review

Volumn 64, Issue 1, 2007, Pages 99-134

  Barkaoui, Khaled ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Essay raters; Essay tests; Examinees; Rating scales; Variability; Writing tasks

Indexed keywords

EID: 41149173137     PISSN: 00084506     EISSN: None     Source Type: Journal    
DOI: 10.3138/cmlr.64.1.099     Document Type: Article

References (157)

1. Asterisked items are the 70 empirical studies included in the review.
2. *Bacha, N. (2001). Writing evaluation: What can analytic versus holistic essay scoring tell us? System, 29, 371-383.
3. Bachman, L.F. (1990). Fundamental considerations in language testing. Oxford, UK: Oxford University Press.
4. *Breland, H., Lee, Y., & Muraki, E. (2004). Comparability of TOEFL CBT writing prompts: Response mode analyses (TOEFL Research Report RR-04-23). Princeton, NJ: Educational Testing Service.
5. *Breland, H., Lee, Y., Najarian, M., & Muraki, E. (2004). An analysis of TOEFL CBT writing prompt difficulty and comparability for different gender groups (TOEFL Research Report RR-04-05). Princeton, NJ: Educational Testing Service.
6. Broad, B. (2003). What we really value: Rubrics in teaching and assessing writing. Logan, UT: Utah State University Press.
7. Brossell, G. (1986). Current research and unanswered questions in writing assessment. In K.L. Greenberg, H.S. Weiner & R.S. Donovan (Eds.), Writing assessment: Issues and strategies (pp. 168-182). New York: Longman.
8. *Brown, J.D. (1991). Do English and ESL faculties rate writing samples differently?. TESOL Quarterly, 25, 587-603.
9. Brown, J.D., & Bailey, K.M. (1984). A categorical instrument for scoring second language writing skills. Language Learning, 34, 21-42.
10. Brown, J.D., & Hudson, T. (1998). The alternatives in language assessment. TESOL Quarterly, 32, 653-675.
11. Burstein, J., & Chodorow, M. (2002). Directions in automated essay analysis. In Kaplan Robert (Ed.), The Oxford handbook of applied linguistics (pp. 487-497). New York: Oxford University Press.
12. Camp, R. (1993). Changing the model for the direct assessment of writing. In M.M. Williamson & B.A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 45-78). Creskill, NJ: Hampton.
13. *Campbell, C. (1990). Writing with others' words: Using background reading text in academic compositions. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 211-230). Cambridge, UK: Cambridge University Press.
14. *Carr, N. (2000). A comparison of the effects of analytic and holistic composition in the context of composition tests. Issues in Applied Linguistics, 11, 207-241.
15. *Chiste, K.B., & O'Shea, J. (1988). Patterns of question selection and writing performance of ESL students. TESOL Quarterly, 22, 681-684.
16. *Chodorow, M., & Burstein, J. (2004). Beyond essay length: Evaluating e-rater's performance on TOEFL essays (TOEFL Research Report RR-04-04). Princeton, NJ: Educational Testing Service.
17. Christian, D. (2006). Introduction. In F. Genesee, K. Lindholm-Leary, W.M. Saunders & D. Christian (Eds.), Educating English language learners: A synthesis of research evidence (pp. 1-13). Cambridge, UK: Cambridge University Press.
18. Clachar, A. (1999). It's not just cognition: The effect of emotion on multiple-level discourse processing in second-language writing. Language Sciences, 21, 31-60.
19. Coffman, W.E. (1971). Essay examinations. In R.L. Thorndike (Ed.), Educational measurement (2nd ed.), (pp. 271-302). Washington, DC: American Council on Education.
20. *Connor, U. (1991). Linguistic/rhetorical measures for evaluating ESL writing. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 215-225). Norwood, NJ: Ablex.
21. *Connor, U., & Carrell, P.L. (1993). The interpretation of the tasks by writers and readers in holistically rated direct assessment of writing. In J.G. Carson & I. Leki (Eds.), Reading in the composition classroom: Second language perspectives (pp. 141-160). Boston, MA: Heine & Heine.
22. *Connor, U.M., & Kramer, M.G. (1995). Writing from sources: Case studies of graduate students in business management. In D. Belcher & G. Braine (Eds.), Academic writing in a second language: Essays on research and pedagogy (pp. 155-182). Norwood, NJ: Ablex.
23. Connor, U., & Mbaye, A. (2002). Discourse approaches to writing assessment. Annual Review of Applied Linguistics, 22, 263-278.
24. *Connor-Linton, J. (1995). Crosscultural comparison of writing standards: American ESL and Japanese EFL. World Englishes, 14, 99-115.
25. Cooper, C.R. (1977). Holistic evaluation of writing. In C. R. Cooper & L. Odell (Eds.), Evaluating writing: Describing, measuring, judging (pp. 3-31). Urbana, IL: NCTE.
26. Cumming, A. (1989). Writing expertise and language proficiency. Language Learning, 39, 81-141.
27. *Cumming, A. (1990). Expertise in evaluating second language compositions. Language Testing, 7, 31-51.
28. Cumming, A. (1997). The testing of writing in a second language. In C. Clapham & D. Corson (Eds.), Encyclopedia of language and education: Vol. 7. Language Testing and Assessment (pp. 51-63). 7Dordrecht, Netherlands: Kluwer.
29. Cumming, A. (2001). ESL/EFL instructors' practices for writing assessment: Specific purposes or general purposes? Language Testing, 18, 207-224.
30. Cumming, A. (2002). Assessing L2 writing: Alternative constructs and ethical dilemmas. Assessing Writing, 8, 73-83.
31. Cumming, A. (2004). Broadening, deepening, and consolidating. Language Assessment Quarterly, 1, 5-18.
32. *Cumming, A., Kantor, R., Baba, K., Erdosy, U., Eouanzoui, K., & James, M. (2005). Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL. Assessing Writing, 10, 5-43.
33. *Cumming, A., Kantor, R., & Powers, D. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal, 86, 67-96.
34. Cumming, A., Kantor, R., Powers, D., Santos, T., & Taylor, C. (2000). TOEFL 2000 writing framework: A working paper (TOEFL Monograph Series 18). Princeton, NJ: Educational Testing Service.
35. Daly, J.A., & Dickson-Markman, F. (1982). Contrast effects in evaluating essays. Journal of Educational Measurement, 19, 309-316.
36. Davidson, F. (1991). Statistical support for training in ESL composition rating. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 155-164). Norwood, NJ: Ablex.
37. Davies, A., Brown, A., Elder, C., Hill, K., Lumley, T., & McNamara, T. (1999). Dictionary of language testing. Cambridge, UK: Cambridge University Press.
38. DeGruijter, D.N.M. (1980). The essay examination. In L.J.T. vander Kamp, W. F. Langerak & D.N.M. Gruijter (Eds.), Psychometrics for educational debates (pp. 245-262). New York: Willey.
39. *Delaruelle, S. (1997). Text type and rater decision-making in the writing module. In G. Brindley & G. Wigglesworth (Eds.), Access: Issues in English language test design and delivery (pp. 215-242). Sydney, Australia: National Center for English Language Teaching and Research, Macquarie University.
40. *DeRemer, M. (1998). Writing assessment: Raters' elaboration of the rating task. Assessing Writing, 5, 7-29.
41. Deville, G., & Chalhoub-Deville, M. (2006). Old and new thoughts on test score variability: Implications for reliability and validity. In M. Chalhoub-Deville, C.A. Chapelle & P. Duff (Eds.), Inference and generalizability in applied linguistics: Multiple perspectives (pp. 9-25). Amsterdam: Benjamins.
42. Diederich, P.B., French, J.W., & Carlton, S.T. (1961). Factors in the judgment of writing quality. Princeton, NJ: Educational Testing Service.
43. *Engber, C. (1995). The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing, 4, 139-155.
44. *Erdosy, M.U. (2004). Exploring variability in judging writing ability in a second language: A study of four experienced raters of ESL compositions (TOEFL Research Report RR-03-17). Princeton, NJ: Educational Testing Service.
45. *Frase, L.T., Faletti, J., Ginther, A., & Grant, L. (1999). Computer analysis of the TOEFL test of written English (TOEFL Research Report N 64). Princeton, NJ: Educational Testing Service.
46. Freedman, S.W., & Calfee, R.C. (1983). Holistic assessment of writing: Experimental design and cognitive theory. In P. Mosenthal, L. Tamor & S.A. Walmsley (Eds.), Research on writing: Principles and methods (pp. 75-98). New York: Longman.
47. Freedman, S.W. (1979). How characteristics of student essays influence teachers' evaluation. Journal of Educational Psychology, 71, 328-338.
48. Fulcher, G. (2003). Testing second language speaking. New York: Longman.
49. *Gamaroff, R. (2000). Rater reliability in language assessment: The bug of all bears. System, 28, 31-53.
50. Gorman, T.P., Purves, A.C., & Degenhart, R.E. (Eds.). (1988). The IEA study of written composition I: The international writing tasks and scoring scales. New York: Pergamon.
51. Goulden, N.R. (1992). Theory and vocabulary for communication assessments. Communication Education, 41, 258-269.
52. Goulden, N.R. (1994). Relationship of analytic and holistic methods to raters' scores for speeches. Journal of Research and Development in Education, 27, 73-82.
53. *Grant, L., & Ginther, A. (2000). Using computer-tagged linguistic features to describe L2 writing differences. Journal of Second Language Writing, 9, 123-145.
54. Groot, P.J.M. (1990). Language testing in research and education: The need for standards. AILA Review, 7, 9-23.
55. Hall, E. (1991). Variations in composing behaviors of academic ESL writers in test and non-test situations. TESL Canada Journal, 8, 9-33.
56. *Hamp-Lyons, L. (1989). Raters respond to rhetoric in writing. In H.W. Dechert & M. Raupach (Eds.), Interlingual processes (pp. 229-244). Tubingen, Germany: Gunter Narr Verlag.
57. Hamp-Lyons, L. (1990). Second language writing: Assessment issues. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 69-87). Cambridge, UK: Cambridge University Press.
58. Hamp-Lyons, L. (1991a). Basic concepts. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 1-15). Norwood, NJ: Ablex.
59. Hamp-Lyons, L. (1991b). Scoring procedures for ESL contexts. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 241-276). Norwood, NJ: Ablex.
60. Hamp-Lyons, L. (1995). Rating non-native writing: The trouble with holistic scoring. TESOL Quarterly, 29, 759-62.
61. Hamp-Lyons, L. (1996). Applying ethical standards to portfolio assessment of writing in English as a second language. In M. Milanovic & N. Saville (Eds.), Performance testing, cognition and assessment: Selected papers from the 15th Language Testing Research Colloquium (pp. 151-164). Cambridge, UK: Cambridge University Press.
62. Hamp-Lyons, L. (2003). Writing teachers as assessors of writing. In B. Kroll (Ed.), Exploring the dynamics of second language writing (pp. 162-189). Cambridge, UK: Cambridge University Press.
63. Hamp-Lyons, L., & Condon, W. (2000). Assessing the portfolio: Principles for practice, theory and research. Cresskill, NJ: Hampton.
64. Hamp-Lyons, L., & Henning, G. (1991). Communicative writing profiles: An investigation of the transferability of a multiple-trait scoring instrument across ESL writing assessment contexts. Language Learning, 41, 337-373.
65. Hamp-Lyons, L., & Kroll, B. (1996). Issues in ESL writing assessment: An overview. College ESL, 6, 52-72.
66. Hamp-Lyons, L., & Kroll, B. (1997). TOEFL 2000-writing: Composition, community and assessment. (TOEFL Monograph Series N 5). Princeton, NJ: Educational Testing Service.
67. *Hamp-Lyons, L., & Mathias, S.P. (1994). Examining expert judgments of task difficulty on essay tests. Journal of Second Language Writing, 3, 85-96.
68. *Hamp-Lyons, L., & Zhang, B.W. (2001). World Englishes: Issues in and from academic writing assessment. In J. Flowerdew & M. Peacock (Eds.), Research perspectives on English for academic purposes (pp. 101-116). Cambridge, UK: Cambridge University Press.
69. Henning, G. (1987). A guide to language testing: Development, evaluation and research. New York: Newbury House.
70. *Hill, K. (1997). Who should be the judge? The use of non-native speakers as raters on a test of English as an international language. In A. Huhta, V. Kohonen, L. Kurki-Suonio & S. Luoma (Eds.), Current developments and alternatives in language assessment: Proceedings of LTRC 96 (pp. 275-290). Jyvaskyla, Finland: University of Jyvaskyla.
71. *Hinkel, E. (2002). Second language writers' text: Linguistic and rhetorical features. Mahwah, NJ: Erlbaum.
72. *Homburg, T.J. (1984). Holistic evaluation of ESL compositions: Can it be validated objectively?. TESOL Quarterly, 18, 87-107.
73. Huot, B. (1993). The influence of holistic scoring procedures on reading and rating student essays. In M.M. Williamson & B.A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and Empirical foundations (pp. 206-236). Creskill, NJ: Hampton.
74. Huot, B. (2002). (Re)Articulating writing assessment for teaching and learning. Logan, UT: Utah University Press.
75. Jacobs, H.L., Zinkgraf, S.A., Wormuth, D.R., Hartfiel, V.F., & Hughey, J.B. (1981). Testing ESL composition: A practical approach. Rowley, MA: Newbury House.
76. *Janopoulos, M. (1992). University faculty tolerance of NS and NNS writing errors: A comparison. Journal of Second Language Writing, 1, 109-121.
77. *Jarvis, S., Grant, L., Bikowskia, D., & Ferris, D. (2003). Exploring multiple profiles of highly rated learner compositions. Journal of Second Language Writing, 12, 377-403.
78. *Jennings, M., Fox, J., Graves, B., & Shohamy, E. (1999). The test-takers' choice: An investigation of the effect of topic on language-test performance. Language Testing, 16, 426-456.
79. *Khalil, A. (1985). Communicative error evaluation: Native speakers' evaluation and interpretation of written errors of Arab EFL learners. TESOL Quarterly, 19, 335-351.
80. *Kobayashi, H., & Rinnert, C. (1996). Factors affecting composition evaluation in an EFL context: Cultural rhetorical pattern and readers' background. Language Learning, 46, 397-437.
81. *Kobayashi, T. (1992). Native and nonnative reactions to ESL compositions. TESOL Quarterly, 26, 81-112.
82. Kondo-Brown, K. (2002). A FACETS analysis of rater bias in measuring Japanese second language writing performance. Language Testing, 19, 3-31.
83. Kroll, B. (1990). What does time buy? ESL student performance on home versus class compositions. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 140-54). Cambridge, UK: Cambridge University Press.
84. Kroll, B. (1998). Assessing writing abilities. Annual Review of Applied Linguistics, 18, 219-240.
85. Kroll, B., & Reid, J. (1994). Designing and assessing effective classroom writing assignments for NES and ESL students. Journal of Second Language Writing, 4, 17-41.
86. *Kunnan, A.J. (1990). DIF in native language and gender groups in an ESL placement test. TESOL Quarterly, 24, 741-746.
87. *Lee, H.K. (2004). A comparative study of ESL writers' performance in a paper-based and a computer-delivered writing test. Assessing Writing, 9, 4-26.
88. *Lee, Y. (2002). A comparison of composing processes and written products in timed-essay tests across paper-and-pencil and computer modes. Assessing Writing, 8, 135-157.
89. *Lee, Y., Breland, H., & Muraki, E. (2004). Comparability of TOEFL CBT writing prompts for different native language groups (TOEFL Research Report RR-04-24). Princeton, NJ: Educational Testing Service.
90. *Leki, I. (1995). Good writing: I know it when I see it. In D. Belcher & G. Braine (Eds.), Academic writing in a second language: Essays on research and pedagogy (pp. 23-46). Norwood, NJ: Ablex.
91. *Li, J. (2006). The mediation of technology in ESL writing and its implications for writing assessment. Assessing Writing, 11, 5-21.
92. Lloyd-Jones, R. (1977). Primary trait scoring. In C.R. Cooper & L. Odell (Eds.), Evaluating writing: Describing, measuring, judging (pp. 33-66). Urbana, IL: NCTE.
93. Lukmani, Y. (1996). Linguistic accuracy versus coherence in assessing examination answers in content subjects. In M. Milanovic & N. Saville (Eds.), Performance testing, cognition and assessment (pp. 130-50). Cambridge, UK: Cambridge University Press.
94. *Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19, 246-76.
95. Lumley, T. (2005). Assessing second language writing: The rater's perspective. New York: Lang.
96. Luoma, S., & Tarnanen, M. (2003). Creating a self-rating instrument for second language writing: From idea to implementation. Language Testing, 20, 440-465.
97. McNamara, T. (1996). Measuring second language performance. London, UK: Longman.
98. McNamara, T., & Roever, C. (2006). Language testing: The social dimension. Oxford: Blackwell.
99. *Mendelsohn, D., & Cumming, A. (1987). Professors' ratings of language use and rhetorical organization in ESL compositions. TESL Canada Journal, 5, 9-26.
100. *Milanovic, M., Saville, N., & Shuhong, S. (1996). A study of the decisionmaking behaviour of composition markers. In M. Milanovic & N. Saville (Eds.), Performance testing, cognition and assessment (pp. 92-114). Cambridge, UK: Cambridge University Press.
101. Moss, PA. (1992). Shifting conceptions of validity in educational measurement: Implications for performance assessment. Review of Educational Research, 62, 229-258.
102. Moss, P.A. (1994). Can there be validity without reliability? Educational Researcher, 23, 5-12.
103. Moss, P.A. (1996). Enlarging the dialogue in educational measurement: Voices from interpretive research traditions. Educational Researcher, 25, 20-28.
104. *O'Laughlin, K. (1994). The assessment of writing by English and ESL teachers. Australian Review of Applied Linguistics, 17, 23-44.
105. Oscarson, M. (1997). Self-assessment of foreign and second language proficiency. In C. Clapham & D. Corson (Eds.), Encyclopedia of language and education: Vol. 7. Language testing and assessment (pp. 175-187). Dordrecht: Kluwer.
106. *Park, Y.M. (1988). Academic and ethnic background as factors affecting writing performance. In A.C. Purves (Ed.), Writing across languages and cultures (pp. 261-272). Beverly Hills, CA: Sage.
107. *Perkins, K. (1980). Using objective methods of attained writing proficiency to discriminate among holistic evaluations. TESOL Quarterly, 14, 61-69.
108. Perkins, K. (1983). On the use of composition scoring techniques, objective measures, and objective tests to evaluate ESL writing ability. TESOL Quarterly, 17, 651-671.
109. Pilliner, A.E.G. (1968). Subjective and objective testing. In A. Davies (Ed.), Language testing symposium: A psycholinguistic approach (pp. 19-35). Oxford, UK: Oxford University Press.
110. Polio, C., Heck, C., & Leder, N. (1998). 'If I only had more time': ESL learners' changes in linguistic accuracy on essay revisions. Journal of Second Language Writing, 7, 43-68.
111. *Polio, C., & Glew, M. (1996). ESL writing assessment prompts: How students choose. Journal of Second Language Writing, 5, 35-49.
112. *Porter, D., & O'Sullivan, B. (1999). The effect of audience age on measured written performance. System, 27, 65-77.
113. Pula, J.J., & Huot, B.A. (1993). A model of background influences on holistic raters. In M.M. Williamson & B.A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and Empirical foundations (pp. 237-265). Creskill, NJ: Hampton.
114. Purves, A.C. (1992a). Reflections on research and assessment in written composition. Research in the Teaching of English, 26, 108-122.
115. Purves, A.C. (Ed.), (1992b). The IEA study of written composition II: Education and performance in fourteen countries. New York: Pergamon.
116. Raimes, A. (1987). Language proficiency, writing ability, and composing strategies: A study of ESL student writers. Language Learning, 37, 439-469.
117. *Reid, J. (1990). Responding to different topic types: A quantitative analysis from a contrastive rhetoric perspective. In B. Kroll. (Ed.), Second language writing: Research insights for the classroom (pp. 191-210). Cambridge, UK: Cambridge University Press.
118. *Rinnert, C., & Kobayashi, H. (2001). Differing perceptions of EFL writing among readers in Japan. Modern Language Journal, 85, 189-209.
119. Ruth, L., & Murphy, S. (1988). Designing writing tasks for the assessment of writing. Norwood, NJ: Ablex.
120. *Santos, T. (1988). Professors' reactions to the academic writing of non-native-speaking students. TESOL Quarterly, 22, 69-90.
121. Sasaki, M., & Hirose, K. (1999). Development of an analytic rating scale for Japanese L1 writing. Language Testing, 16, 457-478.
122. *Scarcella, R.C. (1984). How writers orient their readers in expository essays: A comparative study of native and non-native English writers. TESOL Quarterly, 18, 671-88.
123. *Schneider, M., & Connor, U. (1990). Analyzing topical structure in ESL essays: Not all topics are equal. Studies in Second Language Acquisition, 12, 411-427.
124. Schoonen, R. (2005). Generalizability of writing scores: An application of structural equation modeling. Language Testing, 22, 1-30.
125. Schoonen, R., Vergeer, M., & Eiting, M. (1997). The assessment of writing ability: Expert readers versus lay readers. Language Testing, 14, 157-184.
126. *Shi, L. (2001). Native- and nonnative-speaking EFL teachers' evaluation of Chinese students' English writing. Language Testing, 18, 303-325.
127. Shohamy, E. (2001). The power of tests: A critical perspective on the uses of language tests. New York: Longman.
128. *Shohamy, E., Gordon, C.M., & Kraemer, R. (1992). The effect of raters' background and training on the reliability of direct writing tests. Modern Language Journal, 76, 27-33.
129. *Smith, D. (2000). Rater judgments in the direct assessment of competency-based second language writing ability. In G. Brindley (Ed.), Studies in immigrant English language assessment: Vol. 1 (pp. 159-189). Sydney: Macquarie University Press.
130. Smith, W.L. (1993). Assessing the reliability and adequacy of using holistic scoring of essays as a college composition placement technique. In M.M. Williamson & B.A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 142-205). Creskill, NJ: Hampton.
131. Song, C.B., & Caruso, I. (1996). Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students? Journal of Second Language Writing, 5, 163-182.
132. *Spaan, M. (1993). The effect of prompt on essay examinations. In D. Douglas & C. Chapelle (Eds.), A new decade of language testing research (pp. 98-122). Alexandria, VA: TESOL.
133. Swain, M. (1993). Second language testing and second language acquisition: Is there a conflict with traditional psychometrics? Language Testing, 10, 193-207.
134. Swartz, C.W., Hooper, S.R., Montgomery, J.W., Wakely, M.B., De Kruif, R.E.L., Reed, M., Brown, T.T., Levine, M.D., & White, K.P. (1999). Using generalizability theory to estimate the reliability of writing scores derived from holistic and analytical scoring methods. Educational and Psychological Measurement, 59, 492-506.
135. Sweedler-Brown, C.O. (1985). The influence of training and experience on holistic essay evaluation. English Journal, 74, 49-55.
136. *Sweedler-Brown, C.O. (1993a). ESL essay evaluation: The influence of sentence-level and rhetorical features. Journal of Second Language Writing, 2, 3-17.
137. *Sweedler-Brown, CO. (1993b). The effects of ESL errors on holistic scores assigned by English composition faculty. College ESL, 3, 53-69.
138. *Tedick, D.J. (1990). ESL writing assessment: Subject-matter knowledge and its impact on performance. English for Specific Purposes, 9, 123-143.
139. *Tedick, D.J., & Mathison, M.A. (1995). Holistic scoring in ESL writing assessment: What does an analysis of rhetorical features reveal? In D. Belcher & G. Braine (Eds.), Academic writing in a second language: Essays on research and pedagogy (pp. 205-230). Norwood, NJ: Ablex.
140. Turner, C., & Upshur, J. (1996). Developing rating scales for the assessment of second language performance. In G. Wigglesworth & C. Elder (Eds.), The language testing cycle: From inception to washback (pp. 55-79). Melbourne: Australian Review of Applied Linguistics.
141. Turner, C.E. (2000). Listening to the voices of rating scale developers: Identifying salient features for second language performance assessment. The Canadian Modem Language Review, 56, 555-584.
142. Underhill, N. (1982). The great reliability validity trade-off: Problems in assessing the productive skills. In B. Heaton (Ed.), Language testing (pp. 17-23). Hayes, UK: Modern English Publications.
143. Upshur, J.A., & Turner, C.E. (1999). Systematic effects in the rating of second-language speaking ability: Test method and learner discourse. Language Testing, 16, 82-111.
144. *Vann, R.J., Lorenz, F.O., & Meyer, D.M. (1991). Error gravity: Faculty response to errors in the written discourse of nonnative speakers of English. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 181-195). Norwood, NJ: Ablex.
145. *Vann, R.J., Meyer, D.M., & Lorenz, F.O. (1984). Error gravity: A study of faculty opinions of ESL errors. TESOL Quarterly, 18, 427-440.
146. *Vaughan, C. (1991). Holistic assessment: What goes on in the rater's mind? In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 111-125). Norwood, NJ: Ablex.
147. Weigle, S.C. (1998). Using FACETS to model rater training effects. Language Testing, 15, 263-87.
148. *Weigle, S.C. (1999). Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing, 6, 145-178.
149. Weigle, S.C. (2002). Assessing writing. Cambridge, UK: Cambridge University Press.
150. White, E.M. (1993). Holistic scoring: Past triumphs, future challenges. In M.M. Williamson & B.A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 79-108). Creskill, NJ: Hampton.
151. Wilhelm, K.H. (1996). Combined assessment model for EAP writing workshop: Portfolio decision-making, criterion-referenced grading, and contract negotiation. TESL Canada Journal, 14, 21-33.
152. Williamson, M.M. (1993). An introduction to holistic scoring: Historical and theoretical context for writing assessment. In M.M. Williamson & B.A. Huot (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 1-43). Creskill, NJ: Hampton.
153. Wolfe, E.W., Kao, C., & Ranney, M. (1998). Cognitive differences in proficient and non-proficient essay scorers. Written Communication, 15, 465-492.
154. *Wolfe, E.W., & Manalo, J.R. (2005). An investigation of the impact of composition medium on the quality of TOEFL writing scores (TOEFL Research Report RR-04-29). Princeton, NJ: Educational Testing Service.
155. Wood, R. (1991). Assessment and testing: A survey of research commissioned by the University of Cambridge Local Examination Syndicate. Cambridge, UK: Cambridge University Press.
156. *Yeh, S.S. (1998). Validation of a scheme for assessing argumentative writing of middle school students. Assessing Writing, 5, 123-150.
157. *Zhang, S. (1987). Cognitive complexity and written production in English as a second language. Language Learning, 37, 469-481.