Complementing human judgment of essays written by English language learners with e-rater® scoring

Language Testing

Volumn 27, Issue 3, 2010, Pages 317-334

  Enright, Mary K ^a   Quinlan, Thomas ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

automated essay scoring; e rater; educational measurement; ESL EFL writing assessment; TOEFL writing; validity of writing assessment

Indexed keywords

EID: 77955402806     PISSN: 02655322     EISSN: 14770946     Source Type: Journal    
DOI: 10.1177/0265532210363144     Document Type: Article

References (44)

1. Attali Y. (2007). Construct validity of e-rater in scoring TOEFL essays. (ETS Research Report No. RR-07-21). Princeton, NJ: ETS.
2. Attali Y. (submitted for publication). E-rater performance for TOEFL iBT Independent essays.
3. Attali Y., Burstein J. (2006). Automated essay scoring with e-rater V. 2.0. The Journal of Technology, Learning, and Assessment, 4(3): 13-18.
4. Bachman LF (2005). Building and supporting a case for test use. Language Assessment Quarterly, 2(1): 1-34.
5. Bennett RE (2006). Moving the field forward: Some thoughts on validity and automated scoring. In DM Williamson, RJ Mislevy, II Bejar (Eds.), Automated scoring of complex tasks in computer-based testing (pp. 403-412). Mahwah, NJ: Lawrence Erlbaum.
6. Breland HM, Bridgeman B., and Fowles ME (1999). Writing assessment in admission to higher education: Review and framework. (College Board Report No. 99-3). New York: College Entrance Examination Board.
7. Burstein J., Chodorow M. (1999, June). Automated essay scoring for nonnative English speakers. In Proceedings of the ACL99 workshop on computer-mediated language assessment and evaluation of natural language processing. College Park, MD.
8. Burstein J., Marcu D., and Knight K. (2003). Finding the WRITE stuff: Automatic identification of discourse structure in student essays. In S Harabagiu, F Ciravegna (Eds.), IEEE Intelligent Systems: Special Issue on Advances in Natural Language Processing, 18(1): 32-39.
9. Chapelle CA, Enright MK, and Jamieson JM (Eds.) (2008). Building a validity argument for the Test of English as a Foreign Language. New York: Routledge.
10. Chapelle CA, Enright MK, and Jamieson JM (in press). Does an argument-based approach to validity make a difference? Educational Measurement: Issues and Practices.
11. Chenoweth N., Hayes JR (2001). Fluency in writing. Written Communication, 18(1): 80-98.
12. Chodorow M., Burstein J. (2004). Beyond essay length: Evaluating e-rater. (TOEFL Research Report No. 73; ETS RR-04-04). Princeton, NJ: ETS.
13. Cohen J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
14. Conference on College Composition and Communication (2004, February). CCCC position statement on teaching, learning, and assessing writing in digital environments. Retrieved 14 January 2009 from http://www.ncte.org/cccc/resources/positions/digitalenvironments.
15. College Entrance Examination Board (2003). The neglected 'R': The need for a writing revolution. Report of the National Commission on Writing in America's Schools and Colleges. New York: Author.
16. Cronbach L. (1988). Five perspectives on validity argument. In H Wainer and H Braun (Eds.), Test validity (pp. 3-17). Hillsdale, NJ: Lawrence Erlbaum.
17. Cumming A., Kantor R., and Powers DE (2001). Scoring TOEFL® essays and TOEFL 2000 prototype writing tasks: An investigation into raters' decision making and development of a preliminary analytic framework (TOEFL Monograph No. TOEFL-MS-22). Princeton, NJ: Educational Testing Service.
18. Cumming A., Kantor R., Powers D., Santos T., and Taylor C. (2000). TOEFL 2000 Writing Framework: A working paper. (TOEFL Monograph No. TOEFL-MS-18). Princeton, NJ: Educational Testing Service.
19. Eckes T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2): 155-185.
20. Educational Testing Service (2004). Scoring guides (rubrics) for writing responses (PDF). Retrieved 8 December 2009, from http://www.ets.org/Media/Tests/TOEFL/pdf/Writing_Rubrics.pdf.
21. Gregg N., Coleman C., Davis M., and Chalk JC (2007). Timed essay writing: Implications for high-stakes tests. Journal of Learning Disabilities, 40(4): 306.
22. Hamp-Lyons L (Ed.). (1991a). Assessing second language writing in academic contexts. Norwood, NJ: Ablex.
23. Hamp-Lyons L. (1991b). Scoring procedures for ESL Contexts. In L Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 241-276). Norwood, NJ: Ablex.
24. Huot B. (1990). The literature of direct writing assessment: Major concerns and prevailing trends. Review of Educational Research, 60, 237-263.
25. Janopoulos M. (1993). Comprehension, communicative competence, and construct validity: Holistic scoring from an ESL perspective. In MW Williamson, BA Huot (Eds.), Validating holistic scoring for writing assessment (pp. 303-322). Cresskill, NJ: Hampton Press.
26. Kane MT (2002). Validating high-stakes testing programs. Educational measurement: Issues and practice, 21: 31-41.
27. Kane M. (2006). Validation. In R Brennen (Ed.), Educational measurement (4th ed., pp 17-64). Westport, CT: Greenwood.
28. Kelly PA (2006). Review of the book Automated essay scoring: A cross-disciplinary perspective. Applied Psychological Measurement, 30(1): 66-68.
29. Lee Y-W., Breland H., and Muraki E. (2004). Comparability of TOEFL® CBT writing prompts for different native language groups (TOEFL Research Rep. No. TOEFL-RR-77). Princeton, NJ: Educational Testing Service.
30. Lee Y-W., Chodorow M., and Gentile C. (2006, June). Typical error patterns for different L1 groups in ESL test-takers' essays. Paper presented at the 28th annual Language Testing Research Colloquium, Melbourne, Australia.
31. Lee Y-W., Gentile C., and Kantor R. (in press). Toward automated multi-trait scoring of essays: Investigating links among holistic, analytic, and text feature scores. Applied Linguistics.
32. Lee Y-W., Kantor R. (2005). Dependability of new ESL writing test scores: Evaluating prototype tasks and alternative rating schemes (TOEFL Monograph No. TOEFL-MS-31). Princeton, NJ: Educational Testing Service.
33. Lumley T. (2002). Assessment criteria in a large-scale writing test: What do they mean to the raters? Language Testing, 19(3): 246-276.
34. Quinlan T. (2004). Speech recognition technology and students with writing difficulties: Improving fluency. Journal of Educational Psychology, 96(2): 337-346.
35. Quinlan T., Higgins D., and Wolff S. (2009). Evaluating the construct-coverage of e-rater® scoring engine. (ETS Research Report No. RR-09-01). Princeton, NJ: ETS.
36. Ransdell S., Levy CM (1996). Working memory constraints on writing quality and fluency. In CM Levy, S Ransdell (Eds.), The science of writing (pp. 93-105). Mahwah, NJ: Lawrence Erlbaum.
37. Spandel V., Stiggins RJ (1990). Creating writers: Linking assessment and writing instruction. New York: Longman.
38. Steward MR, Grobe CH (1979). Syntactic maturity, mechanics, and vocabulary and teachers' quality ratings. Research in the Teaching of English, 13: 207-215.
39. Strunk W. (2000). The elements of style. Boston, MA: Allyn & Bacon.
40. Weigle SC (2002). Assessing writing. New York: Cambridge University Press.
41. Weigle SC (2010). Validation of automated scoring of TOEFL iBT tasks against non-test indicators of writing Language Testing, 27(3): 335-353.
42. Williamson MW, Huot BA (Eds.) (1993). Validating holistic scoring for writing assessment. Cresskill, NJ: Hampton Press.
43. Xi X., Higgins D., Zechner K., and Williamson DM (2008). Automated scoring of spontaneous speech using SpeechRaterSM v1.0. (ETS Research Report No. RR-08-62). Princeton, NJ: ETS.
44. Zhang Y. (2008). Repeater analyses for TOEFL® iBT (ETS Research Memorandum No. RM-08-05). Princeton, NJ: ETS.