Toward automated multi-trait scoring of essays: Investigating links among holistic, analytic, and text feature scores

Applied Linguistics

Volumn 31, Issue 3, 2010, Pages 391-417

  Lee, Yong Won ^a   Gentile, Claudia ^a   Kantor, Robert ^a  

^a Seoul National University   (South Korea)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77953001150     PISSN: 01426001     EISSN: 1477450X     Source Type: Journal    
DOI: 10.1093/applin/amp040     Document Type: Article

References (56)

1. Anson, C. M. 2006. 'Can't touch this: reflections on the servitude of computers as Readers' in P. F. Ericsson and R. Haswell (eds): Machine Scoring of Students Essays: Truths and Consequences. Utah State University, pp. 38-56.
2. Attali, Y. and J. Burstein. 2006. 'Automated essay scoring with e-rater® V.2,' Journal of Technology, Learning, and Assessment 4/3: available at http://www.jtla.org. Accessed 16 April 2007.
3. Bacha, N. 2001. 'Writing evaluation: what can analytic versus holistic essay scoring tell us?' System 29: 371-383
4. Ben-Simon, A. and R. E. Bennett. 2007. 'Toward more substantively meaningful automated essay scoring.' Journal of Technology, Learning, and Assessment 6/1: available at http://www.jtla.org. Accessed 6 January 2008.
5. Brennan, R. L. 1999. Manual for mGENOVA Version 2.0. The University of Iowa.
6. Borg, I. and P. Groenen. 1997. Modern Multidimensional Scaling: Theory and Applications. Springer.
7. Carson, S., B. Bridgeman, R. Camp and J. Waanders. 1985. Relationship of Admission Test Scores to Writing Performance of Native and Non-native Speakers of English (TOEFL Research Report No. 19). ETS.
8. Connor-Linton, J. 1995. 'Looking behind the curtain: what do L2 composition ratings really mean?' TESOL Quarterly 29: 762-765
9. Crick, G. E. and R. L. Brennan. 1983. Manual for GENOVA: A Generalized Analysis of Variance System (ACT Tech. Bulletin No. 43). American College Testing Program.
10. Cumming, A. 2007. 'Book review: Machine scoring of students' essays: Truth and consequences,' Assessing Writing 12: 80-82
11. Cumming, A., R. Kantor, and D. Powers. 2002. 'Decision-making while rating ESL/EFL writing tasks: A descriptive framework,' Modern Language Journal 86/1: 67-96.
12. Dikli, S. 2006. 'An overview of automated scoring of essays,' Journal of Technology, Learning, and Assessment 5/1: available at http://www.jtla.org. Accessed 30 March 2007.
13. Educational Testing Service. 1998. Computerbased TOEFL® Score User Guide.
14. Educational Testing Service. 2007. TOEFL® iBT Tips: How to Prepare for the TOEFL iBT.
15. Engber, C. A. 1995. 'The relationship of lexical proficiency to the quality of ESL compositions,' Journal of Second Language Writing 4: 139-155
16. Ericsson, P. F. and R. Haswell. 2006. Machine Scoring of Students Essays: Truths and Consequences. Utah State University.
17. Ferris, D. 1994. 'Lexical and syntactic features of ESL writing by students at different levels of L2 proficiencies,' TESOL Quarterly 28: 414-420
18. Frase, L., J. Faletti, A. Ginther and L. Grant. 1999. Computer Analysis of the TOEFL Test of Written English (TOEFL Research Report No. 64). ETS.
19. Freedman, S. W. 1984. 'The registers of student and professional expository writing. Influences on teacher responses' in R. Beach and S. Bridwell (eds): New Directions in Composition Research. Guilford Press, pp. 334-347
20. Gentile, C., A. Riazantseva, and F. Cline. 2002. A Comparison of Handwritten and Wordprocessed TOEFL Essays: Final Report. Internal document. ETS.
21. Grant, L. and A. Ginther. 2000. 'Using computer-tagged linguistic features to describe L2 writing differences,' Journal of Second Language Writing 9: 123-145.
22. Hamp-Lyons, L. 1991. 'Scoring procedures for ESL contexts' in L Hamp-Lyons (ed.): Assessing Second Language Writing in Academic Contexts. Ablex, pp. 241-276
23. Hamp-Lyons, L. 1995. 'Rating nonnative writing: the trouble with holistic scoring,' TESOL Quarterly 29: 759-762
24. Han, N.-R., M. Chodorow, and C. Leacock. 2006. 'Detecting errors in English article usage by nonnative speakers,' Natural Language Engineering, 12/2: 115-129
25. Haswell, R. 2006. 'Automatons and automated scoring: drudges, black boxes, and Dei Ex Machina' in P. F. Ericsson and R. Haswell (eds): Machine Scoring of Students Essays: Truths and Consequences. Utah State University, pp. 57-78.
26. Herrington, A. and C. Moran. 2001. 'What happens when machines read our students' writing?' College English 63/4: 480-499
27. Huot, B. 1990. 'The literature of direct writing assessment: major concerns and prevailing trends,' Review of Educational Research 60: 237-263
28. Jacobs, H. L., S. A. Zingraf, D. R. Wormuth, V. F. Hartfiel, and J. B. Hughey. 1981. Testing ESL Composition: A Practical Approach. Newbury House.
29. Jarvis, S. 2002. 'Short texts, best-fitting curves, and new measures of lexical diversity,' Language Testing 19: 57-84.
30. Jarvis, S., L. Grant, D. Bikowski, and D. Ferris. 2003. 'Exploring multiple profiles of highly rated learner composition,' Journal of Second Language Writing 12: 377-403.
31. Kondo-Brown, K. 2002. 'A FACETS analysis of rater bias in measuring Japanese second language writing performance,' Language Testing 19/1: 3-31.
32. Kukich, K. 2000, September/October. 'Beyond automated essay scoring,' IEEE Intelligent Systems 15/5: 22-27
33. Leacock, C. and M. Chodorow. 2003. 'Automated grammatical error detection' in M. D. Shermis and J. C. Burstein (eds): Automated Essay Scoring: A Cross-disciplinary Perspective. Lawrence Erlbaum, pp. 195-207.
34. Lee, Y.-W. C. 2006. Variability and Validity of Automated Essay Scores for TOEFL iBT: Generic, Hybrid, and Prompt-specific Models. Internal document. ETS.
35. Lee, Y.-W., C. Gentile, and R. Kantor. 2008. Analytical Scoring of TOEFL CBT Essays: Scores by Humans and e-rater® (TOEFL Research Report No. 81; ETS RR-08-01). ETS.
36. Laufer, B. and P. Nation. 1995. 'A vocabulary size test of controlled productive ability,' Language Testing 16: 33-51.
37. Phillips, S. M. 2007. Automated Essay Scoring: A Literature Review. Society for the Advancement of Excellence in Education.
38. Powers, D., J. Burstein, M. Chodorow, M. E. Fowles, and K. Kukich. 2001a. Comparing the Validity of Automated and Human Essay Scoring (GRE Board Professional Report No. 98-08aR). ETS.
39. Powers, D., J. Burstein, M. Chodorow, M. E. Fowles, and K. Kukich. 2001b. Stumping E-rater: Challenging the Validity of Automated Essay Scoring (GRE Board Professional Report No. 98-08bP). ETS.
40. Quinlan, T., D. Higgins, and S. Wolff. 2009. Evaluating the Construct-coverage of the e-rater® Scoring Engine. (ETS Research Report No. RR 09-01). ETS.
41. Raimes, A. 1990. 'The TOEFL test of written English: causes for concern,' TESOL Quarterly 24: 427-442
42. Reid, J. 1986. 'Using the Writer's Workbench in composition teaching and testing' in C. Stansfield (ed.): Technology and Language Testing. TESOL, pp. 167-188
43. Roid, G. H. 1994. 'Patterns of writing skills derived from cluster analysis of direct writing assessments,' Applied Measurement in Education 7/2: 159-170
44. Santos, T. 1988. ' 'Professors' reactions to the academic writing of non-native speaking students,' TESOL Quarterly 22: 69-90.
45. Sasaki, M. and K. Hirose. 1999. 'Development of an analytic rating scale for Japanese L1 writing,' Language Testing 16/4: 457-478
46. Shermis, M. D. and J. C. Burstein. 2003. Automated Essay Scoring: A Cross-disciplinary Perspective. Lawrence Erlbaum.
47. SPSS. 2003. SPSS Base 12.0 User's Guide. Author.
48. Swartz, C. W., S.R. Hooper, J.W. Montgomery, M. B. Wakely, R. E. L. de Kruif, M. Reed, T. T. Brown, M. D. Levine, and K. P. White. 1999. 'Using generalizability theory to estimate the reliability of writing scores derived from holistic and analytic scoring methods,' Educational and Psychological Measurement 59/3: 492-506.
49. Tetreault, J. and M. Chodorow. 2008. Native judgments of non-native usage: Experiments in preposition error detection. COLING Workshop on Human Judgments in Computational Linguistics. Manchester, UK.
50. Veal, L. R. and S. A. Hudson. 1983. 'Direct and indirect measures for large-scale evaluation of writing,' Research in the Teaching of English 17: 285-296
51. Weigle, S. C. 2002. Assessing Writing. Cambridge University Press.
52. Weir, C. J. 1990. Communicative Language Testing. Prentice Hall Regents.
53. Williamson, M. M. 2004. 'Validity of automated scoring: prologue for a continuing discussion of machine scoring of student writing,' Journal of Writing Assessment 1/2: 85-104.
54. Williamson, M. M. and B. A. Huot. 1993. Validating Holistic Scoring for Writing Assessment: Theoretical and Empirical Foundations. Hampton Press.
55. Yang, Y., C. W. Buckendahl, P. J. Jusziewicz, and D. S. Bhola. 2002. 'A review of strategies for validating computer-automated scoring,' Applied Measurement in Education 15: 391-412.
56. Ziegler, W. W. 2006. 'Computerized writing assessment: community college faculty find reasons to say "not yet" ' in P. F. Ericsson and R. Haswell (eds): Machine Scoring of Students Essays: Truths and Consequences. Utah State University, pp. 138-146