Scoring with the computer: Alternative procedures for improving the reliability of holistic essay scoring

Language Testing

Volumn 30, Issue 1, 2013, Pages 125-141

  Attali, Yigal ^a   Lewis, Will ^a   Steier, Michael ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

Automated scoring; essay writing assessment; reliability

Indexed keywords

EID: 84872318886     PISSN: 02655322     EISSN: 14770946     Source Type: Journal    
DOI: 10.1177/0265532212452396     Document Type: Article

References (34)

1. Alderson J. C.,Clapham C.,Wall D.Language test construction and evaluation. Cambridge: Cambridge University Press; 1995:.
2. Attali Y.Construct validity of e-rater in scoring TOEFL essays. Princeton, NJ: Educational Testing Service; 2007:.
3. Attali Y.A differential word use measure for content analysis in automated essay scoring. Princeton, NJ: Educational Testing Service; 2011a:.
4. Attali Y.Sequential effects in essay ratings.Educational and Psychological Measurement. 2011b;71:68-79.
5. Attali Y.Handbook on automated essay evaluation: Current applications and new directions. Shermis M.D.Burstein J.C., ed. New York, NY: Routledge; 2012:.
6. AttaliY.BursteinJ. (2006). Automated essay scoring with e-rater® V.2. Journal of Technology, Learning, and Assessment, 4(3). Available from http://ejournals.bc.edu/ojs/index.php/jtla/.
7. Bachman L. F.,Lynch B. K.,Mason M.Investigating variability in tasks and rater judgments in a performance test of foreign language speaking.Language Testing. 1995;12:238-257.
8. Baldwin D.,Fowles M.,Livingston S.Guidelines for constructed-response and other performance assessments. Princeton, NJ: Educational Testing Service; 2005:.
9. Barrett S.The impact of training on rater variability.International Education Journal. 2001;2:49-58.
10. Breland H. M.,Bridgeman B.,Fowles M. E.Writing assessment in admission to higher education: Review and framework. New York: College Entrance Examination Board; 1999:.
11. Breland H. M.,Camp R.,Jones R. J.,Morris M. M.,Rock D. A.Assessing writing skill. New York: College Entrance Examination Board; 1987:.
12. Burstein J. C.,Kukich K.,Wolff S.,Lu C.,Chodorow M. Paper presented at the annual meeting of the National Council of Measurement in Education; 1998San Diego, CA; 1998.
13. Daly J. A.,Dickson-Markman F.Contrast effects in evaluating essays.Journal of Educational Measurement. 1982;19:309-316.
14. Eckes T.Rater types in writing performance assessments: A classification approach to rater variability.Language Testing. 2008;25:155-185.
15. Elder C.,Knoch U.,Barkhuizen G.,von Randow J.Individual feedback to enhance rater training: Does it work?.Language Assessment Quarterly. 2005;2:175-196.
16. Elliot S. M. Paper presented at the annual meeting of the American Educational Research Association; 2001Seattle, WA; 2001.
17. Engelhard G.Examining rater errors in the assessment of written composition with a many-faceted Rasch model.Journal of Educational Measurement. 1994;31:93-112.
18. Engelhard G.,Myford C. M.Monitoring faculty consultant performance in the Advanced Placement English Literature and Composition Program with a many-faceted Rasch model. New York: College Entrance Examination Board; 2003:.
19. GRE (2011). Scoring guide for the issue task. Retrieved from www.ets.org/gre/revised_general/prepare/analytical_writing/issue/scoring_guide.
20. Hales L. W.,Tokar E.The effect of the quality of preceding responses on the grades assigned to subsequent responses to an essay question.Journal of Educational Measurement. 1975;12:115-117.
21. Huot B.The literature of direct writing assessment: Major concerns and prevailing trends.Review of Educational Research. 1990;60:237-263.
22. Landauer T. K.,Foltz P. W.,Laham D.Introduction to latent semantic analysis.Discourse Processes. 1998;25:259-284.
23. Landauer T. K.,Laham D.,Foltz P. W.Automated essay scoring: A cross-disciplinary perspective. Shermis M. D.Burstein J., ed. Mahwah, NJ: Lawrence Erlbaum; 2003:87-112.
24. Lee Y.,Gentile C.,Kantor R.Analytic scoring of TOEFL® CBT essays: Scores from humans and e-rater. Princeton, NJ: Educational Testing Service; 2008:.
25. Lumley T.,McNamara T. F.Rater characteristics and rater bias: Implications for training.Language Testing. 1995;12:54-71.
26. Moss P. A.Can there be validity without reliability?.Educational Researcher. 1994;23:5-12.
27. Mussweiler T.Comparison processes in social judgment: Mechanisms and consequences.Psychological Review. 2003;110:472-489.
28. Pelham B. W.,Wachsmuth J. O.The waxing and waning of the social self: Assimilation and contrast in social comparison.Journal of Personality and Social Psychology. 1995;69:825-838.
29. Powers D.'Wordiness': A selective review of its influence, and suggestions for investigating its relevance in tests requiring extended written responses (ETS RM-04-08). Princeton, NJ: Educational Testing Service; 2005:.
30. Sherif M.,Taub D.,Hovland C. I.Assimilation and contrast effects of anchoring stimuli on judgments.Journal of Experimental Psychology. 1958;55:150-155.
31. Spear M.The influence of contrast effects upon teachers' marks.Educational Research. 1997;39:229-233.
32. Weigle S. C.Using FACETS to model rater training effects.Language Testing. 1998;15:263-287.
33. Weigle S. C.Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches.Assessing Writing. 1999;6:145-178.
34. Weigle S. C.Assessing writing. New York: Cambridge University Press; 2002:.