How important is content in the ratings of essay assessments?

Assessment in Education: Principles, Policy and Practice

Volumn 15, Issue 1, 2008, Pages 91-105

  Shermis, Mark D ^a   Shneyderman, Aleksandr ^b   Attali, Yigal ^c  

^a UNIVERSITY OF FLORIDA   (United States)

^b Miami Dade County   (United States)

^c EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85052504410     PISSN: 0969594X     EISSN: 1465329X     Source Type: Journal    
DOI: 10.1080/09695940701876219     Document Type: Article

References (29)

1. Attali, Y., Exploring the feedback and revision features of Criterion. Paper presented at the National Council on Measurement in Education. April13–15, San Diego, CA.
2. Attali, Y., On-the-fly automated essay scoring. Paper presented at the National Council on Measurement in Education. April8–10, San Francisco, USA.
3. Attali, Y., and Burstein, J., 2006. Automated Essay Scoring With e-rater V.2. Journal of Technology, Learning, and Assessment, 4: 3 http://www.jtla.org
4. Burstein, J., 2003. “ The e-rater scoring engine: Automated essay scoring with natural language processing ”. In Automated essay scoring: A cross-disciplinary perspective, Edited by: Shermis, M.D., and Burstein, J., 113–122. Mahwah, NJ: Lawrence Erlbaum Associates.
5. Burstein, J., Chodorow, M., and Leacock, C., Criterion: Online essay evaluation: An application for automated evaluation of test-taker essays. Paper presented at the Fifteenth Annual Conference on Innovative Applications of Artificial Intelligence. August12–14, Acapulco, Mexico.
6. D'Angelo, F.J., 1976. “ Modes of discourse ”. In Teaching composition: 10 bibliographical essays, Edited by: Tate, G., 111–135. Fort Worth, TX: Christian University Press.
7. Elliot, S., 2003. “ Intellimetric: From here to validity ”. In Automated essay scoring: A cross-disciplinary perspective, Edited by: Shermis, M.D., and Burstein, J., 71–86. Mahwah, NJ: Lawrence Erlbaum Associates.
8. Engelhard, G., 1992. The measurement of writing ability with a many-faceted Rasch model. Applied Measurement in Education, 5 (3): 171–191.
9. Ericsson, P.F., and Haswell, R., eds. 2006. Machine scoring of student essays: Truth and consequences, Logan, UT: Utah State University Press.
10. Hamp-Lyons, L., and Mathias, S.P., 1994. Examining expert judgements of task difficulty on essay tests. Journal of Second Language Writing, 3 (1): 49–68.
11. Kawate-Mierzejewska, M., E-rater software. Paper presented at the Japanese Association for Language Teaching monthly chapter meeting. March23, Tokyo, Japan.
12. Keith, T.Z., 2003. “ Validity and automated essay scoring systems ”. In Automated essay scoring: A cross-disciplinary perspective, Edited by: Shermis, M.D., and Burstein, J., 147–168. Mahwah, NJ: Lawrence Erlbaum Associates.
13. Landauer, T.K., Foltz, P.W., and Laham, D., 1998. Introduction to latent semantic analysis. Discourse Processes, 25 (2–3): 259–284.
14. Landauer, T.K., Laham, D., and Foltz, P.W., 2003. “ Automated scoring and annotation of essays with the Intelligent Essay Assessor ”. In Automated essay scoring: A cross-disciplinary perspective, Edited by: Shermis, M.D., and Burstein, J., 87–112. Mahwah, NJ: Lawrence Erlbaum Associates.
15. Leacock, C., and Chodorow, M., 2003. C-rater: Scoring of short-answer questions. Computers and the Humanities, 37 (4): 389–405.
16. Leki, I., and Carson, J., 1997. Completely different worlds: EAP and the writing experiences of ESL students in university courses. TESOL Quarterly, 31 (1): 39–70.
17. McTavish, D.G., Litkowski, K.C., and Schrader, S., 1997. A computer content analysis approach to measuring social distance in residential organisations for older people. Social Science Computer Review, 15 (2): 170–180.
18. Page, E.B., 2003. “ Project Essay Grade: PEG ”. In Automated essay scoring: A cross-disciplinary perspective, Edited by: Shermis, M.D., and Burstein, J., 43–54. Mahwah, NJ: Lawrence Erlbaum Associates.
19. Page, E.B., Keith, T., and Lavoie, M.J., Construct validity in the computer grading of essays. Paper presented at the annual meeting of the American Psychological Association. August11–15, New York, USA.
20. Page, E.B., and Petersen, N.S., 1995. The computer moves into essay grading: Updating the ancient test. Phi Delta Kappan, 76 (7): 561–565.
21. Rajecki, D.W., Dame, J.A., Creek, K.J., Barrickman, P.J., Reid, C.A., and Appleby, D.C., 1993. Gender casting in television toy advertisements: Distributions, message content analysis, and evaluations. Journal of Consumer Psychology, 2 (2): 307–327.
22. Ruth, J., and Murphy, S.M., 1988. Designing writing tasks for the assessment of writing, Norwood, NJ: Ablex.
23. Shermis, M.D., Burstein, J., and Leacock, C., 2006. “ Applications of computers in assessment and analysis of writing ”. In Handbook of writing research, Edited by: MacArthur, C.A., Graham, S., and Fitzgerald, J., 403–416. New York: Guilford Publications.
24. Shermis, M.D., Rasmussen, J.L., Rajecki, D.W., Olson, J., and Marsiglio, C., 2001. All prompts are created equal, but some prompts are more equal than others. Journal of Applied Measurement, 2 (2): 154–170.
25. Shermis, M.D., Shneyderman, A., and Attali, Y., How important is content in the ratings of essay assessments?. Paper presented at the Paper presented at the annual meeting of the National Council of Measurement in Education. April12–14, Montreal, Canada.
26. Vantage Learning. 2001. A preliminary study of the efficacy of IntelliMetric™ for use in scoring Hebrew assessments, Newtown, PA: Vantage Learning.
27. Vantage Learning. 2002. A study of IntelliMetric™ scoring for responses written in Bahasa Malay, Newtown, PA: Vantage Learning. (No. RB-735)
28. Weigle, S.C., 1998. Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing, 6 (2): 145–178.
29. Wolfe, E.W., 2004. Identifying rater effects using latent trait models. Psychology Science, 46 (1): 35–51.