Toward more substantively meaningful automated essay scoring

Journal of Technology, Learning, and Assessment

Volumn 6, Issue 1, 2007, Pages

  Ben Simon, Anat ^a   Bennett, Randy Elliot ^b  

^a National Institute for Testing and Evaluation Jerusalem   (Israel)

^b EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 34547602972     PISSN: None     EISSN: 15402525     Source Type: Journal    
DOI: None     Document Type: Review

References (36)

1. Attali, Y., & Burstein, J. (2005). Automated essay scoring with e-rater version 2.0 (ETS RR-04-45). Princeton, NJ: Educational Testing Service.
2. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater version 2.0. Journal of Technology, Learning and Assessment, 4(3). Retrieved May 20, 2007 from http://www.bc.edu/research/intasc/jtla/ journal/pdf/v4n3_jtla.pdf
3. Bennett, R.E. (2006). Moving the field forward: Some thoughts on validity and automated scoring (pp. 403-412). In D. M. Williamson, R. J. Mislevy, & I. I. Bejar (Eds.), Automated scoring of complex tasks in computer-based testing. Hillsdale, NJ: Erlbaum.
4. Bennett, R.E., & Ben-Simon, A. (2006). Toward theoretically meaningful automated essay scoring (Report 329). Jerusalem, Israel: National Institute for Testing and Evalution. Retrieved May 20, 2007 from http://clickit.ort.org.il/files/upl/224004906/934404432.pdf
5. Burstein, J., Braden-Harder, L., Chodorow, M., Hua, S., Kaplan, B., Kukich, K., et al. (1998). Computer analysis of essay content for automated score prediction (ETS RR-98-15). Princeton, NJ: Educational Testing Service.
6. Burstein, J., & Chodorow, M. (1999, June). Automated essay scoring for nonnative English speakers. In Proceedings of the ACL99 Workshop on Computer-Mediated Language Assessment and Evaluation of Natural Language Processing. Retrieved December 16, 2005, from http://www.ets.org/Media/ Research/pdf/erater_acl99rev.pdf
7. Burstein, J., Chodorow, M., & Leacock, C. (2004, Fall). Automated essay evaluation: the Criterion Online writing service. AI Magazine. Retrieved September 26, 2005, from http://www.findarticles.com/p/articles/ mi_m2483/is_3_25/ai_n6258424
8. Cheville, J. (2004). Automated scoring technologies and the rising influence of error. English Journal, 93(4), 47-52.
9. Cizek, G.J., & Page, B.A. (2003). The concept of reliability in the context of automated essay scoring. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective. Mahwah, NJ: Erlbaum.
10. Elliot, S. (2001). IntelliMetric: From here to validity. Paper presented at the annual meeting of the American Educational Research Association, Seattle, Washington.
11. Elliot, S. (2003). IntelliMetric: From here to validity. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective. Mahwah, NJ: Erlbaum.
12. Elliot, S., & Mikulas, C. (2004). How does IntelliMetricTM score essay responses? A mind based approach. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
13. E-rater [Computer software]. (1997). Princeton, NJ: Educational Testing Service.
14. Foltz, P.W. (1996). Latent Semantic Analysis for text-based research. Behavior Research Methods, Instruments, and Computers, 28(2), 197-202.
15. Foltz, P.W., Laham, D., & Landauer, T.K. (1999). Automated essay scoring: Applications to educational technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 1(2). Retrieved December 16, 2005, from http://imej.wfu.edu/articles/1999/2/04/
16. Hayes, J.R. (1996). A new framework for understanding cognition and affect in writing. In C.M. Levy. & S. Ransdell (Eds.), The science of writing: Theories, methods, individual differences, and applications. Mahwah, NJ: Lawrence Erlbaum.
17. Hayes, J.R., & Flower, L.S. (1980). Identifying the organization of writing processes. In L. Gregg & E.R. Steinberg (Eds.), Cognitive processes in writing (pp. 3-30). Hillsdale, NJ: Lawrence Erlbaum.
18. Horkay, N., Bennett, R.E., Allen, N., Kaplan, B, & Yan, F. (2006). Does it matter if I take my writing test on computer? An empirical study of mode effects in NAEP. Journal of Technology, Learning and Assessment, 5(2). Available: http://escholarship.bc.edu/jtla/vol5/2/
19. Intelligent Essay Assessor [Computer software]. (1997). Boulder, CO: University of Colorado.
20. IntelliMetric Engineer [Computer software]. (1997). Yardley, PA: Vantage Technologies.
21. Keith, T.Z. (2003). Validity of automated essay scoring systems. In M.D. Shermis & J.C. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective. Mahwah, NJ: Erlbaum.
22. Landauer, T.K., & Dumais, S.T. (1997). A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211-240.
23. Landauer, T.K., Foltz, PW, & Laham, D. (1998). Introduction to latent semantic analysis. Discourse Processes, 25, 259-284.
24. Landauer, T.K., Laham, D., & Foltz, P.W. (2001). The Intelligent Essay Assessor: Putting knowledge to the test. Paper presented at the Emerging Technologies and Opportunities for Diverse Applications conference, Tucson, AZ.
25. Landauer, T.K., Laham, D., & Foltz, P.W. (2003). Automated scoring and annotation of essays with the Intelligent Essay Assessor. In M.D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 87-112). Mahwah, NJ: Erlbaum.
26. Landauer, T.K., Laham, D., Rehder, B., & Schreiner, M.E. (1997). How well can passage meaning be derived without using word order? A comparison of Latent Semantic Analysis and humans. In M.G. Shafto & P. Langley (Eds.), Proceedings of the 19th annual meeting of the Cognitive Science Society (pp. 412-417). Mahwah, NJ: Erlbaum.
27. National Center for Education Statistics (NCES). (2004). What does the NAEP writing assessment measure? Retrieved May 20, 2007 from http://nces.ed.gov/nationsreportcard/writing/whatmeasure.asp
28. Olson, L. (2007). NAEP writing exams going digital in 2011. Education Week, 26(27), 23.
29. Page, E.B. (1966). The imminence of grading essays by computer. Phi Delta Kappan, 48, 238-243.
30. Page, E.B. (1994). Computer grading of student prose, using modern concepts and software. Journal of Experimental Education, 62, 127-142.
31. Page, E.B. (2003). Project essay grade: PEG. In M.D. Shermis & J.C. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 43-54). Mahwah, NJ: Erlbaum.
32. Page, E.B., & Petersen, N.S. (1995). The computer moves into essay grading: Updating the ancient test. Phi Delta Kappan, 76, 561-565.
33. Petersen, N.S. (1997) Automated scoring of written essays: Can such scores be valid? Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.
34. Powers, D.E., Burstein, J. C., Chodorow, M., Fowles, M.E., & Kukich, K. (2002) Stumping E-Rater: Challenging the validity of automated essay scoring. Computers in Human Behavior, 18, 103-134.
35. Shermis, M.D., Koch, CM., Page, E.B., Keith, T.Z., & Harrington, S. (2002). Trait rating for automated essay scoring. Educational and Psychological Measurement, 62, 5-18.
36. Yang, Y., Buckendahl, CW, Jusziewicz, P.J., & Bhola, D.S. (2002). A review of strategies for validating computer-automated scoring. Applied Measurement in Education, 15(4), 391-412.