The nature of automated essay scoring feedback

CALICO Journal

Volumn 28, Issue 1, 2011, Pages 99-134

  Dikli, Semire ^a  

^a GEORGIA GWINNETT COLLEGE   (United States)

Author keywords

Feedback; Language; Technology; Writing

Indexed keywords

EID: 85045167486     PISSN: None     EISSN: 20569017     Source Type: Journal    
DOI: 10.11139/cj.28.1.99-134     Document Type: Article

References (42)

1. Ashwell, T. (2000). Patterns of teacher response to student writing in a multiple-draft composition classroom: Is content feedback followed by form feedback the best method? Journal of Second Language Writing, 9, 227-257.
2. Attali, Y. (2004, April). Exploring the feedback and revision features of Criterion. Paper presented at the National Council on Measurement in Education (NCME), San Diego, CA.
3. Attali, Y., & Burstein, J. (2006). An evaluation of IntelliMetric essay scoring system. Journal of Technology, Learning, and Assessment, 4(4). Retrieved From http://escholarship.bc.edu/jtla/vol4/3
4. Burstein, J. (2003). The e-rater scoring engine: Automated essay scoring with natural language processing. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring: A cross disciplinary approach (pp. 113-121). Mahwah, NJ: Lawrence Erlbaum Associates.
5. Burstein, J., & Chodorow, M. (1999, June). Automated essay scoring for nonnative English speakers. In Proceedings of the ACL99 Workshop on Computer-Mediated Language Assessment and Evaluation of Natural Language Processing. College Park, MD.
6. Chandler, J. (2003). The efficacy of various kinds of error feedback for improvement in the accuracy and fluency of L2 student writing. Journal of Second Language Writing, (12), 267-296.
7. Chen, C. E., & Cheng, W. E. (2008). Beyond the design of automated writing evaluation: Peda-gogical practices and perceived learning effectiveness in EFL writing classes. Language Learning & Technology, 12(2), 94-112. Retrieved from http://llt.msu.edu/vol12num2/default.html
8. Chodorow, M., & Burstein, J. (2004). Beyond essay length: Evaluating e-rater’s performance on TOEFL essays (Research report no 73). Princeton, NJ: Educational Testing Service (ETS).
9. Chung, K. W. K., & O’Neil, H. F. (1997). Methodological approaches to online scoring of essays. (ERIC Document Reproduction Service No. ED418101)
10. Chung, G. K. W. K., & Baker, E. L. (2003). Issues in the reliability and validity of automated scoring of constructed responses. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 23-40). Mahwah, NJ: Lawrence Erlbaum Associ-ates.
11. Cizek, G. J., & Page, B. A. (2003). The concept of reliability in the context of automated essay scoring. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 125-145). Mahwah, NJ: Lawrence Erlbaum Associates.
12. Dikli, S. (2006). An overview of automated scoring of essays. Journal of Technology, Learn-ing, and Assessment, 5(1), n.p. Retrieved http://escholarship.bc.edu/jtla/vol5/1
13. Edelblut, P., & Vantage Learning. (2003, November). An analysis of the reliability of computer automated essay scoring by IntelliMetric of essays written in Malay language. Paper presented at TechEX 03, Ruamrudee International School, Bangkok, Thailand.
14. Edelblut, P., & Vantage Learning. (2005, March). The effects of inclusion of native speakers’ writing samples on the domain scoring accuracy of automated essay scoring of writing submitted by Taiwanese English language learners. Paper presented at The 32nd Annual Conference of the International Association for Educational Assessment, Singapore.
15. Elliott, S. (2003). IntelliMetric: From here to validity. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 125-145). Mahwah, NJ: Lawrence Erlbaum Associates.
16. Elliot, S., & Mikulas, C. (2004, April). A summary of studies demonstrating the educational impact of the MY Access online writing instructional application. Paper presented at the National Council on Measurement in Education (NCME), San Diego, CA.
17. Fathman, A. K., & Whalley, E. (1990). Teacher response to student writing: Focus on form versus content. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 178-190). Cambridge, UK: Cambridge University Press.
18. Ferris, D., & Roberts, B. (2001). Error feedback in L2 writing classes: How explicit does it need to be? Journal of Second Language Writing, 10, 161-184.
19. Grimes, D., & Warschauer, M. (2006, April). Automated essay scoring in the classroom. Paper presented at the Annual Meeting of the American Educational Research Association (AERA), San Francisco, CA.
20. Hamp-Lyons, L. (2001). Fourth generation writing assessment. In T. Silva & P. K. Matsuda (Eds.), On second language writing (pp. 117-125). Mahwah, NJ: Lawrence Erlbaum Associ-ates.
21. Hyland, F. (1998). The impact of teacher written feedback on individual writers. Journal of Second Language Writing, 7(3), 255-286.
22. Keith, T. M. (2003). Validity of automated essay scoring systems. In M. D. Shermis & J. Burst-ein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 147-166). Mah-wah, NJ: Lawrence Erlbaum Associates.
23. Landauer, T. K., Foltz, P. W., & Laham, D. (1998). Introduction to latent semantic analysis. Discourse Processes, 25, 259-284. Retrieved from http://lsa.colorado.edu/whatis.html
24. Landauer, T. K., Laham, D., & Foltz, P. W. (2003). Automated essay scoring: A cross disciplinary perspective. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring and annotation of essays with the Intelligent Essay Assessor (pp. 87-112). Mahwah, NJ: Lawrence Erlbaum Associates.
25. Landauer, T. K., Laham, D., Rehder, B., & Schreiner, M. E. (1997). How well can passage meaning be derived without using word order? A comparison of Latent Semantic Analysis and humans. In M.G. Shafto & P. Langley (Eds.), Proceedings of the 19th Annual Conference of the Cognitive Science Society (pp. 412-417). Mahwah, NJ: Lawrence Erlbaum Associates.
26. Nichols, P. D. (2004, April). Evidence for the interpretation and use of scores from an automated essay scorer. Paper presented at the Annual Meeting of the American Educational Research Association (AERA), San Diego, CA.
27. Page, E. B. (2003). Project essay grade: PEG. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 43-54). Mahwah, NJ: Lawrence Erlbaum Associates.
28. Rudner, L. M., Garcia, V., & Welch, C. (2006). An evaluation of IntelliMetric essay scoring sys-tem. Journal of Technology, Learning, and Assessment, 4(4). Retrieved from http://escholar-ship.bc.edu/jtla/vol4/4
29. Rudner, L. M., & Liang, T. (2002). Automated essay scoring using Bayes’ theorem. The Journal of Technology, Learning, and Assessment, 1(2). Retrieved from http://escholarship.bc.edu/jtla/vol1/2
30. Shermis, M. D., & Burstein, J. (2003). Automated essay scoring: A cross disciplinary perspec-tive. Mahwah, NJ: Lawrence Erlbaum Associates.
31. Siegert, K. O., & Guo, F. (2009). Assessing the reliability of GMAT analytical writing assessment (RR-09-02). Reston, VA: Graduate Management Admissions Council (GMAC).
32. Vantage Learning. (n.d.). Retrieved from http://www.vantagelearning.com
33. Vantage Learning. (2000a). A study of expert scoring and IntelliMetric scoring accuracy for dimensional scoring of Grade 11 student writing responses (RB-397). Newtown, PA: Author.
34. Vantage Learning. (2000b). A true score study of IntelliMetric accuracy for holistic and dimensional scoring of college entry-level writing program (RB-407). Newtown, PA: Vantage Learning.
35. Vantage Learning. (2001). A preliminary study of the efficacy of IntelliMetric for use in scoring Hebrew assessments (RB-561). Newtown, PA: Author.
36. Vantage Learning. (2002). A study of expert scoring, standard human scoring and IntelliMet-ric scoring accuracy for statewide eighth grade writing responses (RB-726). Newtown, PA: Author.
37. Vantage Learning. (2003a). A true score study of 11th grade student writing responses using IntelliMetric Version 9.0 (RB-786). Newtown, PA: Author.
38. Vantage Learning. (2003b). Assessing the accuracy of IntelliMetric for scoring a districtwide writing assessment (RB-806). Newtown, PA: Author.
39. Vantage Learning. (2003c). How does IntelliMetric score essay responses? (RB-929). New-town, PA: Author.
40. Vantage Learning. (2004). A practitioner’s guide to scientifically-based MY Access research (RB-931). Newtown, PA: Author.
41. What is artificial intelligence? (n.d.). Retrieved from http://www-formal.standford.edu/jmc/whatisai.html
42. Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2), 1–24. Retrieved from http://ltr.sagepub. com/content/10/2/157.full.pdf+html