Automated essay scoring versus human scoring: A comparative study

Journal of Technology, Learning, and Assessment

Volumn 6, Issue 2, 2007, Pages 1-28

  Wang, Jinhao ^a   Brown, Michelle Stallone ^a,b  

^a South Texas College   (United States)

^b TEXAS A AND M UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 35348949526     PISSN: None     EISSN: 15402525     Source Type: Journal    
DOI: None     Document Type: Article

References (42)

1. Anson, C.M. (2003). Responding to and assessing student writing: The uses and limits of technology. In P. Takayoshi & B. Huot (Eds.), Teaching writing with computers: An introduction (pp. 234-245). New York: Houghton Mifflin Company.
2. Baron, D. (2005, May 6). The College Board's new essay reverses decades of progress toward literacy. The Chronicle of Higher Education, P. B14.
3. Bereiter, (2003). Foreword. In M.D. Shermis & J.C. Burstein (Eds.), Automatic essay scoring: A cross-disciplinary perspective (pp. vii-x). Mahwah, NJ: Lawrence Erlaum Associates.
4. Burstein, J., Kukich, K., Wolff, S., Lu, C., & Chodorow, M. (1998, April). Computer analysis of essays. Paper presented at the annual meeting of the National Council of Measurement in Education, San Diego, CA. Retrieved September 10, 2005, from http://www.ets.org/research/dload/ncmefinal. pdf
5. Calfee, R. (2000). To grade or not to grade. IEEE Intelligent Systems 15(5), 35-37.
6. Cheville, J. (2004). Automated scoring technologies and the rising influence of error. English Journal, 93(4), 47-52.
7. College Board. (2004). ACCUPLACER coordinator's guide. New York: Author.
8. Dikli, S. (2006). An Overview of Automated Scoring of Essays. Journal of Technology, Learning, and Assessment, 5(1). Retrieved July 15, 2007, from http://www.jtla.org.
9. Elliot, S. (2003). IntelliMetric™: From here to validity. In M.D. Shermis & J.C. Burstein (Eds.), Automatic essay scoring: A cross-disciplinary perspective (pp. 71-86). Mahwah, NJ: Lawrence Erlaum Associates.
10. Erickson, J.D. (2000). Using keywords and computers to assess student writing (Doctoral Dissertation, Washington State University, 2000). Dissertation Abstracts International, 61, 3964.
11. Fitzgerald, K.R. (1994). Computerized scoring? A question of theory and practice. Journal of Basic Writing, 13(2), 3-17.
12. th ed.). New York: McGraw-Hill.
13. Garson, G.D. (2006). Reliability analysis. Retrieved September 9, 2006, from http://www2.chass.ncsu.edu/garson/pa765/reliab.htm
14. th ed.). Upper Saddle River, NJ: Pearson Education, Inc.
15. Greer, G. (2002). Are computer scores on essays the same as essay scores from human experts? In Vantage Learning, Establishing WritePlacer Validity: A summary of studies (pp. 10-11). (RB-781). Yardley, PA: Author.
16. Hellwig, H. (1990, March). Computational text analysis for predicting holistic writing scores. Paper presented at Conference on College Composition and communication, Chicago, IL.
17. Herrington, A., & Moran, C. (2001). What happens when machines read our students' writing? College English, 63(4), 480-499.
18. Huot, B. (2002). (Re)Articulating writing assessment for teaching and learning. Logan, UT: Utah State University Press.
19. Kukich, K. (2000). Beyond Automated Essay Scoring. IEEE Intelligent Systems, 15(5), 22-27.
20. Landauer, T.K., & Laham, D. (2000). The Intelligent Essay Assessor. IEEE Intelligent Systems, 15(5), 27-31.
21. McCurry, N., & McCurry, A. (1992). Writing assessment for the twenty-first century. Computer Teacher, 19, 35-37.
22. McGraw, K.O., & Wong, S.P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46.
23. Murphy, S. (2002). The relationship between WritePlacer Plus scores and course placements at Richland College. In Vantage Learning, Establishing WritePlacer Validity: A summary of studies (pp. 13-14). (RB-781). Yardley, PA: Author.
24. National Evaluation Systems. (2005). THEA faculty manual: A guide to THEA test results. Retrieved November 24, 2005, from http://www.thea.nesinc. com/PDFs/THEA_FacultyManual.pdf
25. Nivens-Bower, C. (2002). Faculty-WritePlacer Plus score comparisons. In Vantage Learning, Establishing WritePlacer Validity: A summary of studies (p. 12). (RB-781). Yardley, PA: Author. Norusis, M.J. (2004). SPSS 12.0 guide to data analysis. Upper Saddle River, NJ: Prentice Hall.
26. Norusis, M.J. (2004). SPSS 12.0 guide to data analysis. Upper Saddle River, NJ: Prentice Hall.
27. Page, E.B. (1966). The imminence of grading essays by computers. Phi Delta Kappan, 47, 238-243.
28. Page, E.B., Poggio, J.P., & Keith, T.Z. (1997, March). Computer analysis of student essays: Finding trait differences in student profile. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL. (ERIC Document Reproduction Service No. ED411316).
29. Powers, D.E., Burstein, J.C., Fowles, M.E., & Kukich, K. (2001, March). Stumping e-rater: Challenging the validity of automated essay scoring. (GRE Board Research Report No. 98-08bP). Princeton, NJ: Educational Testing Service. Retrieved October 15, 2005, from http://www.ets.org/Media/ Research/pdf/RR-01-03-Powers.pdf
30. Reed, W. (1990, April). The effect of composing process software on the revision and quality of persuasive essays. Paper presented at the Eastern Educational Research Association, Clearwater, FL.
31. th Annual Meeting of the American Educational Research Association, Montreal, CA.
32. Rudner, L.M., Garcia, V., & Welch, C. (2006). An evaluation of the IntelliMetric™ essay scoring system. Journal of Technology, Learning, and Assessment, 4(4). Retrieved January 6, 2007, from http://escholarship. bc.edu/jtla/vol4/4/
33. Shermis, M.D., Koch, C.M., Page, E. B., Keith, T.Z., & Harrington, S. (2002). Trait ratings for automated essay grading. Educational and Psychological Measurement, 62(1), 5-18.
34. Texas Higher Education Coordinating Board. (n.d.). THEA, ASSET, COMPASS and ACCUPLACER. Retrieved September 12, 2006, from http://www.thecb.state.tx.us/ facts/cd/Page8.htm
35. Vantage Learning (2001a). Applying IntelliMetric™ to the scoring of entry-level college student essays. (RB-539). Retrieved September 1, 2005, from http://www.vantagelearning.com/content_pa ges/research.html
36. Vantage Learning (2001b). IntelliMetric™: From here to validity. (RB-504). Retrieved September 1, 2005, from http://www. vantagelearning.com/content_pa ges/research.html
37. Vantage Learning (2002). A study of expert scoring, standard human scoring and IntelliMetric™ scoring accuracy for statewide eighth grade writing responses. (RB-726). Retrieved September 1, 2005, from http://www.vantagelearning.com/content_pa ges/research.html
38. Vantage Learning. (2003). A comparison of IntelliMetric™ and expert scoring for the evaluation of literature essays. (RB-793). Retrieved September 1, 2005, from http://www.vantagelearning. com/content_pa ges/research.html
39. Wang, J. (2006). An analysis of automated essay scoring versus human scoring (Doctoral dissertation, Texas A&M University - Kingsville, 2006). Dissertation Abstracts International, 68(02).
40. Warschauer, M. & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10, 2, 1-24. Retrieved June 05, 2006, from http://www.gse.uci.edu/faculty/markw/awe.pdf
41. Wresch, W. (1993). The imminence of grading essays by computers - 25 years later. Computers and Composition 10(2), 45-58.
42. Yang, Y., Buckendahl, C.W., Juszkiewicz, P.J., & Bhola, D.S. (2002). A review of strategies for validating computer automated scoring. Applied Measurement in Education, 15(4), 391-412.