Automated versus human scoring: A case study in an EFL context

Electronic Journal of Foreign Language Teaching

Volumn 11, Issue SUPPL. 1, 2014, Pages 149-164

  Huang, Shih Jen ^a  

^a NATIONAL KAOHSIUNG UNIVERSITY OF APPLIED SCIENCES   (Taiwan)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84904092604     PISSN: None     EISSN: 02199874     Source Type: Journal    
DOI: None     Document Type: Article

References (47)

1. Attali, Y. (2004). Exploring the feedback and revision features of Criterion. Paper presented at the National Council on Measurement in Education, San Diego, CA.
2. Attali, Y., & Burstein, J. (2006). AES with e-rater V.2. The Journal of Technology, Learning, and Assessment, 4(3), 1-31.
3. Bejar, I. I. (2011). A validity-based approach to quality control and assurance of automated scoring. Assess-ment in Education: Principles, Policy & Practice, 18(3), 319-341.
4. Bridgeman, B., Trapani, C., & Yigal, A. (2012). Comparison of human and machine scoring of essays: Dif-ferences by gender, ethnicity, and country. Applied Measurement in Education 25(1), 27-40.
5. Burstein, J., Braden-Harder, L., Chodorow, M., Hua, S., Kaplan, B., Kukich, K., Lu, C., Nolan, J., Rock, D., & Wolff, S. (1998). Computer analysis of essay content for automated score prediction: A prototype auto-mated scoring system for GMAT analytical writing assessment essays. Princeton, NJ: Educational Testing Service.
6. Burstein, J., & Chodorow, M. (1999). AES for nonnative English speakers. Paper presented at the ACL99 Workshop on Computer-Mediated Language Assessment and Evaluation of Natural Language Processing, College Park, MD.
7. Burstein, J., Chodorow, M., & Leacock, C. (2003). Criterion online essay evaluation: An application for au-tomated evaluation of student essays. In J. Riedl & R. Hill (Eds.), Proceedings of the Fifteenth Annual Conference on Innovative Applications of Artificial Intelligence (pp. 3-10). Menlo Park, CA: AAAI Press.
8. Burstein, J, & Marcus, D. (2003). A machine learning approach for identification of thesis and conclusion statements in student essays. Computers and the Humanities, 37, 455-467.
9. Calfee, R. (2000). To grade or not to grade. IEEE Intelligent Systems, 15(5), 35-37.
10. Chen, C. F., & Cheng, W. Y. (2008). Beyond the design of automated writing evaluation: Pedagogical prac-tices and perceived learning effectiveness in EFL writing classes. Language Learning & Technology, 12(2), 94-112.
11. Chen, H. J., Chiu, T. L., & Liao, P. (2009). Analyzing the grammar feedback of two automated writing evalu-ation systems: My Access and Criterion. English Teaching and Learning, 33(2), 1-43.
12. Cheville, J. (2004). Automated scoring technologies and the rising influence of error. English Journal, 93(4), 47-52.
13. Chodorow, M., & Burstein, J. (2004). Beyond essay length: Evaluating e-rater's performance on TOEFL essays. Princeton, NJ: Educational Testing Service.
14. Crusan, D. (2010). Assessment in the second language writing classroom. Ann Arbor, MI: University of Michigan Press.
15. Donahue, C. (2005). Student writing as negotiation. In T. Kostouli (Ed.), Writing in context(s): Textual prac-tices and learning Processes in sociocultural settings (pp. 137-163). New York: Springer.
16. Elliot, S., & Mikulua, C. (2004). The impact of MyAccess use on student writing performance: A technology overview and four studies. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA.
17. Ericsson, P., & Haswell, R. (Eds.). (2006). Machine scoring of student essays: Truth and consequences. Lo-gan, UT: Utah State University Press.
18. Fang, Y. (2010). Perceptions of the computer-assisted writing program among EFL College Learners. Educa-tional Technology & Society, 13(3), 246-256.
19. Foltz, P. W., Landauer, T. K., & Laham, D. (1999). AES: applications to educational technology. In B. Collis & R. Oliver (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Tele-communications 1999 (pp. 939-944). Chesapeake, VA: AACE.
20. Grimes, D., & Warschauer, M. (2006). AES in the classroom. Paper presented at the American Educational Research Association, San Francisco, CA.
21. Higgins, D., Burstein, J., & Attali, Y. (2006). Identifying off-topic student essays without topic-specific train-ing data. Natural Language Engineering, 12, 145-159.
22. Higgins, D., Burstein, J., Marcu, D., & Gentile, C. (2004). Evaluating multiple aspects of coherence in stu-dents essays. Retrieved from: http://www.ets.org/media/research/pdf/erater_higgins_dis_coh.pdf
23. Hillocks, G. (2002). The testing trap: How state writing assessments control learning. New York: Teachers College Press.
24. Hyland, K., & Hyland, F. (2006). Feedback on second language students' writing. Language Teaching, 39(2), 83-101.
25. Kondo-Brown, K. (2002). A FACETS analysis of rater bias in measuring Japanese second language writing performance. Language Testing, 19, 3-31.
26. Lai, Y. H. (2010). Which do students prefer to evaluate their essays: Peers or computer program. British Journal of Educational Technology, 41(0), 432-454.
27. Landauer, T. K., Laham, D. L., & Foltz, P. W. (2003). Automatic essay assessment. Assessment in Education, 10(3), 295-308.
28. Larkey, L. (1998). Automatic essay grading using text categorization techniques. Paper presented at 21st In-ternational Conference of the Association for Computing Machinery-Special Interest Group on Information Retrieval (ACM-SIGIR), Melbourne, Australia. Retrieved from http://ciir.cs.umass.edu/pubfiles/ir-121.pdf.
29. Mason, O., & Grove-Stephenson, I. (2002). Automated free text marking with paperless school. In M. Dan-son (Ed.), Proceedings of the Sixth International Computer Assisted Assessment Conference (pp. 216-222). Loughborough: Loughborough University.
30. Ming, P. Y., Mikhailov, A. A., & Kuan, T. L. (2000). Intelligent essay marking system. In C. Cheers (Ed.), Learners together. Singapore: Ngee Ann Polytechnic. Retrieved from http://ipdweb.np.edu.sg/lt/feb00/ intelligent_essay_marking.pdf
31. Mitchell, T., Russel, T., Broomhead, P., & Aldridge N. (2002). Towards robust computerized marking of free-text responses. In M. Danson (Ed.), Proceedings of the Sixth International Computer Assisted Assess-ment Conference (pp. 233-249). Loughboroug: Loughborouh University.
32. Otoshi, J. (2005). An analysis of the use of Criterion in a writing class in Japan. The JALT CALL Journal, 1(1), 30-38
33. Page, E. B. (1968). The use of the computer in analyzing student essays. International Review of Education, 14, 210-225.
34. Powers, D. E., Burstein, J. C., Chodorow, M., Fowles, M. E., & Kukich, K. (2001). Stumping e-rater: Chal-lenging the validity of automated essay scoring. Princeton, NJ: Educational Testing Service.
35. Shermis, M. D., & Burstein, J. (2003). AES: A cross disciplinary perspective. NJ: Lawrence Erlbaum Associ-ate.
36. Shermis, M. D., & Burstein, J. (2013). The handbook of AES: Current applications and new directions. NJ: Lawrence Erlbaum Associate.
37. Shermis, M. D., Burstein, J., & Bliss, L. (2004). The impact of AES on higher stakes writing assessment. Pa-per presented at the Annual Meetings of American Education Research Association and the National Council on Measurement in Education Conference, San Diego, CA.
38. Shermis, M. D., Koch, C. M., Page, E. B., Keith, T. Z., & Harrington, S. (2002). Trait ratings for automated essay grading. Educational and Psychological Measurement, 62(1), 5-18.
39. Shohamy, E., Gordon, C. M., & Kraemer, R. (1992). The effect of raters' background and training on the reliability of direct writing tests. The Modern Language Journal, 76, 27-33.
40. Shi, L. (2001). Native- and nonnative-speaking EFL teachers' evaluation of Chinese students' English writing. Language Testing, 18, 303-325.
41. Tribble, C. (1996). Writing. Oxford: Oxford University Press.
42. Wang, J., & Brown, M. S. (2008). AES versus human scoring: A correlational study. Contemporary Issues in Technology and Teacher Education, 8(4), 310-325.
43. Wang, M. J., & Goodman, D. (2012). Automated writing evaluation: Students' perceptions and emotional involvement. English Teaching and Learning, 36(3), 1-37.
44. Wang, Y. J., Shang, H. F., & Briody, P. (2013). Exploring the impact of using automated writing evaluation in English as a foreign language university students' writing. Computer Assisted Language Learning, 26(3), 234-257.
45. Warschauer, M., & Grimes, D. (2008). Automated writing assessment in the classroom. Pedagogies: An In-ternational Journal, 3, 22-36.
46. Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 1(2), 1-24.
47. Yang, N. D. (2004). Using MyAccess in EFL writing. In The Proceedings of 2004 International Conference and Workshop on TEFL & Applied Linguistics (pp. 550-564). Taipei: Ming Chuan University.