Automated essay evaluation software in English Language Arts classrooms: Effects on teacher feedback, student motivation, and writing quality

Computers and Education

Volumn 100, Issue , 2016, Pages 94-109

  Wilson, Joshua ^a   Czik, Amanda ^a  

^a UNIVERSITY OF DELAWARE   (United States)

Author keywords

Automated essay evaluation; English Language Arts; Interactive learning environments; Writing

Indexed keywords

AUTOMATION; COMPUTER AIDED INSTRUCTION; MOTIVATION; QUALITY CONTROL; STUDENTS; TECHNICAL WRITING;

AUTOMATED ESSAY EVALUATION; COMBINED FEEDBACKS; ENGLISH LANGUAGES; EVALUATION SOFTWARE; FEEDBACK TO STUDENTS; INTERACTIVE LEARNING ENVIRONMENT; STUDENT MOTIVATION; WRITING INSTRUCTION;

TEACHING;

EID: 84969663859     PISSN: 03601315     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.compedu.2016.05.004     Document Type: Conference Paper

References (92)

1. R.D. Abbott, V.W. Berninger, and M. Fayol Longitudinal relationships of levels of language in writing and between writing and reading in grades 1 to 7 Journal of Educational Psychology 102 2010 281 298
2. A.F. AbuSeileek Using track changes and word processor to provide corrective feedback to learners in writing Journal of Computer Assisted Learning 29 2013 319 333
3. Y. Ahmed, R.K. Wagner, and D. Lopez Developmental relations between reading and writing at the word, sentence and text levels: a latent change score analysis Journal of Educational Psychology 106 2014 419 434
4. L. Allal, L. Chanquoy, and P. Largy Revision: Cognitive and instructional processes 2004 Kluwer Academic Publishers Boston, MA
5. I.E. Allen, and C.A. Seaman Likert scales and data analyses Quality Progress 40 7 2007 64 65
6. American Psychological Association, Coalition for Psychology in Schools and Education Top 20 principles from psychology for preK-12 teaching and learning 2015 Retrieved from http://www.apa.org/ed/schools/cpse/top-twenty-principles.pdf
7. R.R. Andridge, and R.J.A. Little A review of hot deck imputation for survey non-response International Statistics Review 78 2010 40 64
8. C.M. Anson Response and the social construction of error Assessing Writing 7 2000 5 21
9. T. Asparouhov, and B. Muthén Multiple imputation with Mplus Mplus Web Notes 2010 Retrieved from http://ww.statmodel2.com/download/Imputations7.pdf
10. N.L. Baker "Get it off my stack": teachers' tools for grading papers Assessing Writing 19 2014 36 50
11. R.L. Bangert-Drowns, C.L. Kulik, J.A. Kulik, and M.T. Morgan The instructional effect of feedback in test-like events Review of Educational Research 61 1991 213 238
12. I.I. Bejar, M. Flor, Y. Futagi, and C. Ramineni On the vulnerability of automated scoring to construct-irrelevant response strategies (CIRS): an illustration Assessing Writing 22 2014 48 59
13. Y. Benjamini, and Y. Hochberg Controlling the false discovery rate: a practical and powerful approach to multiple testing Journal of the Royal Statistical Society. Series B (Methodological) 57 1995 289 300
14. D. Biber, T. Nekrasova, and B. Horn The effectiveness of feedback for L1-English and L2-writing development: A meta-analysis. TOEFL iBT™ research report 2011 Educational Testing Service Princeton, NJ
15. P. Black, and D. William Assessment and classroom learning Assessment in Education 5 1998 7 74
16. P. Black, and D. William Developing the theory of formative assessment Educational Assessment, Evaluation and Accountability 21 2009 5 31
17. M.B. Bunch, D. Vaughn, and S. Miel Automated scoring in assessment systems Y. Rosen, S. Ferrara, M. Mosharraf, Handbook of research on technology tools for real-world skill development 2016 IGI Global Hershey, PA 611 626
18. P.E. Chan, M. Konrad, V. Gonzalez, M.T. Peters, and V.A. Ressa The critical role of feedback in formative instructional practices Intervention in School and Clinic 50 2 2014 96 104
19. C.E. Chen, and W.E. Cheng Beyond the design of automated writing evaluation: pedagogical practices and perceived learning effectiveness in EFL writing classes Learning and Technology 12 2 2008 94 112
20. A.C.K. Cheung, and R.E. Slavin The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: a meta-analysis Educational Research Review 9 2013 88 113
21. K. Cho, C.D. Schunn, and D. Charney Commenting on writing: typology and perceived helpfulness of comments from novice peer reviewers and subject matter experts Written Communication 23 2006 260 294
22. L. Clare, R. Valdés, and G.G. Patthey-Chavez Learning to write in urban elementary and middle schools: An investigation of teachers' written feedback on student compositions Center for the Study of Evaluation Technical Report No. 526 2000 University of California, Center for Research on Evaluation, Standards, and Student Testing (CRESST) Los Angeles
23. Common Core State Standards Initiative Common core state standards for English language arts & literacy in history/social studies, science, and technical subjects 2010 Retrieved from http://www.corestandards.org/assets/CCSSI-ELA%20Standards.pdf
24. Conference on College Composition and Communication Writing assessment: A position statement 2014 Retrieved from http://www.ncte.org/cccc/resources/positions/writingassessment
25. L. DeGroff Process-writing teachers' responses to fourth grade writers' first drafts The Elementary School Journal 93 1992 131 144
26. S. Dikli, and S. Bleyle Automated essay scoring feedback for second language writers: how does it compare to instructor feedback? Assessing Writing 22 2014 1 17
27. H. Dujinhower, F.J. Prins, and K.M. Stokking Progress feedback effects on students' writing mastery goal, self-efficacy beliefs, and performance Educational Research and Evaluation 16 2010 53 74
28. P.F. Ericcson, and R.J. Haswell Machine scoring of student essays: Truth and consequences 2006 Utah State University Press Logan, UT
29. B. Ferster, T.C. Hammond, R.C. Alexander, and H. Lyman Automated formative assessment as a tool to scaffold student documentary writing Journal of Interactive Learning Research 23 2012 81 99
30. A. Field Discovering statistics using IBM SPSS statistics 4th ed. 2014 Sage Los Angeles, CA
31. P.W. Foltz Improving student writing through automated formative assessment: Practices and results Paper presented at the International Association for Educational Assessment (IAEA), Singapore 2014
32. P.W. Foltz, L.A. Streeter, K.E. Lochbaum, and T.K. Landauer Implementation and applications of the intelligent essay assessor M.D. Shermis, J. Burstein, Handbook of automated essay evaluation 2013 Routledge New York, NY 68 88
33. ®: computer support for comprehension and writing Journal of Educational Computing Research 33 2005 53 80
34. S. Graham, V. Berninger, and W. Fan The structural relationship between writing attitude and writing achievement in first and third grade students Contemporary Educational Psychology 32 3 2007 516 536
35. S. Graham, K.R. Harris, and T. Santangelo Research-based writing practices and the common Core Elementary School Journal 115 2015 498 522
36. S. Graham, M. Hebert, and K.R. Harris Formative assessment and writing: a meta-analysis Elementary School Journal 115 2015 523 547
37. S. Graham, M. Hebert, M.P. Sandbank, and K.R. Harris Assessing the writing achievement of young struggling writers: application of generalizability theory Learning Disability Quarterly 2014 1 11
38. S. Graham, D. McKeown, S. Kiuhara, and K.R. Harris A meta-analysis of writing instruction for students in the elementary grades Journal of Educational Psychology 104 2012 879 896
39. S. Graham, and D. Perin Writing next: Effective strategies to improve writing of adolescents in middle and high schools - A report to Carnegie Corporation of New York 2007 Alliance for Excellent Education Washington, DC
40. D. Grimes, and M. Warschauer Utility in a fallible tool: a multi-site case study of automated writing evaluation Journal of Technology, Learning, and Assessment 8 6 2010 1 44 Retrieved December 12, 2014 from http://www.jtla.org
41. S. Harward, N. Peterson, B. Korth, J. Wimmer, B. Wilcox, T.G. Morrison, and et al. Writing instruction in elementary classrooms: why teachers engage or do not engage students in writing Literacy Research and Instruction 53 2014 205 224
42. J. Hattie, and H. Timperley The power of feedback Review of Educational Research 77 2007 81 112
43. J.R. Hayes Modeling and remodeling writing Written Communication 29 3 2012 369 388
44. J.R. Hayes, and V.W. Berninger Relationships between idea generation and transcription: how the act of writing shapes what children write C. Braverman, R. Krut, K. Lundsford, S. McLeod, S. Null, P. Rogers, Traditions of writing research 2010 Routledge New York, NY 166 180
45. A. Herrington, and C. Moran What happens when machines read our students' writing? College English 63 4 2001 480 499
46. D. Higgins, and M. Heilman Managing what we can measure: quantifying the susceptibility of automated scoring systems to gaming behavior Educational Measurement Issues and Practice 33 3 2014 36 46
47. F. Hyland, and K. Hyland Sugaring the pill: praise and criticism in written feedback Journal of Second Language Writing 10 3 2001 185 212
48. T.Z. Keith Validity and automated essay scoring systems M.D. Shermis, J.C. Burstein, Automated essay scoring: A cross-disciplinary perspective 2003 Lawrence Erlbaum Associates, Inc Mahwah, NJ 147 167
49. R.T. Kellogg, and A.P. Whiteford Training advanced writing skills: the case for deliberative practice Educational Psychologist 44 2009 250 266
50. R.T. Kellogg, A.P. Whiteford, and T. Quinlan Does automated feedback help students learn to write? Journal of Educational Computing Research 42 2 2010 173 196
51. A.N. Kluger, and A. DeNisi Feedback interventions: toward the understanding of a double-edged sword Current Directions in Psychological Science 7 3 1998 67 72
52. I. Lee Feedback in writing: issues and challenges Assessing Writing 19 2014 1 5
53. H.B. Mann, and D.R. Whitney On a test of whether one of two random variables is stochastically larger than the other Annals of Mathematical Statistics 18 1947 50 60
54. B. Marshall, and M.J. Drummond How teachers engage with assessment for learning: lessons from the classroom Research Papers in Education 21 2006 133 149
55. L.C. Matsumara, G.G. Patthey-Chavez, R. Valdés, and G. Garnier Teacher feedback, writing assignment quality, and third-grade students' revision in lower- and higher-achieving urban schools Elementary School Journal 103 2002 3 25
56. S. Messick Validity of performance assessments G.W. Philips, Technical issues in large-scale performance assessment 1996 National Center for Educational Statistics Washington, DC 1 18
57. P. Morphy, and S. Graham Word processing programs and weaker writers/readers: a meta-analysis of research findings Reading and Writing 25 2012 641 678
58. L.K. Muthén, and B.O. Muthén Mplus user's guide 7th ed. 1998-2012 Muthén & Muthén Los Angeles, CA
59. National Assessment Governors Board Writing framework for the 2011 National Assessment of Educational Progress 2010 U.S. Department of Education Washington, DC
60. National Council of Teachers of English NCTE position statement on machine scoring 2013 Retrieved from http://www.ncte.org/positions/statements/machine-scoring
61. M.M. Nelson, and C.D. Schunn The nature of feedback: how different types of peer feedback affect writing performance Instructional Science 37 2009 375 401
62. E.B. Page The imminence of grading essays by computer Phi Delta Kappan 48 1966 238 243
63. E.B. Page Computer grading of student prose, using modern concepts and software The Journal of Experimental Education 62 2 1994 127 142
64. E.B. Page Project essay grade: PEG M.D. Shermis, J.C. Burstein, Automated essay scoring: A cross-disciplinary perspective 2003 Lawrence Erlbaum Associates, Inc Mahwah, NJ 43 54
65. F. Pajares Self-efficacy beliefs, motivation, and achievement in writing: a review of the literature Reading & Writing Quarterly 19 2003 139 158 10.1080/10573560308222
66. J.M. Parr, and H.S. Timperley Feedback to writing, assessment for teaching and learning and student progress Assessing Writing 15 2010 68 85
67. S. Peterson, R.M. Childs, and K. Kennedy Written feedback and scoring of sixth-grade girls' and boys' narrative and persuasive writing Assessing Writing 9 2004 160 180
68. J.L. Peugh, and C.K. Enders Missing data in educational research: a review of reporting practices and suggestions for improvement Review of Educational Research 74 2004 525 556
69. C.L. Piazza, and C.F. Siebert Development and validation of a writing dispositions scale for elementary and middle school students The Journal of Educational Research 101 5 2008 275 286
70. C. Ramineni, T. Calico, and C. Li Integrating product and process data in an online automated writing evaluation system O.C. Santos, Proceedings of the 8th International Conference on Educational Data Mining 2015 Madrid, Spain
71. D.B. Rubin Multiple imputation for nonresponse in surveys 1987 John Wiley & Sons New York
72. J.L. Schafer Analysis of incomplete multivariate data 1997 Chapman & Hall London
73. D.M. Schmelzer Turning out better writers: practice makes perfect, and computer-based writing makes for more practice Multimedia & Internet @ Schools 11 3 2004 34
74. R. Schoonen The validity and generalizability of writing scores. The effects of rater, task, and language E. Van Steendam, M. Tillema, G. Rijlaarsdam, H. Van Den Bergh, Measuring writing: Recent insights into theory, methodology, and practices 2012 Brill Leiden, The Netherlands 1 22
75. M.D. Shermis State-of-the-art automated essay scoring: competition, results, and future directions from a United States demonstration Assessing Writing 20 2014 53 76
76. M.D. Shermis, and J.C. Burstein Handbook of automated essay evaluation 2013 Routledge New York, NY
77. M.D. Shermis, J.C. Burstein, N. Elliot, S. Miel, and P.W. Foltz Automated writing evaluation: an expanding body of knowledge C.A. McArthur, S. Graham, J. Fitzgerald, Handbook of writing research 2nd ed. 2015 Guilford New York, NY 395 409
78. M.D. Shermis, C.W. Garvan, and Y. Diao The impact of automated essay scoring on writing outcomes Paper presented at the annual meeting of the National Council on Measurement in Education, New York, NY 2008, March
79. M.D. Shermis, C.M. Koch, E.B. Page, T.Z. Keith, and S. Harrington Trait ratings for automated essay grading Educational and Psychological Measurement 62 2002 5 18
80. V.J. Shute Focus on formative feedback Review of Educational Research 78 2008 153 189
81. M. Stevenson, and A. Phakiti The effects of computer-generated feedback on the quality of writing Assessing Writing 19 2014 51 65
82. D. Thissen, L. Steinberg, and D. Kuang Quick and easy implementation of the Benjamini-Hochberg Procedure for controlling the false positive rate in multiple comparisons Journal of Educational and Behavioral Statistics 27 2002 77 83
83. G.A. Troia Writing instruction for students with learning disabilities C.A. MacArthur, S. Graham, J. Fitzgerald, Handbook of writing research 2006 Guilford New York, NY 324 336
84. G.A. Troia, A.G. Harbaugh, R.K. Shankland, K.A. Wolbers, and A.M. Lawrence Relationships between writing motivation, writing activity, and writing performance: effects of grade, sex, and ability Reading and Writing 26 2013 17 44
85. U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse What Works Clearinghouse: Procedures and standards handbook (version 3.0) 2013, March Retrieved from http://whatworks.ed.gov
86. D. Wade-Stein, and E. Kintsch Summary street: interactive computer support for writing Cognition and Instruction 22 2004 333 362
87. M. Warschauer, and D. Grimes Automated writing assessment in the classroom Pedagogies: An International Journal 3 2008 22 36
88. L. Wilkinson, and Task Force on Statistical Inference Statistical methods in psychology journals: guidelines and explanations American Psychologist 54 1999 594 604
89. J.C.F. de Winter, and D. Dodou Five-point Likert items: T-test versus Mann-Whitney-Wilcoxon Practical Assessment, Research & Evaluation 15 11 2010 1 16
90. J. Wilson, and G.N. Andrada Using automated feedback to improve writing quality: Opportunities and challenges Y. Rosen, S. Ferrara, M. Mosharraf, Handbook of research on technology tools for real-world skill development 2016 IGI Global Hershey, PA 678 703
91. J. Wilson, N.G. Olinghouse, and G.N. Andrada Does automated feedback improve writing quality? Learning Disabilities: A Contemporary Journal 12 2014 93 118
92. S. Zumbrunn, S. Mars, and C. Mewborn Toward a better understanding of student perceptions of written feedback: a mixed methods study Reading and Writing 29 2016 349 370