On the relation between automated essay scoring and modern views of the writing construct

Assessing Writing

Volumn 18, Issue 1, 2013, Pages 7-24

  Deane, Paul ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

Assessment; Automated essay scoring (AES); Cognitively Based Assessments of, for, and as Learning (CBAL); Writing; Writing construct

Indexed keywords

EID: 84872414967     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.asw.2012.10.002     Document Type: Article

References (102)

1. Anson C.M. Closed systems and standardized writing tests. College Composition and Communication 2008, 60:113-128.
2. Attali Y., Bridgeman B., Trapani C. Performance of a generic approach in automated essay scoring. Journal of Technology, Learning & Assessment 2010, 10. Retrieved from. http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1603/1455.
3. Attali Y., Burstein J. Automated essay scoring with e-rater v.2. Journal of Technology, Learning & Assessment 2006, 4. Retrieved from. http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1650/1492.
4. Attali, Y., & Powers, D. (2008). A developmental writing scale (ETS Research Report RR-08-19). Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/Media/Research/pdf/RR-08-19.pdf.
5. Beck S.W., Jeffery J. Genres of high stakes writing assessments and the construct of writing competence. Assessing Writing 2007, 12:60-79.
6. Bejar I.I. A validity-based approach to quality control and assurance of automated scoring. Assessment in Education: Principles, Policy & Practice 2011, 18:319-341.
7. Belanoff P., Dickson M. Portfolios: Process and product 1991, Heinemann, Portsmouth, NH.
8. Bennett R.E. Moving the field forward: Some thoughts on validity and automated scoring. Automated scoring of complex tasks in computer-based testing 2006, 403-412. Lawrence Erlbaum, Hillsdale, NJ. D.M. Williamson, R.J. Mislevy, I.I. Bejar (Eds.).
9. Bennett R.E. Automated scoring of constructed-response literacy and mathematics items 2011, Arabella Philanthropic Advisors, Washington, DC, Retrieved from http://www.ets.org/s/k12/pdf/k12_commonassess_automated_scoring_math.pdf.
10. Bennett, R. E. (2011b). CBAL: Results from piloting innovative K-12 assessments (ETS Research Report RR-11-23). Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/Media/Research/pdf/RR-11-23.pdf.
11. Ben-Simon A., Bennett R.E. Toward more substantively meaningful automated essay scoring. Journal of Technology, Learning & Assessment 2007, 6. Retrieved from. http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1631/1475.
12. Ben-Simon, A., & Cohen, Y. (2011). The Hebrew Language Project: Automated essay scoring & readability analysis. Paper presented at the annual meeting of the IAEA, Manila.
13. Bereiter C., Scardamalia M. The psychology of written composition 1987, Lawrence Erlbaum, Hillsdale, NJ.
14. Black L., Daiker D., Sommers J., Stygall G. New directions in portfolio assessment 1994, Heinemann, Portsmouth, NH.
15. Braun H.I., Bennett R.E., Frye D., Soloway E. Scoring constructed responses using expert systems. Journal of Educational Measurement 1990, 27:93-108.
16. Brent E., Townsend M. Automated essay scoring in the classroom: Finding common ground. Machine scoring of student essays: Truth and consequences 2006, 177-198. Utah State University Press, Logan, UT. P.F. Ericsson, R.H. Haswell (Eds.).
17. Bridgeman B., Trapani C.S., Attali Y. Comparison of human and machine scoring of essays: Differences by gender, ethnicity and country. Applied Measurement in Education 2012, 25:27-40.
18. Brigham C.C. The reading of the comprehensive examination in English: An analysis of the procedures followed during the five reading periods from 1929 to 1933 1933, Princeton University Press, Princeton.
19. Broad B., Adler-Kassner L., Alford B., Detweiler J., Estrem H., Harrington S., et al. Organic writing assessment: Dynamic criteria mapping in action 2009, Utah State University Press, Logan, UT.
20. Burstein J., Chodorow M. Directions in automated essay scoring. Handbook of applied linguistics 2003, 487-497. Oxford University Press, Oxford, UK. R. Kaplan (Ed.).
21. Burstein J., Chodorow M. Progress and new directions in technology for automated essay evaluation. The Oxford handbook of applied linguistics 2010, 487-497. Oxford University Press, Oxford, UK. 2nd ed. R. Kaplan (Ed.).
22. Burstein J., Chodorow M., Leacock C. Criterion: Online essay evaluation: An application for automated evaluation of student essays. Proceedings of the fifteenth annual conference on innovative applications of artificial intelligence 2003, 3-10. Association for the Advancement of Artificial Intelligence, Acapulco, Mexico.
23. Burstein J., Marcu D. Automated evaluation of discourse structure in student essays. Automated essay scoring: A cross-disciplinary perspective 2003, 200-219. Lawrence Erlbaum, Mahwah, NJ. M.D. Shermis, J. Burstein (Eds.).
24. Burstein J., Marcu D., Andreyev S., Chodorow M. Towards automatic classification of discourse elements in essays. Proceedings of the 39th annual meeting of the association for computational linguistics 2001, 98-105. Association for Computational Linguistics, Toulouse, France.
25. Burstein J., Marcu D., Knight K. Finding the WRITE stuff: Automatic identification of discourse structure in student essays. IEEE Intelligent Systems: Special Issue on Advances in Natural Language Processing 2003, 18:32-39.
26. Burstein J., Tetreault J., Andreyev S. Using entity-based features to model coherence in student essays. Human language technologies: The 2010 annual conference of the North American chapter of the ACL 2010, 681-684. Association for Computational Linguistics.
27. Electronic portfolios 2.0: Emergent research on implementation and impact 2009, Stylus, Washington, DC. D. Cambridge, B. Cambridge, K.B. Yancey (Eds.).
28. Castro-Castro D., Lannes-Losada R., Maritxalar M., Niebla I., Pérez-Marqués C., álamo-Suárez N.C., et al. A multilingual application for automated essay scoring. Advances in artificial intelligence. Lecture Notes in Computer Science 2008, 5290:243-251.
29. Chang T.-H., Lee C.H., Tam H.-P. On issues of feature extraction in Chinese automatic essay scoring system. Proceedings of the 2007 conference on artificial intelligence in education: building technology rich learning contexts that work 2007, 545-547. IOS Press, Amsterdam.
30. Chang T.-H., Lee C.-H., Chang Y.-M. Enhancing automatic Chinese essay scoring system from figures-of-speech. Proceedings of the twentieth pacific Asia conference on language, information and computation 2006, 28-34.
31. Chenoweth N.A., Hayes J.R. Fluency in writing: Generating text in L1 and L2. Written Communication 2001, 18:80-98.
32. Cheville J. Automated scoring technologies and the rising influence of error. English Journal 2004, 93:47-52.
33. Chodorow, M., & Burstein, J. (2004). Beyond essay length: Evaluating e-rater's performance on TOEFL® Essays (TOEFL® Research Report RR-73, ETS Research Report RR-04-04). Princeton, NJ: Educational Testing Service.
34. Chodorow M., Gamon M., Tetreault J. The utility of article and preposition error correction systems for English Language Learners: Feedback and assessment. Language Testing 2010, 27:419-436.
35. Chodorow M., Leacock C. Techniques for detecting syntactic errors in text. Technical Report of the Institute of Electronics, Information, and Communication Engineers 2002, 102:37-41.
36. Cohen, Y., Ben-Simon, A., & Hovav, M. (2003). The effect of specific language features on the complexity of systems for automated essay scoring. Paper presented at the international association of educational assessment annual conference, October 2003, Manchester.
37. Conference on College Composition and Communication (2004). Position statement on teaching, learning, and assessing writing in digital environments. Retrieved from http://www.ncte.org/cccc/resources/positions/digitalenvironments.
38. Condon W. The future of portfolio-based writing assessment: A cautionary tale. Writing assessment in the 21st century: Essays in honor of Edward M. White 2012, 233-245. Hampton Press, New York, NY. N. Elliot, L. Perelman (Eds.).
39. Evaluating writing: Describing, measuring, judging 1977, NCTE, Urbana, IL. C.R. Cooper, L. Odell (Eds.).
40. Council of Europe Common European framework of reference for languages: Learning, teaching, assessment 2001, Cambridge University Press, Cambridge, UK.
41. Council of Writing Program Administrators, National Council of Teachers of English & National Writing Project. (2011). Framework for success in postsecondary writing. Retrieved from http://wpacouncil.org/files/framework-for-success-postsecondary-writing.pdf.
42. Cronbach L.J. Test validation. Educational measurement 1971, 443-507. American Council on Education, Washington, DC. 2nd ed. R.L. Thorndike (Ed.).
43. Deane, P. (2011). Writing assessment and cognition (ETS Research Report RR-11-14). Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/Media/Research/pdf/RR-11-14.pdf.
44. Deane P., Quinlan T. What automated analyses of corpora can tell us about students' writing skills. Journal of Writing Research 2010, 2:151-177.
45. Diederich P.B., French J.W., Carlton S.T. Factors in judgments of writing ability (ETS research bulletin RB-61-15) 1961, Educational Testing Service, Princeton, NJ.
46. Elliot N. On a scale: A social history of writing assessment in America 2005, Peter Lang, New York, NY.
47. Elliot S. IntelliMetric: from here to validity. Automated essay scoring: a cross disciplinary approach 2003, Lawrence Erlbaum Associates, Mahwah, NJ. M.D. Shermis, J.C. Burstein (Eds.).
48. Ericsson P.F. The meaning of meaning: Is a paragraph more than an equation?. Machine scoring of student essays: Truth and consequences 2006, 28-37. Utah State University Press, Logan, UT. P.F. Ericsson, R.H. Haswell (Eds.).
49. Machine scoring of student essays: Truth and consequences 2006, Utah State University Press, Logan, UT. P.F. Ericsson, R.H. Haswell (Eds.).
50. Feng W.F., Hirst G. Classifying arguments by scheme. Proceedings of the 49th annual meeting of the association for computational linguistics 2011, 987-996. Association for Computational Linguistics, Portland, Oregon.
51. Flower L. The construction of negotiated meaning: A social cognitive theory of writing 1994, Southern Illinois Press, Carbondale, IL.
52. Futagi Y., Deane P., Chodorow M., Tetreault J. A computational approach to detecting collocation errors in the writing of non-native speakers of English. Computer Assisted Language Learning 2008, 21:353-367.
53. Godshalk F.I., Swineford F., Coffman W.E. The measurement of writing ability 1966, College Entrance Examination Board, New York, NY.
54. Haberman, S. J. (2011). Use of e-rater in scoring of the TOEFL iBT Writing Test (ETS Research Report RR-11-25). Princeton: Educational Testing Service. Retrieved from http://www.ets.org/Media/Research/pdf/RR-11-25.pdf.
55. Haberman S.J., Sinharay S. The application of the cumulative logistic regression model to automated essay scoring. Journal of Educational & Behavioral Statistics 2010, 35:586-602.
56. Hamp-Lyons L., Condon W. Assessing the portfolio: Principles for practice, theory, and research 2000, Hampton Press, Creskill, NJ.
57. Haswell R.H. Automatons and automated scoring: Drudges, black boxes, and dei ex machina. Machine scoring of student essays: Truth and consequences 2006, 57-78. Utah State University Press, Logan, UT. P.F. Ericsson, R.H. Haswell (Eds.).
58. Hayes J.R. From idea to text. The Sage handbook of writing development 2009, 65-79. Sage, London, UK. R. Beard, D. Myhill, J. Riley, M. Nystrand (Eds.).
59. Hayes J.R. Modeling and remodeling writing. Written Communication 2012, 29:369-388.
60. Herrington A., Moran C. Writeplacer plus in place: An exploratory case study. Machine scoring of student essays: Truth and consequences 2006, 114-129. Utah State University Press, Logan, UT. P.F. Ericsson, R.H. Haswell (Eds.).
61. SM: Promoting the standard. Race and writing assessment 2012, 47-61. Peter Lang, New York, NY. A.B. Inoue, M. Poe (Eds.).
62. Higgins D., Burstein J., Attali Y. Identifying off-topic student essays without topic-specific training data. Natural Language Engineering 2006, 12:145-159.
63. Higgins D., Burstein J., Marcu D., Gentile C. Evaluating multiple aspects of coherence in student essays. HLT-NAACL 2004: Main proceedings 2004, 185-192. Association for Computational Linguistics, Boston, MA. S. Dumais, D. Marcu, S. Roukos (Eds.).
64. Jones E. ACCUPLACER'S essay-scoring technology: When reliability does not equal validity. Machine scoring of student essays: Truth and consequences 2006, 93-113. Utah State University Press, Logan, UT. P.F. Ericsson, R.H. Haswell (Eds.).
65. Kellogg R.T. Attentional overload and writing performance: Effects of rough draft and outline strategies. Journal of Experimental Psychology: Learning, Memory and Cognition 1988, 14:355-365.
66. Kellogg R.T. Improving the writing skills of college students. Psychonomic Bulletin & Review 2007, 14:237-242.
67. Klobucar A., Deane P., Elliot N., Ramineni C., Deess P., Rudniy A. Automated essay scoring and the search for valid writing assessment. International advances in writing research: Cultures, places, measures 2012, 103-119. WAC Clearinghouse/Anderson, SC: Parlor Press, Fort Collins, Colorado. http://wac.colostate.edu/books/wrab2011/chapter6.pdf, C. Bazerman, C. Dean, J. Early, K. Lunsford, S. Null, P. Rogers, A. Stansell (Eds.).
68. Landauer T.K., Laham D., Foltz P.W. Learning human-like knowledge by singular value decomposition: A progress report. Advances in neural information processing systems 10 1988, 45-51. MIT Press, Cambridge, MA. M.I. Jordan, M.J. Kearns, S.A. Solla (Eds.).
69. Leacock, C., & Chodorow, M. (2001). Automatic assessment of vocabulary usage without negative evidence (TOEFL® Research Report 67; ETS Research Report RR-01-21). Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/Media/Research/pdf/RR-01-21.pdf.
70. Lemaire B., Dessus P. A system to assess the semantic content of student essays. Journal of Educational Computing Research 2001, 24:305-320.
71. Lockridge, S. (2012, April). Truscore. Paper presented as part of the symposium on contrasting state-of-the-art in automated scoring of essays at the annual meeting of the National Council on Measurement in Education, Vancouver.
72. McCutchen D. A capacity theory of writing: Working memory in composition. Educational Psychology Review 1996, 8:299-325.
73. McCutchen D. Knowledge, processing, and working memory: Implications for a theory of writing. Educational Psychologist 2000, 35:13-23.
74. McCutchen D., Covill A., Hoyne S.H., Mildes K. Individual differences in writing: Implications of translating fluency. Journal of Educational Psychology 1994, 86:256-266.
75. McGee T. Taking a spin on the Intelligent Essay Assessor. Machine scoring of student essays: Truth and consequences 2006, 79-92. Utah State University Press, Logan, UT. P.F. Ericsson, R.H. Haswell (Eds.).
76. Messick S. Validity. Educational measurement 1989, 13-103. American Council on Education and Macmillan, New York, NY. 3rd ed. R.K. Linn (Ed.).
77. Mislevy R.J. Cognitive psychology and educational assessment. Educational measurement 2006, 257-305. American Council on Education/Praeger, Westport, CT. 4th ed. R.L. Brennan (Ed.).
78. Mislevy R.J. Validity by design. Educational Researcher 2007, 36:463-469. 10.3102/0013189X07311660.
79. Mislevy R.J., Haertel G.D. Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice 2006, 25:6-20.
80. Insights from the common European framework 2004, Oxford University Press, Oxford, UK. K. Morrow (Ed.).
81. Murphy S., Camp R. Toward systemic coherence: A discussion of conflicting perspectives on portfolio assessment. Writing portfolios in the classroom: Policy and practice, promise and peril 1996, Lawrence Erlbaum, Mahwah, NJ. R. Calfee, P. Perfumo (Eds.).
82. Murphy S., Yancey K.B. Construct and consequence: Validity in writing assessment. Handbook of writing research: History, society, school, individual, text 2008, 365-386. Lawrence Erlbaum, New York, NY. C. Bazerman (Ed.).
83. Noyes E.A., Sale W.M., Stalnaker J.M. Report on the first six tests in English composition, with sample answers from the tests of April and June, 1944 1945, College Entrance Examination Board, New York, NY.
84. O'Neill P., Adler-Kassner L., Fleischer C., Hall A.M. Creating the framework for success in postsecondary writing. College English 2012, 76:520-524.
85. Perelman L. Construct validity, length, score, and time in holistically graded writing assessments: The case against automated essay scoring (AES). International advances in writing research: Cultures, places, measures 2012, 121-131. WAC Clearinghouse/Anderson, SC: Parlor Press, Fort Collins, Colorado. http://wac.colostate.edu/books/wrab2011/chapter7.pdf, C. Bazerman, C. Dean, J. Early, K. Lunsford, S. Null, P. Rogers, A. Stansell (Eds.).
86. Powers, D. E., Burstein, J., Chodorow, M., Fowles, M. E., & Kukich, K. (2001). Stumping e-rater: Challenging the validity of automated essay scoring (GRE® Board Professional Report 98-08bP, ETS Research Report RR-01-03). Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/Media/Research/pdf/RR-01-03-Powers.pdf.
87. Quinlan, T., Higgins, D., & Wolf, S. (2009). Evaluating the construct coverage of the e-rater scoring engine (ETS Research Report RR-09-01). Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/Media/Research/pdf/RR-09-01.pdf.
88. Ruth L., Murphy S. Designing topics for writing assessment: Problems of meaning. College Composition and Communication 1984, 35(4):410-422.
89. Ruth L., Murphy S. Designing writing tasks for the assessment of writing 1988, Ablex, Norwood, NJ.
90. Sebrechts M.M., Bennett R.E., Rock D.A. Agreement between expert system and human raters' scores on complex constructed-response quantitative items. Journal of Applied Psychology 1991, 76:856-862.
91. Automated essay scoring: A cross-disciplinary perspective 2003, Lawrence Erlbaum, Mahwah, NJ. M.D. Shermis, J. Burstein (Eds.).
92. Shermis, M. D., & Burstein, J. (in press). Handbook of automated essay evaluation: current applications and new directions. New York, NY: Routledge.
93. Shermis M.D., Burstein J., Higgins D., Zechner K. Automated essay scoring: Writing assessment and instruction. International encyclopedia of education 2010, Vol. 4:20-26. Elsevier Science, Oxford, UK. 3rd ed.
94. Shermis, M. D., & Hamner, B. (in press). Contrasting state-of-the-art automated scoring of essays. In: M. D. Shermis, & J. Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new directions. New York, NY: Routledge.
95. Spandel V. Creating writers through 6-trait writing assessment and instruction 2005, Pearson, Allyn & Bacon, Boston, MA. 4th ed.
96. Torrance M., Galbraith D. The processing demands of writing. Handbook of writing research 2008, 67-82. The Guilford Press, New York, NY. C.A. MacArthur, S. Graham, J. Fitzgerald (Eds.).
97. White E. Teaching and assessing writing: Recent advances in understanding, evaluating, and improving student performance 1985, Jossey-Bass, San Francisco, CA.
98. Weigle S.C. English language learners and automated scoring of essays: Critical considerations. Assessing Writing 2013, 18. 10.1016/j.asw.2012.10.006.
99. Wild F., Stahl C., Stermsek G., Penya Y., Neumann G. Factors influencing effectiveness in automated essay scoring with LSA 2005, 947-949. IOS Press, Amsterdam. C.K. Looij (Ed.).
100. Willard-Traub M.K. Assessing the portfolio: Hamp-Lyons and Condon [Rev. of the book Assessing the portfolio: Principles for practice, theory, and research, by L. Hamp-Lyons & W. Condon]. Assessing Writing 2002, 8:65-69.
101. Yancey K. The rhetorical situation of writing assessment: Exigence, location, and the making of knowledge. Writing assessment in the 21st century: Essays in honor of Edward M. White 2012, 475-492. Hampton Press, New York, NY. N. Elliot, L. Perelman (Eds.).
102. Yancey K., Weiser I. Situating portfolios: Four perspectives 1997, Utah State University Press, Logan, UT.