State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration

Assessing Writing

Volumn 20, Issue , 2014, Pages 53-76

  Shermis, Mark D ^a  

^a UNIVERSITY OF AKRON   (United States)

Author keywords

Automated essay scoring; High stakes assessment; Human raters; Performance assessment; Race to the Top; Writing

Indexed keywords

EID: 84897599648     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.asw.2013.04.001     Document Type: Article

References (34)

1. Ajay H.B., Tillett P.I., Page E.B. Analysis of essays by computer (AEC-II) 1973, 231. U.S. Department of Health, Education, and Welfare, Office of Education, National Center for Educational Research and Development, Washington, DC.
2. Applebee A.N., Langer J. What is happening in the teaching of writing?. English Journal 2009, 98(5):18-22.
3. Attewell P., Lavin D., Domina T., Levey T. New evidence on college remediation. Journal of Higher Education 2006, 55(5):886-924.
4. Bennett R.E. Automated scoring of constructed-response literacy and mathematics items Advancing Consortium Assessment Reform (ACAR) 2011, Arabella Philanthropic Advisors, Washington, DC.
5. Bowen W.G., Chingos M.M., McPherson M.S. Crossing the finish line: Completing college at America's public universities 2009, Princeton University Press, Princeton, NJ.
6. Bridgeman B., Tripani C., Attali Y. Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education 2012, 25:27-40.
7. Burstein J., Kukich K., Wolff S., Lu C., Chodorow M. Computer analysis of essays. Proceedings of Proceedings of the NCME Symposium on Automated Scoring 1998.
8. Condon W. Large-scale assessment, locally-developed measures, and automated scoring of essays: Fishing for red herrings?. Assessing Writing 2013, 18:100-108.
9. Elliot S. IntelliMetric: From here to validity. Automated essay scoring: A cross-disciplinary perspective 2003, 71-86. Erlbaum Associates, Mahwah, NJ. M.D. Shermis, J. Burstein (Eds.).
10. Finkel J.R., Grenager T., Manning C. Incorporating non-local information into information extraction systems by Gibbs Sampling. Paper presented at the 43rd annual meeting of the Association for Computational Linguistics 2005, http://nlp.stanford.edu/~manning/papers/gibbscrf3.pdf.
11. Kane M.T. Validation. Educational measurement 2006, 17-64. American Council on Education/Praeger, Westport, CT. 4th ed. R.L. Brennan (Ed.).
12. Klobucar A., Deane P., Elliot N., Ramineni C., Deess P., Rudniy A. Automated essay scoring and the search for valid writing assessment. International advances in writing research: Cultures, places, measures 2012, 103-119. The WAC Clearninghouse, Fort Collins, CO. C. Bazerman, C. Dean, J. Early, K. Lunsford, S. Null, P. Rogers, A. Stansell (Eds.).
13. Landauer T.K., Foltz P.W., Laham D. Introduction to latent semantic analysis. Discourse Processes 1998, 25(2-3):259-284.
14. Landauer T.K., Laham D., Rehder B., Schreiner M.E. How well can passage meaning be derived without using word order? A comparison of latent semantic analysis and humans. Proceedings of the 19th annual meeting of the Cognitive Science Society 1997, 412-417. Lawrence Erlbaum Associates, Inc., Mahwah, NJ. M.G. Shafto, P. Langley (Eds.).
15. Mayfield E., Rosé C. An interactive tool for supporting error analysis for text mining. Paper presented at the demonstration session at the international conference of the North American Association for Computational Linguistics 2010.
16. Mayfield E., Rosé C. LightSIDE: Open source machine learning for text. Handbook of automated essay assessment: Current applications and new directions 2013, 124-135. Routledge, New York, NY. M.D. Shermis, J. Burstein (Eds.).
17. Mazzeo J., Lazer S., Ziesky M.J. Monitoring educational progress with group-score assessments. Educational measurement 2006, 469-481. American Council on Education/Praeger, Westport, CT. R.L. Brennan (Ed.).
18. McCurry D. Can machine scoring deal with broad and open writing tests as well as human raters?. Assessing Writing 2010, 15:118-129.
19. Mislevy R.J. Validity by design. Educational Researcher 2007, 36(8):463-469.
20. onward Kaggle Inc. (2010). Home Page. Retrieved from http://www.kaggle.com/.
21. Page E.B. The imminence of grading essays by computer. Phi Delta Kappan 1966, 48:238-243.
22. Porter A., McMaken J., Hwang J., Yang R. Common Core Standards: The new U.S. intended curriculum. Educational Researcher 2011, 40(3):103-116.
23. Ramineni C., Williamson D.M. Automated essay scoring: Psychometric guidelines and practices. Assessing Writing 2013, 18:25-39.
24. Salton G., Wong A., Yang C.S. A vector space model for automatic indexing. Communications of the ACM 1975, 18(11):613-620.
25. Scriven M. Validity in personnel evaluation. Journal of Personnel Evaluation in Education 1987, 1:9-23.
26. Shermis M.D., Burstein J. Automated essay scoring: A cross-disciplinary perspective 2003, Lawrence Erlbaum Associates, Inc., Mahwah, NJ.
27. Handbook of automated essay evaluation: Current applications and new directions 2013, Routledge, New York, NY. M.D. Shermis, J. Burstein (Eds.).
28. Shermis M.D., Hamner B. Contrasting state-of-the-art automated scoring of essays: Analysis. Paper presented at the National Council of Measurement in Education 2012.
29. Shermis M.D., Hamner B. Contrasting state-of-the-art automated scoring of essays. Handbook of automated essay evaluation: Current applications and new directions 2013, 313-346. Routledge, New York, NY. M.D. Shermis, J. Burstein (Eds.).
30. Stemler S.E. A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation 2004, 9(4). http://pareonline.net/getvn.asp%3Fv=9%26n=4.
31. Tucker B. The next generation of testing. Multiple Measures 2009, 67(3):48-53.
32. Weigle S.C. English language learners and automated scoring of essays: Critical considerations. Assessing Writing 2013, 18:85-99.
33. Weir C.J. Limitations of the Common European Framework for developing comparable examinations and tests. Language Testing 2005, 22(3):261-300.
34. Williamson D.M., Xi X., Breyer F.J. A framework for the evaluation and use of automated essay scoring. Educational Measurement: Issues and Practice 2012, 31(1):2-13.