Automated essay scoring: Psychometric guidelines and practices

Assessing Writing

Volumn 18, Issue 1, 2013, Pages 25-39

  Ramineni, Chaitanya ^a   Williamson, David M ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

Automated essay scoring; Constructed response items; Large scale writing assessments

Indexed keywords

EID: 84872386748     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.asw.2012.10.004     Document Type: Article

References (74)

1. Anson C.R. Responding to and assessing student writing: The uses and limits of technology. Teaching writing with computers 2003, 234-246. Houghton Mifflin, Boston, MA. P. Takayoshi, B. Huot (Eds.).
2. Attali Y. Modified CVA features (unpublished report) 2011, Educational Testing Service, Princeton, NJ.
3. Attali Y., Bridgeman B., Trapani C.S. Performance of a generic approach in automated essay scoring. Journal of Technology Learning, and Assessment 2010, 10(3). Retrieved from http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1603/1455.
4. Attali Y., Burstein J.C. Automated essay scoring with e-rater v.2. Journal of Technology Learning, and Assessment 2006, 4(3). Retrieved from http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1650/1492.
5. Bennett R.E. Moving the field forward: Some thoughts on validity and automated scoring. Automated scoring of complex-tasks in computer-based testing 2006, 403-412. Erlbaum, Mahwah, NJ. D.M. Williamson, R.J. Mislevy, I.I. Bejar (Eds.).
6. Bennett R.E. Automated scoring of constructed-response literacy and mathematics items. 2011, Arabella Philanthropic Advisors, Washington, DC, Retrieved from http://www.ets.org/s/k12/pdf/k12_commonassess_automated_scoring_math.pdf.
7. Bennett R.E., Bejar I.I. Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice 1998, 17(4):9-17.
8. Bernstein J., De Jong J., Pisoni D., Townshend B. Two experiments on automatic scoring of spoken language proficiency. Proceedings of InSTIL2000 (Integrating Speech Tech. in Learning) 2000, 57-61. University of Abertay, Dundee, Scotland.
9. Braun H.I. Understanding scoring reliability: Experiments in calibrating essay readers. Journal of Educational Statistics 1988, 13:1-18.
10. Braun H.I., Bejar I.I., Williamson D.M. Rule-based methods for automatic scoring: Application in a licensing context. Automated scoring for complex constructed response tasks in computer based testing 2006, 83-122. Erlbaum, Mahwah, NJ. D.M. Williamson, R.J. Mislevy, I.I. Bejar (Eds.).
11. Breland H., Kubota M., Nickerson K., Trapani C.S., Walker M. New SAT writing prompt study: Analyses of group impact and reliability. (Research Report 2004-1) 2004, College Board, New York, NY.
12. Bridgeman B., Trapani C.S., Attali Y. Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education 2012, 25:27-40.
13. ® scoring engine: Automated essay scoring with natural language processing. Automated essay scoring: A cross-disciplinary perspective 2003, 113-121. Erlbaum, Hillsdale, NJ. M.D. Shermis, J.C. Burstein (Eds.).
14. Callear D., Jerrams-Smith J., Soh V. CAA of short non-MCQ answers. Proceedings of the 5th International CAA Conference 2001, 55-69. Loughborough University, Loughborough, UK.
15. Chevalier S. Speech interaction with Saybot player, a CALL software to help Chinese learners of English. Proceedings of The International Speech Communication Association Special Interest Group on Speech and Language Technology in Education (SLaTE) 2007, Retrieved from http://www.cs.cmu.edu/~max/mainpage_files/files/SLaTE07_Chevalier_Saybot.pdf.
16. Cheville J. Automated scoring technologies and the rising influence of error. English Journal 2004, 93:47-52.
17. Clauser B.E., Kane M.T., Swanson D.B. Validity issues for performance-based tests scored with computer-automated scoring systems. Applied Measurement in Education 2002, 15:413-432.
18. Cohen J. Statistical power analysis for the behavioral sciences 1988, Erlbaum, Hillsdale, NJ. 2nd ed.
19. Conference on College Composition and Communication (2004). CCCC position statement on teaching, learning, and assessing writing in digital environments. Retrieved from http://www.ncte.org/cccc/resources/positions/123773.htm.
20. Cumming A., Kantor R., Powers D.E. Decision-making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal 2002, 86(1):67-96.
21. Deane P. On the relation between automated essay scoring and modern views of the writing construct. Assessing Writing 2013, 18:7-24. 10.1016/j.asw.2012.10.002.
22. DeVore R. Considerations in the development of accounting simulations. Paper presented at the meeting of the National Council on Measurement in Education 2002, April.
23. Dikli S. An overview of automated scoring of essays. Journal of Technology Learning and Assessment 2006, 5(1). Retrieved from http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1640/1489.
24. Elliot N. On a scale: A social history of writing assessment in America 2005, Peter Lang, New York, NY.
25. Fleiss J.L., Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement 1973, 33:613-619.
26. Franco H., Abrash V., Precoda K., Bratt H., Rao R., Butzberger J., et al. The SRI EduSpeak™ system: Recognition and pronunciation scoring for language learning. Proceedings of InSTILL (Integrating Speech Technology in Language Learning) 2000, 123-128. University of Abertay Dundee, Scotland.
27. Hales L.W., Tokar E. The effect of the quality of preceding responses on the grades assigned to subsequent responses to an essay question. Journal of Educational Measurement 1975, 12:115-117.
28. Hamp-Lyons L. Worrying about rating. Assessing Writing 2007, 12:1-9.
29. Herrington A., Moran C. What happens when machines read our students' writing?. College English 2001, 63:480-499.
30. Hughes D.C., Keeling B. The use of model essays to reduce context effects in essay scoring. Journal of Educational Measurement 1984, 21:277-281.
31. Huot B. (Re) Articulating writing assessment in teaching and learning 2002, Utah State University Press, Logan, UT.
32. Huot B., Neal M. Writing assessment: A techno-history. Handbook of writing research 2006, 417-432. Guilford Press, New York, NY. C.A. MacArthur, S. Graham, J. Fitzgerald (Eds.).
33. Landauer T.K., Laham D., Foltz P.W. Automated scoring and annotation of essays with the Intelligent Essay Assessor. Automated essay scoring: A cross-disciplinary perspective 2003, 87-112. Erlbaum, Hillsdale, NJ. M.D. Shermis, J.C. Burstein (Eds.).
34. Leacock C., Chodorow M. C-rater: Scoring of short-answer questions. Computers and the Humanities 2003, 37:389-405.
35. Leckie G., Baird J. Rater effects on essay scoring: A multilevel analysis of severity drift, central tendency, and rater experience. Journal of Educational Measurement 2011, 48:399-418.
36. Lunz M.E., Wright B.D., Linacre J.M. Measuring the impact of judge severity on examination scores. Applied Measurement in Education 1990, 3:331-345.
37. Margolis M.J., Clauser B.E. A regression-based procedure for automated scoring of a complex medical performance assessment. Automated scoring of complex tasks in computer based testing 2006, 123-167. Erlbaum, Hillsdale, NJ. D. Williamson, R. Mislevy, I. Bejar (Eds.).
38. Mitchell T., Russell T., Broomhead P., Aldridge N. Towards robust computerized marking of free-text responses. Proceedings of the Sixth International Computer Assisted Assessment Conference 2002, 233-249. Loughborough University, Loughborough, UK.
39. National Governors Association Center for Best Practices & Council of Chief State School Officers Common Core State Standards 2010, Authors, Washington, DC.
40. Page E.B. The imminence of grading essays by computer. Phi Delta Kappan 1966, 48:238-243.
41. Page E.B. The use of the computer in analyzing student essays. International Review of Education 1968, 14:210-225.
42. Page E.B. Project essay grade: PEG. Automated essay scoring: A cross-disciplinary perspective 2003, 43-54. Erlbaum, Hillsdale, NJ. M.D. Shermis, J.C. Burstein (Eds.).
43. Pearson (2009, March). PTE academic automated scoring. Retrieved from http://www.pearsonpte.com/SiteCollectionDocuments/AutomatedScoringUS.pdf.
44. Powers D.E., Burstein J.C., Chodorow M.S., Fowles M.E., Kukich K. Comparing the validity of automated and human scoring of essays. Educational Computing Research 2002, 26:407-425.
45. Powers D.E., Burstein J.C., Chodorow M.S., Fowles M.E., Kukich K. Stumping e-rater: Challenging the validity of automated essay scoring. Computers in Human Behavior 2002, 18:103-134.
46. ® scoring engine (Research Report 09-01) 2009, Educational Testing Service, Princeton, NJ.
47. Ramineni C. Validating automated essay scoring for online writing placement. Assessing Writing 2013, 18:40-61. 10.1016/j.asw.2012.10.005.
48. ® for GRE Issue and Argument prompts (Research Report 12-02) 2012, Educational Testing Service, Princeton, NJ.
49. ® scoring engine for TOEFL Independent and Integrated prompts (Research Report 12-06) 2012, Educational Testing Service, Princeton, NJ.
50. Risse T. Testing and assessing mathematical skills by a script based system. Paper presented at the 10th International Conference on Interactive Computer aided Learning 2007.
51. Rudner L.M. Reducing errors due to the use of judges. Practical assessment, Research and Evaluation 1992, 3(3). Retrieved from http://PAREonline.net/getvn.asp?v=3&n=3.
52. Rudner L.M., Garcia V., Welch C. An evaluation of IntelliMetric™ essay scoring system. The Journal of Technology, Learning and Assessment 2006, 4(4). Retrieved from http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1651/1493.
53. Saal F.E., Downey R.G., Lahey M.A. Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin 1980, 88:413-428.
54. Sargeant J., Wood M.M., Anderson S.M. A human-computer collaborative approach to the marking of free text answers. Proceedings of the 8th International CAA Conference 2004, 361-370. Loughborough University, Loughborough, UK.
55. Shermis M.D., Burstein J.C. Automated essay scoring: A cross-disciplinary perspective 2003, Erlbaum, Hillsdale, NJ.
56. Shermis M.D., Burstein J.C., Leacock C. Applications of computers in assessment and analysis of writing. Handbook of writing research 2006, 403-416. Guilford Press, New York, NY. C.A. McArthur, S. Graham, J. Fitzgerald (Eds.).
57. Shermis M.D., Shneyderman A., Attali Y. How important is content in the ratings of essay assessments?. Assessment in Education: Principles, Policy and Practice 2008, 15:91-105.
58. Singley M.K., Bennett R.E. Validation and extension of the mathematical expression response type: Applications of schema theory to automatic scoring and item generation in mathematics (GRE Board Professional Report 93-24P) 1998, Educational Testing Service, Princeton, NJ.
59. Spandel V. Creating writers through 6-trait Writing Assessment and Instruction 2005, Pearson, Allyn & Bacon, Boston, MA. 4th ed.
60. Stalnaker J.M. The problem of the English examination. Educational Record 1936, 17:35-48.
61. Sukkarieh J.Z., Pulman S.G. Information extraction and machine learning: Auto-marking short free text responses to science questions. Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED) 2005, 629-637.
62. Valenti S., Neri F., Cucchiarelli A. An overview of current research on automated essay grading. Journal of Information Technology Education 2003, 2:319-330.
63. Vaughan C. Holistic assessment: What goes on in the rater's mind?. Assessing second language writing in academic contexts 1991, 11-125. Ablex, Norwood, NJ. L. Hamp-Lyons (Ed.).
64. Warschauer M., Ware P. Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research 2006, 10:157-180.
65. Weigle S.C. Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing 1999, 6:145-178.
66. White E.M. Teaching and assessing writing: Recent advances in understanding, evaluating, and improving student performance 1994, Jossey-Bass, San Francisco, CA.
67. Williamson D.M., Bejar I.I., Hone A.S. 'Mental model' comparison of automated and human scoring. Journal of Educational Measurement 1999, 36:158-184.
68. Williamson D.M., Bennett R.E., Lazer S., Bernstein J., Foltz P.W., Landauer T.K., et al. Automated Scoring for the Assessment of Common Core Standards 2010 July, Retrieved from http://www.ets.org/s/commonassessments/pdf/AutomatedScoringAssessCommonCoreStandards.pdf.
69. Williamson D.M., Xi X., Breyer F.J. A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practices 2012, 31(1):2-13.
70. Wiseman C. Rater effects: Ego engagement in rater decision-making. Assessing Writing 2012, 17:150-173.
71. Wolfe E., Kao C.-W., Ranney M. Cognitive differences in proficient and non- proficient essay scorers. Written Communication 1998, 15(4):469-482.
72. Xi X., Higgins D., Zechner K., Williamson D.M. Automated scoring of spontaneous speech using SpeechRater v1.0 (Research Report 08-62) 2008, Educational Testing Service, Princeton, NJ.
73. Yang Y., Buckendahl C.W., Juszkiewicz P.J., Bhola D.S. A review of strategies for validating computer-automated scoring. Applied Measurement in Education 2002, 15:391-412.
74. Zechner K., Bejar I.I. Towards automatic scoring of non-native spontaneous speech. Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL 2006, 216-223.