On the vulnerability of automated scoring to construct-irrelevant response strategies (CIRS): An illustration

Assessing Writing

Volumn 22, Issue , 2014, Pages 48-59

  Bejar, Isaac I ^a   Flor, Michael ^a   Futagi, Yoko ^a   Ramineni, Chaintanya ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

Automated scoring; Essay scoring; Response strategy; Validation

Indexed keywords

EID: 84904889677     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.asw.2014.06.001     Document Type: Article

References (23)

1. American Educational Research Association, American Psychological Association, National Council on Measurement in Education Standards for educational and psychological testing 1999, American Educational Research Organization, Washington, D.C.
2. ® V.2. Journal of Technology, Learning, and Assessment 2006, 4(3). Available from http://www.jtla.org.
3. Beigman Klebanov B., Flor M. Word association profiles and their use for automated scoring of essays. Proceedings of the 51st annual meeting of the association for computational linguistics (Sofia, Bulgaria) 2013, 1148-1158. http://aclweb.org/anthology/P/P13/P%2013-1113.pdf.
4. Ben-Simon A., Bennett R.E. Toward more substantively meaningful automated essay scoring. Journal of Technology, Learning, and Assessment 2007, 6(1). Available from http://www.jtla.org.
5. Bennett R.E., Bejar I.I. Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice 1998, 17(4):9-16.
6. Bejar I.I. A validity-based approach to quality control and assurance of automated scoring. Assessment in Education 2011, 18(3):319-341.
7. Breland H.M. Word frequency and word difficulty: A comparison of counts in four corpora. Psychological Science 1996, 7(2):96-99.
8. Breland H.M., Jones R.J., Jenkins L. The College Board vocabulary study (No. College Board Report No. 94-4; Educational Testing Service Research Report No. 94-26) 1994, College Entrance Examination Board, New York.
9. Elliot N. On a scale: A social history of writing assessment in America 2005, Peter Lang, New York.
10. Embretson S.E. Construct validity: Construct representation versus nomothetic span. Psychological Bulletin 1983, 93:179-197.
11. Fellbaum C. WordNet: An electronic lexical database 1998, MIT Press, Cambridge, MA.
12. Godshalk F.I., Swineford F., Coffman W.E. The measurement of writing ability 1966, College Entrance Examination Board, New York.
13. Hillocks G. The testing trap 2002, Teacher College Press, New York.
14. Huot B. Toward a new theory of writing assessment. College Composition and Communication 1996, 47(4):549-566.
15. Kane M.T. Validation. Educational measurement 2006, 17-64. Praeger Publishers, Westport, CT. 4th ed. R.L. Brennan (Ed.).
16. Koretz D., Hamilton L.S. Testing for accountability in K-12. Educational measurement 2006, 531-578. Praeger, Westport, CT. 4th ed. R. Brennan (Ed.).
17. Page E.B. The imminence of grading essays by computer. Phi Delta Kappan 1966, 47:238-243.
18. Powers D.E., Burstein J., Chodorow M., Fowles M., Kukich K. Comparing the validity of automated and human scoring of essays. Journal of Educational Computing Research 2002, 26(4):407-425.
19. Powers D.E., Burstein J.C., Chodorow M.S., Fowles M.E., Kukick K. Stumping e-rater: Challenging the validity of automated essay scoring. Computers in Human Behavior 2002, 18:103-134.
20. Powers D.E., Fowles M.E., Willard A.E. Direct assessment, direct validation? An example from the assessment of writing. Educational Assessment 1994, 2(1):89-100.
21. ® (RR 09-01) 2009, ETS, Princeton, NJ.
22. Yu G. Lexical diversity in writing and speaking task performances. Applied Linguistics 2010, 31(2):236-259.
23. Zeno S., Ivens S.M., Koslin S.H., Zeno B.L. The educator's word frequency guide 1995, Touchstone Applied Science Associates, Brewster, NY.