Automated scoring of short-answer reading items: Implications for constructs

Language Assessment Quarterly

Volumn 7, Issue 3, 2010, Pages 205-218

  Carr, Nathan T ^a   Xi, Xiaoming ^b  

^a CALIFORNIA STATE UNIVERSITY   (United States)

^b EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77955681447     PISSN: 15434303     EISSN: 15434311     Source Type: Journal    
DOI: 10.1080/15434300903443958     Document Type: Article

References (27)

1. Attali, Y., & Burstein, J. (2005). Automated essay scoring with e-rater® v.2.0 (ETS Research Report No. RR-04-45). Princeton, NJ: Educational Testing Service.
2. Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford, UK: Oxford University Press.
3. Bachman, L. F. (2003, March). Validity issues in a web-based language assessment system (WebLAS). Colloquium paper presented at the American Association for Applied Linguistics Conference, Arlington, VA.
4. Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice. Oxford, UK: Oxford University Press.
5. Bennett, R. E., & Bejar, I. I. (1998). Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice, 17, 9-17.
6. Bennett, R. E., Steffen, M., Singley, M. K., Morley, M., & Jacquemin, D. (1997). Evaluating an automatically scorable, open-ended response type for measuring mathematical reasoning in computer-adaptive tests. Journal of Educational Measurement, 34, 162-176.
7. Burstein, J. C., Kukich, K., Wolff, S., Lu, C., & Chodorow, M. (1998, April). Computer analysis of essays. Paper presented at the annual meeting of the National Council of Measurement in Education. San Diego, CA. Retrieved November 6, 2007, from http://www.ets.org/Media/Research/pdf/erater-ncmefinal. pdf
8. Clauser, B. E., Harik, P., & Clyman, S. G. (2000). The generalizability of scores for a performance assessment scored with a computer-automated scoring system. Journal of Educational Measurement 37, 245-261.
9. Clauser, B. E., Kane, M. T., & Swanson, D. B. (2002). Validity issues for performance-based tests scored with computer-automated scoring systems. Applied Measurement in Education, 15, 413-432.
10. Clauser, B. E., Margolis, M. J., Clyman, S. G., & Ross, L. P. (1997). Development of automated scoring algorithms for complex performance assessments: A comparison of two approaches. Journal of Educational Measurement, 34, 141-161.
11. Clauser, B. E., Ross, L. P., Clyman, S. G., Rose, K. M., Margolis, M. J., Nungester, R. J., et al. (1997). Development of a scoring algorithm to replace a complex performance-based assessment. Applied Measurement in Education, 10, 345-358.
12. Clauser, B. E., Subhiyah, R. G., Nungester, R. J., Ripkey, D. R., Clyman, S. G., & McKinley, D. (1995). Scoring a performance-based assessment by modeling the judgments of experts. Journal of Educational Measurement, 32, 397-415.
13. Clauser, B. E., Swanson, D. B., & Clyman, S. G. (2000). A comparison of the generalizability of scores produced by expert raters and automated scoring systems. Applied Measurement in Education, 12, 281-299.
14. Educational Testing Service. (2005). Criterion details: English language learning: Frequently asked questions about Criterion. Retrieved September 22, 2007, from http://www.ets.org/portal/site/ets/menuitem. 1488512ecfd5b8849a77b13bc3921509/?vgnextoid= e9872d3631df4010VgnVCM10000022f95190RCRD&vgnextchannel= 547f253b164f4010VgnVCM10000022f95190RCRD
15. Fitzpatrick, S., & Triscari, R. (2005, April). Comparability studies of the Virginia computer-delivered tests. Paper presented at the annual meeting of the American Educational Research Association. Montreal, Canada. Retrieved April 6, 2008, from http://www.doe.virginia.gov/VDOE/Assessment/Compstudy- AERA2005.pdf
16. Higgins, D., Burstein, J., Marcu, D., & Gentile, C. (2004). Evaluating multiple aspects of coherence in student essays. In Proceedings of the Annual Meeting of HLT/NAACL, Boston, MA. Retrieved October 19, 2007, from http://ftp.ets.org/pub/res/erater-higgins-dis-coh.pdf
17. Hirschman, L., Breck, E., Light, M., Burger, J. D., & Ferro, L. (2000). Automated grading of short-answer tests. IEEE Intelligent Systems, 15(5), 31-35.
18. Leacock, C. (2004). Scoring free-responses automatically: A case study of a large-scale assessment. Examens, 1(3). English version retrieved April 12, 2008, from http://www.ets.org/Media/Research/pdf/erater-examens-leacock.pdf
19. Leacock, C., & Chodorow, M. (2003). C-rater: Automated scoring of short answer questions. Computers and the Humanities, 37, 389-405.
20. Nichols, P. (2005). Evidence for the interpretation and use of scores from an automated essay scorer (Pearson Educational Measurement Research Rep. 0502). Retrieved April 6, 2008, from http://www.pearsonedmeasurement.com/ research/research.htm
21. Powers, D. E., Burstein, J. C., Chodorow, M., Fowles, M.E., & Kukich, K. (2001). Stumping E-Rater: Challenging the validity of automated essay scoring (ETS Research Rep. 01-03). Princeton, NJ: Educational Testing Service.
22. Shaw, S. D. (2004). Automated writing assessment: A review of four conceptual models. Research Notes, 17, 13-18.
23. UCLA Department of Applied Linguistics and TESL & Center for Digital Humanities. (2003). WebLAS (Web-based Language Assessment System). Retrieved August 10, 2005, from http://www.weblas.ucla.edu/
24. Vongpumivitch, V. (2004). Measuring the knowledge of text structure in academic English as a second language (ESL). Unpublished doctoral dissertation, University of California, Los Angeles.
25. Williamson, D. M., Bejar, I. I., & Hone, A. S. (1999). "Mental model" comparison of automated and human scoring. Journal of Educational Measurement, 36, 158-184.
26. Williamson, D. M., Bejar, I. I., & Saxe, A. (2004). Automated tools for subject matter expert evaluation of automated scoring. Applied Measurement in Education, 17, 323-357.
27. Yang, Y., Buckendahl, C. W., Juszkiewicz, P. J., & Bhola, D. S. (2002). A review of strategies for validating computer-automated scoring. Applied Measurement in Education, 15, 391-412.