Probable cause: Developing warrants for automated scoring of essays

Handbook of Automated Essay Evaluation: Current Applications and New Directions

Volumn , Issue , 2013, Pages 153-180

  Williamson, David M ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATION; BEHAVIORAL RESEARCH; TESTING;

CENTRAL TENDENCIES; HALO EFFECT; PERFORMANCE FEEDBACK; PERSONAL ASSISTANTS; POTENTIAL BENEFITS; STANDARDIZED TESTING;

COMPUTER GAMES;

EID: 84924804105     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203122761     Document Type: Chapter

References (32)

1. Attali, Y. (2011). Sequential effects in essay ratings. Educational and Psychological Measurement, 71, 68–79
2. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater V.2. Journal of Technology, Learning, and Assessment, 4(3). Available from http://www.jtla.org
3. Bridgeman, B., Trapani, C., & Attali, Y. (2012). Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education, 25, 27–40
4. Burstein, J., & Marcu, D. (2003). A machine learning approach for identification of thesis and conclusion statements in student essays. Computers and the Humanities, 37(4), 455–467
5. Clauser, B. E., Kane, M. T., & Swanson, D. B. (2002). Validity issues for performance-based tests scored with computer-automated scoring systems. Applied Measurement in Education, 15, 413–432
6. Clauser, B. E., Margolis, M. J., Clyman, S. G., & Ross, L. P. (1997). Development of automated scoring algorithms for complex performance assessments: A comparison of two approaches. Journal of Educational Measurement, 34(2), 141–161
7. Culham, R. (2003). Six + 1 traits of writing. New York, NY: Scholastic Professional Books
8. Deane, P., Fowles, M., & Sabatini, J. (2012). Rethinking K-12 writing assessment to support best practices. In C. Bazerman & S. Null (Eds.), New directions in international writing research. Anderson, SC: Parlor Press
9. DiPardo, A., Storms, B. A., & Selland, M. (2011). Seeing voices: Assessing writerly stance in the NWP analytic writing continuum. Assessing Writing, 16, 170–188
10. ETS. (2010), from http://www.ets.org/toefl/news/toefl_testtakers_exceed_25_mil_globally
11. Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A. A., & Welty, C. (2010). Building Watson: An overview of the DeepQA Project. Al Magazine, 31, 59–79
12. Johnson, R. L., Penny, J. A., & Gordon, B. (2009). Assessing performance: Designing, scoring, and, validating performance tasks. New York, NY: Guilford
13. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed). (pp. 17–64). Westport, CT: American Council on Education/Praeger
14. Leacock, C., & Chodorow, M. (2003). Automated grammatical error detection. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 195–208). Mahwah, NJ: Lawrence Erlbaum Associates, Inc
15. Leckie, G., & Baird, J. (2011). Rater effects on essay scoring: A multilevel analysis of severity drift, central tendency, and rater experience. Journal of Educational Measurement, 48, 399–418
16. Mislevy, R. J. (1996). Test theory reconceived. Journal of Educational Measurement, 33, 379–416
17. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3–67
18. Myford, C. M., & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4, 386–422
19. National Governors Association Center for Best Practices & Council of Chief State School Officers. (2010). Common core state standards. National Governors Association Center for Best Practices & Council of Chief State School Officers. Washington, DC. Available from http://www.corestandards.org/the-standards
20. Page, E. B. (1968). The use of the computer in analyzing student essays. International Review of Education, 14, 210–225
21. Quillen, I. (2012). Hewlett Automated-Essay-Grader Winners Announced, May 9. Retrieved from http://blogs.edweek.org/edweek/DigitalEducation/2012/05/essay_grader_winners_announced.html
22. Ramineni, C. (2013). Validating automated essay scoring for online writing placement. Assessing Writing, 18, 40–61
23. Ramineni, C., Trapani, C. S., Williamson, D. M., Davey, T., & Bridgeman, B. (2012a). Evaluation of e-rater® for the GRE® issue and argument prompts (ETS RR-12-02). Princeton, NJ: Educational Testing Service
24. Ramineni, C., Trapani, C. S., Williamson, D. M., Davey, T., & Bridgeman, B. (2012b). Evaluation of e-rater® for the TOEFL® independent and integrated prompts (ETS RR-12-06). Princeton, NJ: Educational Testing Service
25. Rudner, L. M. (1992). Reducing errors due to the use of judges. Practical Assessment, Research & Evaluation, 3(3)
26. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413–428
27. Shermis, M. D., & Hamner, B. (2013). Contrasting state-of-the-art automated scoring of essays. In Shermis & Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new directions (pp. 313–346). New York, NY: Routledge
28. Toulmin, S. E. (1958). The uses of argument. Cambridge, UK: Cambridge University Press
29. Trapani, C., Bridgeman, B., & Breyer, J. (2011). Using automated scoring as a trend score: The implications of score separation over time. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA
30. Williamson, D. M. Xi, X., & Breyer, J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31, 2–13
31. Wolfe, E. W. (2004). Identifying rater effects using latent trait models. Psychological Science, 46, 35–51
32. Yang, Y., Buckendahl, C., Juszkiewicz, P., & Bhola, D. (2002). A review of strategies for validating computer-automated scoring. Applied Measurement in Education, 15, 391–412