Validity and reliability of automated essay scoring

Handbook of Automated Essay Evaluation: Current Applications and New Directions

Volumn , Issue , 2013, Pages 181-198

  Attali, Yigal ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATED ESSAY SCORING; ERICSSON; NEW YORK TIME; VIRGINIA;

AUTOMATION;

EID: 84930388917     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203122761     Document Type: Chapter

References (59)

1. AERA, APA, and NCME (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association
2. Almond, R., Quinlan, T. H., & Attali, Y. (2011). Using timing logs to diagnose problems in writing performance. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA
3. Anson, C. M. (2006). Can’t touch this: Reflections on the servitude of computers as readers. In P. F. Ericsson & R. H. Haswell (Eds.), Machine scoring of student essays: Truth and consequences (pp. 211–220). Logan, UT: Utah State University Press
4. Attali, Y. (2007). Construct validity of e-rater in scoring TOEFL essays (ETS RR-07–21). Princeton, NJ: Educational Testing Service
5. Attali, Y. (2009). Evaluating automated scoring for operational use in consequential language assessment. Paper presented at the National Council on Measurement in Education, San Diego, CA
6. Attali, Y. (2011a). A differential word use measure for content analysis in automated essay scoring (ETS RR-11–36). Princeton, NJ: Educational Testing Service
7. Attali, Y. (2011b). Automated subscores for TOEFL iBT Independent essays (ETS RR-11–39). Princeton, NJ: Educational Testing Service
8. Attali, Y. (2012a). Factor structure of the e-rater automated essay scoring system. Paper presented at the National Council of Measurement in Education, Vancouver, CA
9. Attali, Y. (2012b). When raters score like a machine: Exploring rating styles among essay raters. Manuscript in preparation
10. Attali, Y., Bridgeman, B., & Trapani, C. S. (2010). Performance of a generic approach in automated essay scoring. Journal of Technology, Learning, and Assessment, 10(3). Available from http://wwwjtla.org
11. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater V.2. Journal of Technology, Learning, and Assessment, 4(3). Available from http://www.jtla.org
12. Attali, Y., & Powers, D. (2009). Validity of scores for a developmental writing scale based on automated scoring. Educational and Psychological Measurement, 69, 978–993
13. Attali, Y., & Powers, D. E. (2008). A developmental writing scale. (ETS RR-08-19. Princeton, NJ: Educational Testing Service
14. Bachman, L. F., Lynch, B. K. & Mason, M. (1995). Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing, 12, 238–257
15. Barrett, S. (2001). The impact of training on rater variability. International Education Journal, 2, 49–58
16. Ben-Simon, A., & Bennett, R. E. (2007). Toward more substantively meaningful automated essay scoring. Journal of Technology, Learning, and Assessment, 6(1). Available from http://www.jtla.org
17. Bennett, R. E. (2006). Moving the field forward: Some thoughts on validity and automated scoring. In D. M. Williamson, R. J. Mislevy, & I. I. Bejar (Eds.), Automated scoring of complex tasks in computer-based testing (pp. 403–412). Mahwah, NJ: Erlbaum
18. Bennett, R. E., & Bejar, I. I. (1998). Validity and automated scoring: It’s not only the scoring. Educational Measurement: Issues and Practice, 17, 9–17
19. Biber, D. (1988). Variation across speech and writing. Cambridge, England: Cambridge University Press
20. Brennan, R. L. (2001). Generalizability Theory. New York, NY: Springer-Verlag
21. Bridgeman, B., Trapani, C., & Attali, Y. (2012). Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education, 25, 27–40
22. Burstein, J., & Chodorow, M. (1999). Automated essay scoring for nonnative English speakers. Proceedings of the ACL99 Workshop on Computer-Mediated Language Assessment and Evaluation of Natural Language Processing
23. Clauser, B. E., Kane, M. T., & Swanson, D. B. (2002). Validity issues for performance-based tests scored with computer-automated scoring systems. Applied Measurement in Education, 15, 413–432
24. Communication, Conference on College Composition and. (2004). Position statement on teaching, learning, and assessing writing in digital environments. Paper presented at the Conference on College Composition and Communication, http://www.ncte.org/cccc/resources/positions
25. Condon, W. (2006). Why less is not more: What we lose by letting the computer score writing samples. In P. F. Ericsson & R. H. Haswell (Eds.), Machine scoring of student essays: Truth and consequences (pp. 211–220). Logan, UT: Utah State University Press
26. Cronbach, L. J. (1988). Five perspectives on validity argument. In H. Wainer & H. Braun, (Eds.), Test validity (pp. 3–17). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc
27. Deane, P., Quinlan, T., & Kostin, I. (2011). Automated scoring within a developmental, cognitive model of writing proficiency (ETS RR-11-16). Princeton, NJ: Educational Testing Service
28. Eckes, T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25, 155–185
29. Elder, C., Knoch, U., Barkhuizen, G., & von Randow, J. (2005). Individual feedback to enhance rater training: Does it work? Language Assessment Quarterly, 2, 175–196
30. Embretson, S. (1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93, 179–197
31. Engelhard, G., Jr. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93–112
32. Engelhard, G., Jr., & Myford, C. M. (2003). Monitoring faculty consultant performance in the Advanced Placement English Literature and Composition Program with a many-faceted Rasch model. College Board Research Report (No. 2003–1). New York, NY: College Entrance Examination Board
33. Ericsson, P. F. (2006). The meaning of meaning: Is a paragraph more than an equation? In P. F. Ericsson & R. H. Haswell (Eds.), Machine scoring of student essays: Truth and consequences (pp. 28–37). Logan, UT: Utah State University Press
34. Ericsson, P. F., & Haswell, R. H. (Eds.). (2006). Machine scoring of student essays: Truth and consequences. Logan, UT: Utah State University Press
35. Flesch, R. (1974). The art of readable writing. New York, NY: Harper & Row
36. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65–110). Westport, CT: American Council on Education/Praeger
37. Higgins, D., Burstein, J., & Attali, Y. (2006). Identifying off-topic student essays without topicspecific training data. Natural Language Engineering, 12(2), 145–159
38. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: American Council on Education/Praeger
39. Keith, T. Z. (2003). Validity of automated essay scoring systems. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 147–168). Mahwah, NJ: Lawrence Erlbaum Associates, Inc
40. Landauer, T. K., Laham, R. D., & Foltz, P. W. (2003). Automated scoring and annotation of essays with the Intelligent Essay Assessor. In M. Shermis & J. Bernstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 87–112). Mahwah, NJ: Lawrence Erlbaum Associates, Inc
41. Lumley, T., & McNamara, T. F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12, 54–71
42. Page, E. B. (1966). The imminence of grading essays by computer. Phi Delta Kappan, 48, 238–243
43. Page, E. B., & Petersen, N. S. (1995). The computer moves into essay grading: Updating the ancient test. Phi Delta Kappan, 76, 561–565
44. Petersen, N. S. (1997). Automated scoring of writing essays: Can such scores be valid? Paper presented at the National Council of Measurement in Education, Chicago, IL
45. Petersen, N. S., & Page, E. B. (1997). New developments in Project Essay Grade: Second ETS blind test with GRE essays. Paper presented at the American Educational Research Association, Chicago, IL
46. Powers, D. E. (2005). “Wordiness”: A selective review of its influence, and suggestions for investigating its relevance in tests requiring extended written responses (ETS RM-04–08). Princeton, NJ: Educational Testing Service
47. Powers, D. E. (2011). Scoring the TOEFL independent essay automatically: Reactions of test takers and test score users (ETS RM-11–34). Princeton, NJ: Educational Testing Service
48. Powers, D. E., Burstein, J. C., Chodorow, M., Fowles, M. E., & Kukich, K. (2002a). Comparing the validity of automated and human scoring of essays. Journal of Educational Computing Research, 26, 407–425
49. Powers, D. E., Burstein, J. C., Chodorow, M., Fowles, M. E., & Kukich, K. (2002b). Stumping e-rater: Challenging the validity of automated essay scoring. Computers in Human Behavior, 18(2), 103–134
50. Ramineni, C., Trapani, C. S., Williamson, D. M., Davey, T., & Bridgeman, B. (2012a). Evaluation of e-rater® for the GRE® issue and argument prompts (ETS RR-12–02). Princeton, NJ: Educational Testing Service
51. Ramineni, C., Trapani, C. S., Williamson, D. M., Davey, T., & Bridgeman, B. (2012b). Evaluation of e-rater® for the TOEFL® independent and integrated prompts (ETS RR-12-06). Princeton, NJ: Educational Testing Service
52. Scott, J. (1999). Looking for the tidy mind, alas. The New York Times, January 31
53. Sheehan, K. M., Kostin, I., Futagi, Y., & Flor, M. (2010). Generating automated text complexity classifications that are aligned with targeted text complexity standards (ETS Research Report RR-10–28). Princeton, NJ: Educational Testing Service
54. Shermis, M. D., & Burstein, J. (2003). Introduction. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective. Mahwah, NJ: Lawrence Erlbaum Associates, Inc
55. Stenner, A. J., Horabin, I., Smith, D. R., & Smith, M. (1988). Most comprehension tests do measure reading comprehension: A response to McLean and Goldstein. Phi Delta Kappan, 79, 765–767
56. Weigle, S. C. (1998). Using FACETS to model rater training effects. Language Testing, 15, 263–287
57. Weigle, S. C. (1999). Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing, 6, 145–178
58. Wolf-Quintero, K., Inagaki, S., & Kim, H. Y. (1998). Second language development in writing: Measures of fluency, accuracy & complexity. Honolulu, HI: University of Hawaii Press
59. Yang, Y., Buckendahl, C., Juszkiewicz, P., & Bhola, D. (2002). A review of strategies for validating computer-automated scoring. Applied Measurement in Education, 15, 391–412