STUMPING E-RATER: CHALLENGING THE VALIDITY OF AUTOMATED ESSAY SCORING

ETS Research Report Series

Volumn 2001, Issue 1, 2001, Pages i-44

  Powers, Donald E ^a   Burstein, Jill C ^a   Chodorow, Martin ^a   Fowles, Mary E ^a   Kukich, Karen ^a  

^a NONE

Author keywords

automated scoring; e rater; essay scoring; graduate record examinations (GRE); validity; writing assessment

Indexed keywords

EID: 85164826546     PISSN: None     EISSN: 23308516     Source Type: Journal    
DOI: 10.1002/j.2333-8504.2001.tb01845.x     Document Type: Article

References (24)

1. Aschbacher, P. R. (1991). Performance assessment: State activity, interest, and concerns. Applied Measurement in Education, 4, 275–288.
2. Bennett, R. E., & Ward, W. C. (Eds.). (1993). Construction versus choice in cognitive assessment: Issues in constructed response, performance testing, and portfolio assessment. Hillsdale, NJ: Lawrence Erlbaum Associates.
3. Breland, H. M. (1996). Computer-assisted writing assessment: The politics of science versus the humanities. In E. M. White, W. D. Lutz, & S. Kamusikiri (Eds.), Assessment of writing: Politics, policies, practices (pp. 249–256). New York: Modern Language Association of America.
4. Burstein, J., Braden-Harder, L., Chodorow, M., Hua, S., Kaplan, B., Kukich, K., Lu, C., Nolan, J., Rock, D., & Wolff, S. (1998). Computer analysis of essay content for automated score prediction: A prototype automated scoring system for GMAT analytical writing assessment essays (ETS Research Report No. 98-15). Princeton, NJ: Educational Testing Service.
5. Burstein, J., Kukich, K., Wolff, S., Lu, C., & Chodorow, M. (1998, April). Computer analysis of essays. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.
6. DeLoughry, T. J. (1995, October 20). Duke professor pushes concept of grading essays by computer. Chronicle of Higher Education, pp. A24–25.
7. Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York: John Wiley & Sons.
8. Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 527–535.
9. Kaplan, R. M., Wolff, S., Burstein, J., Lu, C., Rock, D., & Kaplan, B. (1998). Scoring essays automatically using surface features (GRE Report No. 94-21P). Princeton, NJ: Educational Testing Service.
10. Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25, 259–284.
11. th annual conference of the Cognitive Science Society (pp. 412–417). Mahwah, NJ: Lawrence Erlbaum Associates.
12. Landis, J. D., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.
13. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13–103). New York: Macmillan.
14. Messick, S. (1995). Validity of psychological assessment. American Psychologist, 50, 741–749.
15. Mitchell, J. S. (1998, May 27). Commentary: SATs don't get you in. Education Week on the Web [On-line serial]. Available: http://www.edweek.org/ew/current/3mitch.h17.
16. Page, E. B. (1966). The imminence of grading essays by computer. Phi Delta Kappan, 48, 238–243.
17. Page, E. B. (1968). Analyzing student essays by computer. International Review of Education, 14, 210–225.
18. Page, E. B., & Petersen, N. S. (1995). The computer moves into essay grading: Updating the ancient test. Phi Delta Kappan, 76, 561–565.
19. Perfetti, C. A. (1998). The limits of co-occurrence: Tools and theories in language research. Discourse Processes, 25, 363–377.
20. Powers, D. E., Fowles, M. E., & Welsh, C. K. (1999). Further validation of a writing assessment for graduate admissions (GRE Board Research Report No. 96-13R and ETS Research Report No. 99-18). Princeton, NJ: Educational Testing Service.
21. Schaeffer, G. A., Fowles, M. A., & Briel, J. B. (in press). Assessment of the psychometric characteristics of the GRE writing test (GRE Board Report No. 96-11). Princeton, NJ: Educational Testing Service.
22. Schwartz, A. E. (1998, April 26). Graded by machine. Washington Post, p. C07.
23. Weinstein, B. (1998, June 21). Software designed to grade essays. Boston Globe Online [On-line serial]. Available: http://www.Boston.com/dailyglobe/g…ware_designed_to_grade_essays.htm/.
24. Wresch, W. (1993). The imminence of grading essays by computer – 25 years later. Computers and Composition, 10, 46–58.