Human ratings and automated essay evaluation

Handbook of Automated Essay Evaluation: Current Applications and New Directions

Volumn , Issue , 2013, Pages 221-232

  Bridgeman, Brent ^a  

^a UNIVERSITY OF WISCONSIN MADISON   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATION;

AUTOMATED SYSTEMS; BURSTEIN; TWO-POINT;

QUALITY CONTROL;

EID: 84896511991     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203122761     Document Type: Chapter

References (28)

1. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater V.2. Journal of Technology, Learning, and Assessment, 4(3). Available from http://www.jtla.org
2. Bennett, R., & Bejar, I. (1997). Validity and automated scoring: It’s not only the scoring. (ETS RR-97–13). Princeton, NJ: Educational Testing Service
3. Braun, H. I. (1988). Understanding score reliability: Experiments in calibrating essay readers. Journal of Educational Statistics, 13, 1–18
4. Breland, H. M., Bridgeman, B., & Fowles, M. E. (1999). Writing assessment in admission to higher education: Review and framework. College Board Report No. 99–3; GRE Board Research Report No. 96-12R; ETS RR 99–3. New York, NY: College Entrance Examination Board
5. Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer-Verlag
6. Brenner, H., & Kleissch, U. (1996). Dependence of weighted kappa coefficients on the number of categories. Epidemiology, 7, 199–202
7. Bridgeman, B., Trapani, C., & Attali, Y. (2012). Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education, 25, 27–40
8. Bridgeman, B., Trapani, C., & Williamson, D. (2011). The question of validity of automated essay scores and differentially valued evidence. Paper presented at the National Council on Measurement in Education, New Orleans, LA
9. Cohen, J. A. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement of partial credit. Psychological Bulletin, 70, 213–220
10. Congdon, P. J., & McQueen, J. (2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37, 163–178
11. Cronbach, L. J., Nageswari, R., & Gleser, G. C. (1963). Theory of generalizability: A liberation of reliability theory. The British Journal of Statistical Psychology, 16(2), 137–163
12. DeCarlo, J. T., Kim, Y. K., & Johnson, M. S. (2011). A hierarchical rater model for constructed responses, with a signal detection rater model. Journal of Educational Measurement, 48, 333–356
13. Dikli, S. (2006). An overview of automated scoring of essays. The Journal of Technology, Learning, and Assessment, 5(1), http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1640/1489
14. Engelhard, G., Jr. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31(2), 93–112
15. GRE analytical writing scoring guide (2011), from http://www.ets.org/gre/revised_general/prepare/analytical_writing/issue/scoring_guide
16. Leckie, G., & Baird, J. (2011). Rater effects on essay scoring: A multilevel analysis of severity drift, central tendency, and rater experience. Journal of Educational Measurement, 48, 399–418
17. Lee, Y. W., & Kantor, R. (2005). Dependability of new ESL writing tests scores: Evaluating prototype tasks and alternative rating schemes ETS-RR-05-14; TOEFL-MS-31. Princeton, NJ: Educational Testing Service
18. Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19, 246–276
19. Monaghan, W., & Bridgeman, B. (2005). E-rater as a quality control on human scores. (R&D Connections RDC-02). Princeton, NJ: Educational Testing Service
20. Murakim, E. (1992). A generalized Partial Credit Model: Applications of an EM algorithm. Applied Psychological Measurement, 16, 159–176
21. Myford, C. M., & Wolfe, E. W. (2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of Applied Measurement, 5, 189–227
22. Myford, C. M., & Wolfe, E. W. (2009). Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale category use. Journal of Educational Measurement, 46, 371–389
23. Sawaki, Y., Stricker, L., & Oranje, A. (2009). Factor structure of the TOEFL Internet-based Test (TOEFL iBT). Language Testing, 26, 5–30
24. Schuster, C. (2004). A note on the interpretation of weighted kappa and its relations to other rater agreement statistics for metric scales. Educational and Psychological Measurement, 64, 243–253
25. Shermis, M. D., & Burstein, J. (2003). Introduction. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective. Mahwah, NJ: Lawrence Erlbaum Associates, Inc
26. Suto, I. (2011). A critical review of some research used to explore rater cognition. Paper presented at the annual conference of the American Educational Research Association New Orleans
27. Wolfe, E. W. (2004). Identifying rater effects using latent trait models. Psychological Science, 46, 35–51
28. Wolfe, E. W. (2005). Uncovering rater’s cognitive processing and focus using think-aloud protocols. Journal of Writing Assessment, 2, 37–56