Monitoring of Scoring Using the e-rater® Automated Scoring System and Human Raters on a Writing Test

ETS Research Report Series

Volumn 2014, Issue 1, 2014, Pages 1-21

  Wang, Zhen ^a   von Davier, Alina A ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

automated scoring; Constructed response items; human scoring; quality control

Indexed keywords

EID: 84964439576     PISSN: None     EISSN: 23308516     Source Type: Journal    
DOI: 10.1002/ets2.12005     Document Type: Article

References (41)

1. American Society for Testing and Materials. (1976). ASTM manual on presentation of data and control chart analysis (Pub. No. STP15D). Philadelphia, PA: Author.
2. Attali, Y. (2007). Construct validity of e-rater in scoring TOEFL essays (Research Report No. RR–07–21). Princeton, NJ: Educational Testing Service.
3. Attali, Y. (2008). e-rater evaluation for TOEFL iBT independent essays. Unpublished manuscript.
4. Attali, Y., Bridgeman, B., & Trapani, C. (2010). Performance of a generic approach in automated scoring. Journal of Technology, Learning, and Assessment, 10(3), 1–16.
5. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater V.2. Journal of Technology, Learning, and Assessment, 4(3), 1–31.
6. Bejar, I. I. (2011). A validity-based approach to quality control and assurance of automated scoring. Assessment in Education: Principles, Policy and Practice, 18(3), 319–341.
7. Burstein, J., & Chodorow, M. (1999). Automated essay scoring for nonnative English speakers. In M. Broman Olsen (Ed.), Computer mediated language assessment and evaluation in natural language processing (pp. 68–75). Morristown, NJ: Association for Computational Linguistics.
8. Chodorow, M., & Burstein, J. (2004). Beyond essay length: Evaluating e–rater's performance on TOEFL essays (Research Report No. RR–04–04). Princeton, NJ: Educational Testing Service.
9. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Mahwah, NJ: Erlbaum.
10. Davey, T. (2009, April). Principles for building and evaluating e–rater models. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.
11. DeCarlo, L. T. (2010). Studies of a latent class signal detection model for constructed-response scoring II: Incomplete and hierarchical designs (Research Report No. RR-10-08). Princeton, NJ: Educational Testing Service.
12. Donoghue, J. R., McClellan, C. A., & Gladkova, L. (2006). Using rater effects models in NAEP. Unpublished manuscript.
13. Engelhard, G., Jr. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93–112.
14. Engelhard, G., Jr. (2002). Monitoring raters in performance assessments. In G. Tindal & T. M. Haladyna (Eds.), Large-scale assessment programs for all examinees: Validity, technical adequacy, and implementation (pp. 261–287). Mahwah, NJ: Erlbaum.
15. Gao, R. (2009). Detect cheating using statistical control methods for computer based CLEP examinations with item exposure risks. Unpublished manuscript.
16. Haberman, S. (2011). Use of e-rater in scoring of the TOEFL iBT writing test (Research Report No. RR–11–25). Princeton, NJ: Educational Testing Service.
17. Haberman, S. (2012). Measure of agreement. Unpublished manuscript.
18. Haberman, S., & Sinharay, S. (2008). Sample-size requirements for automated essay scoring (Research Report No. RR–08–32). Princeton, NJ: Educational Testing Service.
19. Kingsbury, F. A. (1922). Analyzing ratings and training raters. Journal of Personnel Research, 1, 377–383.
20. Lane, S., & Stone, C. A. (2006). Performance assessment. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 387–431). Westport, CT: Praeger.
21. Lee, Y.-H., & von Davier, A. A. (2012). Using data mining and quality control techniques to monitor scaled scores. Manuscript submitted for publication.
22. Longford, N. (1995). Models for uncertainty in educational testing. New York, NY: Springer-Verlag.
23. Luecht, R. M. (2010, April). Some small sample statistical quality control procedures for constructed response scoring in language testing. Paper presented at the annual meeting of the National Council on Measurement in Education, Denver, CO.
24. Montgomery, D. C. (1997). Introduction to statistical process control (3rd ed.). New York, NY: Wiley.
25. Myford, C. M., & Wolfe, E. W. (2009). Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale category use. Journal of Educational Measurement, 46, 371–389.
26. National Institute of Standards and Technology. (n.d.). NIST/SEMATECH e–handbook of statistical methods. Retrieved from http://www.itl.nist.gov/div898/handbook/pmc/section3/pmc32.htm
27. Omar, M. H. (2010). Statistical process control charts for measuring and monitoring temporal consistency of ratings. Journal of Educational Measurement, 47(1), 18–35.
28. Patz, R. J., Junker, B. W., Johnson, M. J., & Mariano, L. T. (2002). The hierarchical rater model for rated test items and its application to large-scale educational assessment data. Journal of Educational and Behavioral Statistics, 27, 341–384.
29. Ramineni, C., Trapani, C., Williamson, D. M., Davey, T., & Bridgeman, B. (2012). Evaluation of e-rater for the GRE issue and argument prompts (Research Report No. RR–12–06). Princeton, NJ: Educational Testing Service.
30. Ramineni, C., Williamson, D., & Weng, V. (2011, April). Understanding mean score differences between e–rater and humans for demographic–based groups in GRE. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
31. Shewhart, W. A. (1931). Quality control chart. Bell System Technical Journal, 5, 593–603.
32. Walker, M. (2005). Proposed rater statistics for TOEFL iBT constructed response. Unpublished manuscript.
33. Wang, Z. (2010). TOEFL Writing Prompt 2 (independent prompt) health check report: Human rater and e-rater. Unpublished manuscript.
34. Wang, Z., & von Davier, A. A. (2010). Proposed procedures to monitor the performance of the human & electronic ratings for all programs. Unpublished manuscript.
35. Wang, Z., & Yao, L. (2011, April). The effects of scoring designs and rater severity on students' ability estimation for constructed response items. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
36. Wang, Z., & Yao, L. (2012, April). Investigation of the effects of scoring designs and rater severity on students' ability estimation using different rater models. Paper presented at the annual meeting of the National Council on Measurement in Education, Vancouver, BC.
37. Way, W. D., Vickers, D., & Nichols, P. (2008, March). Effects of different training and scoring approaches on human constructed response scoring. Paper presented at the annual meeting of the National Council on Measurement in Education, New York, NY.
38. Western Electronic Company. (1958). Statistical quality control (2nd ed.). New York, NY: Author.
39. Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13.
40. Wilson, M., & Hoskens, M. (2001). The rater bundle model. Journal of Educational and Behavioral Statistics, 26, 283–306.
41. Wolfe, E. W., & Myford, C. M. (1997, April). Detecting order effects with a multi–faceted Rasch scale model. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.