The effect of rating augmentation on inter-rater reliability: An empirical study of a holistic rubric

Assessing Writing

Volumn 7, Issue 2, 2000, Pages 143-164

  Penny, Jim ^a   Johnson, Robert L ^b   Gordon, Belita ^c  

^a CENTER FOR CREATIVE LEADERSHIP   (United States)

^b UNIVERSITY OF SOUTH CAROLINA   (United States)

^c NONE

Author keywords

Holistic rubric; Inter rater reliability; Rating augmentation

Indexed keywords

EID: 0141603477     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/S1075-2935(00)00012-X     Document Type: Article

References (26)

1. Baxter, G., Shavelson, R., Goldman, S., & Pine, J. (1992). Evaluation of procedure-based scoring for hands-on science assessment. Journal of Educational Measurement, 29 (1), 1-17.
2. Breland, H. (1983). The direct assessment of writing skill: a measurement review. Technical Report No. 83-6. Princeton, NJ: College Entrance Examination Board.
3. Brennan, R. L., & Kane, M. T. (1977). An index of dependability for mastery tests. Journal of Educational Measurement, 14, 277-289.
4. Cherry, R. D., & Meyer, P. R. (1993). Reliability issues in holistic assessment. In: M. Williamson, & B. Huot (Eds.), Validating holistic scoring for writing assessment: theoretical and empirical foundations (pp. 109-141). Cresskill, NJ: Hampton Press.
5. Coffman, W. (1971). On the reliability of ratings of essay examinations in English. Research in the Teaching of English, 5, 24-36.
6. Cooper, P. (1984). The assessment of writing ability: a review of research. Technical Report No. 84-12. Princeton, NJ: Education Testing Service.
7. Crick, J. E., & Brennan, R. L. (1983). Manual for GENOVA: a generalized analysis of variance system. Iowa City: The American College Testing Program.
8. Cronbach, L., Linn, R., Brennan, R., & Haertel, E. (1998). Generalizability analysis for educational assessments. Los Angeles: Center for the Study of Evaluation, Standards, and Student Testing, University of California at Los Angeles.
9. Diedrich, P. (1974). Measuring growth in English. Urbana, IL: National Council of Teachers of English.
10. Elbow, P., & Yancey, K. B. (1994). On the nature of holistic scoring: an inquiry composed on Email. Assessing Writing, 1 (1), 91-107.
11. Ercikan, K., Schwarz, R. D., Julian, M. W., Burket, G. R., Weber, M. M., & Link, V. (1998). Calibration and scoring of tests with multiple-choice and constructed-response item types. Journal of Educational Measurement, 35, 137-154.
12. Georgia Department of Education. (1993). Georgia high school writing test: assessment and instructional guide. Atlanta, GA: Georgia Department of Education.
13. Godshalk, F., Swineford, F., & Coffman, W. (1966). The measurement of writing ability. Princeton: College Entrance Examination Board.
14. Gosling, G. (1966). Marking English compositions. Victoria: Australian Council for Educational Research.
15. Huot, B. (1990a). Reliability, validity, and holistic scoring: what we know and what we need to know. College Composition and Communication, 41 (2), 201-213.
16. Huot, B. (1990b). The literature of direct writing assessment: major concerns and prevailing trends. Review of Educational Research, 60 (2), 237-263.
17. Huot, B. (1993). The influence of holistic scoring procedures on reading and rating student essays. In: M. Williamson, & B. Huot (Eds.), Validating holistic scoring for writing assessment: theoretical and empirical foundations (pp. 206-236). Cresskill, NJ: Hampton Press.
18. Johnson, R., & Bergman, T. (1996). The development and implementation of a family literacy portfolio system. Paper presented at the Fifth Annual Conference on Family Literacy in Louisville, KY, April.
19. Johnson, R., Penny, J., & Johnson, C. (1998). Score resolution in the rating of performance assessments: practices and issues. Paper presented at the Council of Chief State School Officers National Conference on Large Scale Assessment, Colorado Springs, CO.
20. Longford, N. (1994). A case for adjusting subjectively rated scores in the Advanced Placement tests. Technical Report No. 94-5. Princeton, NJ: Education Testing Service.
21. McColly, W., & Remstad, R. (1965). Composition rating scales for general merit: an experimental evaluation. Journal of Educational Research, 59, 55-56.
22. Meyers, M. (1980). A procedure for writing assessment and holistic scoring. Urbana, IL: National Council of Teachers of English.
23. Pula, J. J., & Huot, B. A. (1993). A model of background influences on holistic raters. In: M. Williamson, & B. Huot (Eds.), Validating holistic scoring for writing assessment: theoretical and empirical foundations (pp. 237-265). Cresskill, NJ: Hampton Press.
24. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory. Newbury Park, CA: Sage.
25. Smith, W. (1993). Assessing the reliability and adequacy of using holistic scoring of essays as a college composition placement technique. In: M. Williamson, & B. Huot (Eds.), Validating holistic scoring for writing assessment: theoretical and empirical foundations (pp. 142-205). Cresskill, NJ: Hampton Press.
26. Wolfe, E. W. (1997). The relationship between essay reading style and scoring proficiency in a psychometric scoring system. Assessing Writing, 4 (1), 83-106.