Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale category use

Journal of Educational Measurement

Volumn 46, Issue 4, 2009, Pages 371-389

  Myford, Carol M ^a   Wolfe, Edward W ^b  

^a UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

^b PEARSON   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 71549124344     PISSN: 00220655     EISSN: 17453984     Source Type: Journal    
DOI: 10.1111/j.1745-3984.2009.00088.x     Document Type: Article

References (22)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC : American Educational Research Association.
2. Braun, H.I. (1988). Understanding score reliability: Experiments in calibrating essay readers. Journal of Educational Statistics, 13, 1 18.
3. College Entrance Examination Board (2002a). AP English Literature and Composition 2002 free-response questions. Princeton, NJ : Educational Testing Service. Retrieved July 23, 2008, from http://apcentral.collegeboard.com/apc/ public/repository/eng-02-11390.pdf
4. College Entrance Examination Board (2002b). AP English Literature & Composition 2002 scoring guidelines. Princeton, NJ : Educational Testing Service. Retrieved July 23, 2008, from http://apcentral.collegeboard.com/apc/ public/repository/sg-english
5. College Entrance Examination Board (n.d.). Exam scoring. In AP research technical manual. Retrieved July 23, 2008, from http://apcentral.collegeboard. com/apc/public/courses/1994.html
6. Engelhard, G. Jr. 1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93 112.
7. Engelhard, G. Jr., & Myford, C. M. (2003). Monitoring faculty consultant performance in the Advanced Placement English Literature and Composition program with a many-faceted Rasch model (College Board Research Report No. 2003-1). New York : College Entrance Examination Board.
8. Hoskens, M., & Wilson, M. (2001). Real-time feedback on rater drift in constructed response items: An example from the Golden State Examination. Journal of Educational Measurement, 38, 121 146. (Pubitemid 33319335)
9. Linacre, J. M. (1989). Many-facet Rasch measurement. Chicago : MESA Press.
10. Linacre, J. M. (2003). Facets - Rasch measurement computer program (Version 3.47) [Computer software]. Chicago : MESA Press.
11. Linacre, J. M. (2008). A user's guide to Facets Rasch-model computer programs Computer software manual Retrieved May 28, 2008, from http://www.winsteps.com/aftp/facets.pdf
12. Myford, C. M., & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4, 386 422.
13. Myford, C. M., & Wolfe, E. W. (2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of Applied Measurement, 5, 189 227.
14. Myford, C. M., & Wolfe, E. W. (2008, April). Detecting differential rater severity/leniency in the Advanced Placement English Literature and Composition examination using benchmark essays. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.
15. Wolfe, E. W. (1998, April). Criterion-referenced rater monitoring through optimal appropriateness measurement. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
16. Wolfe, E. W. (2004). Identifying rater effects using latent trait models. Psychology Science, 46, 35 51.
17. Wolfe, E. W. (2005). Identifying rater effects in performance ratings. In S. Reddy (Ed. Performance appraisals: A critical view (pp. 91 103). Hyderabad, India : ICFAI University Press.
18. Wolfe, E. W., Chiu, C. W. T., & Myford, C. M. (2000). Detecting rater effects with a multi-faceted Rasch rating scale model. In M. Wilson & G. Engelhard Jr. Eds. Objective measurement: Theory into practice (Vol. 5, pp. 147 164). Stamford, CT : Ablex Publishing.
19. Wolfe, E. W., & Dobria, L. (2008). Applications of the multifaceted Rasch model. In J. W. Osborne (Ed. Best practices in quantitative methods (pp. 71 85). Thousand Oaks, CA : Sage.
20. Wolfe, E. W., & Moulder, B. C. (1999, April). Examining differential reader functioning over time in rating data: An application of the multi-faceted Rasch rating scale model. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada.
21. Wolfe, E. W., Moulder, B. C., & Myford, C. M. (2001). Detecting differential rater functioning over time (DRIFT) using a Rasch multi-faceted rating scale model. Journal of Applied Measurement, 2, 256 280.
22. Wolfe, E. W., Myford, C. M., Engelhard, G. E., & Manalo, J. R. (2007). Monitoring reader performance and DRIFT in the AP English Literature and Composition examination using benchmark essays (College Board Research Report No. 2007-2). New York : The College Board.