An examination of rater drift within a generalizability theory framework

Journal of Educational Measurement

Volumn 46, Issue 1, 2009, Pages 43-58

  Harik, Polina ^a,c   Clauser, Brian E ^a,c   Grabovsky, Irina ^a,c   Nungester, Ronald J ^a,c   Swanson, Dave ^a,c   Nandakumar, Ratna ^b  

^a National Board of Medical Examiners   (United States)

^b UNIVERSITY OF DELAWARE   (United States)

^c NATIONAL BOARD OF MEDICAL EXAMINERS   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 61349170677     PISSN: 00220655     EISSN: 17453984     Source Type: Journal    
DOI: 10.1111/j.1745-3984.2009.01068.x     Document Type: Article

References (23)

1. Bock, R. D. (1995). Open-ended exercises in large-scale educational assessment. In L. B. Resnick & J. G. Wirt (Eds Linking school and work: Roles for standards and assessment (pp. 305 338). San Francisco : Jossey-Bass.
2. Bock, R. D., Muraki, E., & Pfeiffenberger, W. (1988). Item pool maintenance in the presence of item parameter drift. Journal of Educational Measurement, 25, 275 285.
3. Boulet, J. R., Rebbecchi, T., Denton, E. C., McKinley, D. W., & Whalen, G. P. (2007). Assessing the written communication skills of medical school graduates. Advances in Health Science Education, 9, 47 60.
4. Braun, H. I. (1988). Understanding scoring reliability: Experiments in calibrating essay readers. Journal of Educational Statistics, 13, 1 18.
5. Braun, H. I., & Wainer, H. (1989). Making essay test scores fairer with statistics. In J. Tanur, F. Mosteller, W. H. Kruskal, E. L. Lehmann, R. F. Link, R. S. Pieters & G. S. Rising (Eds Statistics: A guide to the unknown (3rd ed., pp. 178 188). Pacific Grove, CA : Wadsworth.
6. Brennan, R. L. (2001). Generalizability theory. New York : Springer-Verlag.
7. Clauser, B. E., Harik, P., & Margolis, M. J. (2006). A multivariate generalizability analysis of data from a performance assessment of physicians' clinical skills. Journal of Educational Measurement, 43, 173 191.
8. Congdon, P. J., & McQueen, J. (2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37, 163 178.
9. De Gruijter, D. N. M. (1984). Two simple models for rater effects. Applied Psychological Measurement, 8, 213 218.
10. Engelhard, G., Jr. 1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93 112.
11. Engelhard, G., Jr. 1996). Evaluating rater accuracy in performance assessments. Journal of Educational Measurement, 33, 56 70.
12. Fleiss, J. L. (1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement, 5, 105 112.
13. Guilford, J. P. (1936). Psychometric methods. New York : McGraw-Hill.
14. Hartog, P., & Rhodes, E. C. (1936). The marks of examiners. London : Macmillan.
15. Hubbard, J. P., & Levit, E. J. (1985) The National Board of Medical Examiners: The first seventy years. Philadelphia : National Board of Medical Examiners.
16. Linacre, J. M., & Wright, B. D. (2002). Understanding Rasch measurement: Construction of measures from many-facet data. Journal of Applied Measurement, 3, 486 512.
17. Longford, N. T. (1994). Reliability of essay rating and score adjustment. Journal of Educational and Behavioral Statistics, 19, 171 200.
18. Margolis, M. J., Clauser, B. E., Swanson, D. B., & Boulet, J. R. (2003). Analysis of the relationship between score components on a standardized patient clinical skills examination. Academic Medicine, 78, S68 S71.
19. McKinley, D., & Boulet, J. R. (2004). Detecting score drift in a high-stakes performance-based assessment. Advances in Health Sciences Education, 9, 29 38.
20. Raymond, M. R., & Viswesvaran, C. (1993). Least squares models to correct for rater effects in performance assessment. Journal of Educational Measurement, 30, 253 268.
21. Stanley, J. C. (1962). Analysis-of-variance principles applied to the grading of essay tests. Journal of Experimental Education, 30, 279 283.
22. Van Der Linden, W. J., & Hambleton, R. K 1997). Handbook of modern item response theory. New York : Springer-Verlag.
23. Wolfe, E. W., Moulder, B. C., & Myford, C. M. (2001). Detecting differential rater functioning over time (DRIFT) using a Rasch multi-faceted rating scale model. Journal of Applied Measurement, 2, 256 280.