Components of rater error in a complex performance assessment

Journal of Educational Measurement

Volumn 36, Issue 1, 1999, Pages 29-45

  Clauser, Brian E ^a   Clyman, Stephen G ^b   Swanson, David B ^a  

^a NATIONAL BOARD OF MEDICAL EXAMINERS   (United States)

^b National Board of Medical Examiners   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0033245931     PISSN: 00220655     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.1745-3984.1999.tb00544.x     Document Type: Article

References (19)

1. Braun, H. I. 1988). Understanding score reliability: Experiments in calibrating essay readers. Journal of Educational Statistics, 13, 1-18.
2. Brennan, R. L. (1983). Elements of Generalizability Theory. Iowa City, Iowa: ACT Publications.
3. Bridgeman, B., & McHale, F. (1996). Gender and ethnic group differences on the GMAT analytical writing assessment (RR-96-2). Princeton, NJ: Educational Testing Service.
4. Clauser, B. E., Ross, L. P., Clyman, S. G., Rose, K. M., Margolis, M. J., Nungester, R. J., Piemme, T. E., Chang, L., El-Bayoumi, G., Malakoff, G. L., & Pincetl, P. S. (1997). Development of a scoring algorithm to replace expert rating for scoring a complex performance-based assessment. Applied Measurement in Education, 10, 345-358.
5. Clauser, B. E., Subhiyah, R. G., Nungester, R. J., Ripkey, D. R., Clyman, S. G., McKinley, D. (1995). Scoring a performance-based assessment by modelling the judgments of experts. Journal of Educational Measurement, 32, 397-415.
6. Clauser, B. E., Swanson, D. B., &Clyman, S. G. (1996). The generalizability of scores from a performance assessment of physicians' patient management skills. Academic Medicine (RIME Supplement), 71, S109-111.
7. Clyman, S. G., Melnick, D. E., & Clauser, B. E. (1995). Computer-based case simulations. In E. L. Mancall & P. G. Bashook (Eds.), Assessing clinical reasoning: The oral examination and alternative methods (pp. 139-149). Evanston, IL: American Board of Medical Specialities.
8. Crick, J. E., & Brennan, R. L. (1983). Manual for GENOVA: A generalized analysis of variance system. Iowa City, Iowa: ACT Publications.
9. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley & Sons.
10. Cronbach, L. J., Linn, R. L., Brennan, R. L., & Haertel, E. H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373-399.
11. Dunbar, S. B., Koretz, D., & Hoover, H. D. (1991). Quality control in development and use of performance assessments. Applied Measurement in Education, 4, 289-304.
12. Engelhard, G. (1994). Examining rater errors in the assessment of written composition with a many faceted Rasch model. Journal of Educational Measurement, 31, 93-112.
13. Mazzeo, J., Schmitt, A., & Cook, L. (1987). The comparability of adjudicated and non-adjudicated scores on the ATP English Composition Test with Essay. Unpublished manuscript.
14. Linn, R. L. (1994). Performance assessment: Policy promised and technical measurement standards. Educational Researcher, 23, 4-14.
15. Longford, N. T. (1995). Models for uncertainty in Educational Testing. New York: Springer-Verlag.
16. Raymond, M. B., & Viswesvaran, C. ( 1993). Least squares models to correct for rater effects in performance assessment. Journal of Educational Measurement, 30, 253-268.
17. Ruiz-Primo, M. A., Baxter, G. P., & Shavelson, R. J. (1993). On the stability of performance assessments. Journal of Educational Measurement, 30, 41-53.
18. Shavelson, R. J., Baxter, G. P., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30, 215-232.
19. Van der Vleuten, C.P.M., & Swanson, D. B. (1990). Assessment of clinical skills with standardized patients: The state of the art. Teaching and Learning in Medicine, 2, 58-76.