Real-time feedback on rater drift in constructed-response items: An example from the golden state examination

Journal of Educational Measurement

Volumn 38, Issue 2, 2001, Pages 121-145

  Hoskens, Machteld ^a   Wilson, Mark ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0035536108     PISSN: 00220655     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.1745-3984.2001.tb01119.x     Document Type: Article

References (30)

1. Andrich, A. (1985). An elaboration of Guttman scaling with Rasch models for measurement. In T. Nancy Brandon (Ed.), Sociological methodology (pp. 33-80). San Francisco: Jossey-Bass Inc.
2. Ashburn, R. (1938). An experiment in the essay-type question. Journal of Experimental Education, 7, 1-3.
3. Braun, H. I. (1988). Understanding scoring reliability: Experiments in calibrating essay readers. Journal of Educational Statistics, 13(1).
4. Bryk, A. S., Raudenbush, S. W., & Congdon, R. T. (1996). HLM (Version 4). Lincolnwood, IL: Scientific Software.
5. Craven, R. G., Marsh, H. W., & Debus, R. L. (1999). Diffusion effects: Control group contamination threats to the validity of teacher-administered interventions. Unpublished manuscript, University of Western Sydney, Campbelltown, Australia.
6. Eells, W. C. (1930). Reliability of repeated grading of essay type examinations. Journal of Educational Psychology, 21, 48-52.
7. Engelhard, G. J. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93-112.
8. Engelhard, G. J. (1996). Evaluating rater accuracy in performance assessments. Journal of Educational Measurement, 33, 56-70.
9. Fischer, G. H. (1973). The linear logistic test model as an instrument in educational research. Acta Psychologica, 37, 359-374.
10. Fischer, G. H. (1989). An IRT-based model for dichotomous longitudinal data. Psychometrika, 54, 599-624.
11. Fischer, G. H., & Ponocny, I. (1994). An extension of the partial credit model with an application to the measurement of change. Psychometrika, 59, 177-192.
12. Kaplan, R. M., & Saccuzzo, D. P. (1997). Psychological testing. Principles, applications, and issues. (4th ed.). Pacific Grove, CA: Brooks/Cole.
13. Linacre, J. M. (1989). Many-facet Rasch measurement. Chicago: MESA Press.
14. Longford, N. T. (1995). Models for uncertainty in educational testing. New York: Springer-Verlag.
15. Lunz, M., Wright, B. D., & Linacre, J. M. (1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3, 331-345.
16. Masters, G. N. (1988). The analysis of partial credit scoring. Applied Measurement in Education, 1, 279-297.
17. Muraki, E. (1992). A generalized partial credit model. Applied Psychological Measurement, 16, 159-176.
18. Myford, C. M., & Mislevy, R. J. (1995). Monitoring and improving a portfolio assessment system. (CSE Technical Report 402). Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing.
19. Patz, R. (1996). Markov chain montecarlo methods for item response theory models with applications for the National Assessment of Educational Progress. Unpublished Dissertation, Carnegie-Mellon University, Pittsburgh.
20. Patz, R. J., & Junker, B. W. (1997). Fitting item response models to incomplete, multiple item type educational assessment data using Markov chain Monte Carlo methods. Manuscript submitted for publication.
21. Patz, R. J., Wilson, M., & Hoskens, M. (1997). Optimal rating procedures and methodology for NAEP open-ended items: Final report to the National Center for Education Statistics under the redesign of NAEP.
22. Raymond, M. R., & Viswesvaran, C. (1993). Least squares models to correct for rater effects in performance assessment. Journal of Educational Measurement, 30, 253-268.
23. Rost, J. (1988). Measuring attitudes with a threshold model drawing on a traditional scaling concept. Applied Psychological Measurement, 12, 397-409.
24. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
25. Shavelson, R. J., Baxter, G. P., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30, 215-232.
26. Wilson, M., & Case, H. (2000). An examination of variation in rater severity over time: A study in rater drift. In M. Wilson & G. Engelhard (Eds.), Objective Measurement: Theory into Practice (Vol. V, pp. 113-133). Stamford, CT: Ablex.
27. Wilson, M., & Wang, W.-C. (1995). Complex composites: Issues that arise in combining different modes of assessment. Special Issue: Polytomous item response theory. Applied Psychological Measurement, 19, 51-71.
28. Wolfe, E. W., & Myford, C. M. (1997). Detecting rater effects with a multi-faceted rating scale model. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.
29. Wu, M., Adams, R., & Wilson, M. (1997). Conquest [Computer program]. Melbourne, Australia: Australian Council for Educational Research.
30. Wu, M. L. (1997). The development and application of a fit test for use with marginal maximum likelihood estimation and generalised item response models. Unpublished Master's Dissertation, University of Melbourne, Melbourne.