Covariates of the rating process in hierarchical models for multiple ratings of test items

Journal of Educational and Behavioral Statistics

Volumn 32, Issue 3, 2007, Pages 287-314

  Mariano, Louis T ^a   Junker, Brian W ^b  

^a RAND CORPORATION   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Bayesian hierarchical models; Item response theory; Rater bias; Repeated measures

Indexed keywords

EID: 34948909683     PISSN: 10769986     EISSN: None     Source Type: Journal    
DOI: 10.3102/1076998606298033     Document Type: Article

References (29)

1. Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-479). Reading, MA: Addison-Wesley.
2. Bock, R. D. (1975). Multivariate statistical methods in behavioral research. New York: McGraw-Hill.
3. Brennan, R. L. (1992). Elements of generalizability theory. Iowa City, IA: American College Testing.
4. Chib, S., & Greenberg, E. (1995). Understanding the Metropolis-Hastings algorithm. The American Statistician, 49, 327-335.
5. Engelhard, G. (1994). Examining rater errors in the assessment of written composition with many-faceted Rasch models. Journal of Educational Measurement, 31, 93-112.
6. Engelhard, G. (1996). Evaluating rater accuracy in performance assessments. Journal of Educational Measurement, 33, 56-70.
7. Fischer, G. H. (1973). The linear logistic test model as an instrument in educational research. Acta Psychologica, 37, 359-374.
8. Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (1995). Bayesian data analysis. London: Chapman & Hall.
9. Hemker, B. T., Sijtsma, K., Molenaar, I. W., & Junker, B. W. (1996). Polytomous IRT models and monotone likelihood ratio of the total score. Psychometrika, 61, 679-693.
10. Junker, B. W., & Patz, R. J. (1998, June). The hierarchical rater model for rated test items. Presented at the annual North American meeting of the Psychometric Society, Urbana-Champaign, IL.
11. Kass, R., & Raftery, A. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773-795.
12. Kirk, R. E. (1982). Experimental design: Procedures for the behavioral sciences (2nd ed.). Monterey, CA: Brooks/Cole.
13. Linacre, J. M. (1989). Many-faceted Rasch measurement. Chicago: MESA Press.
14. Mariano, L. T. (2002). Information accumulation, model selection and rater behavior in constructed response student assessments. Unpublished doctoral dissertation, Department of Statistics, Carnegie Mellon University.
15. Maris, E. (1995). Psychometric latent response models. Psychometrika, 60, 523-547.
16. Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
17. Muraki, E. (1997). A generalized partial credit model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 153-164). New York: Springer-Verlag.
18. Nandram, B., & Chen, M. (1996). Reparameterizing the generalized linear model to accelerate Gibbs sampler convergence. Journal of Statistical Computation and Simulation, 54, 129-144.
19. Patz, R. J. (1996). Markov Chain Monte Carlo methods for item response theory models with applications for the National Assessment of Educational Progress. Unpublished doctoral dissertation, Department of Statistics, Carnegie Mellon University.
20. Patz, R. J., Awamleh, J., & Kelly, R. (1999). Golden State Exam Image-Scoring Pilot Study (Tech. rep.). Monterey, CA.
21. Patz, R. J., & Junker, B. W. (1999a). A straightforward approach to Markov Chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24, 146-178.
22. Patz, R. J., & Junker, B. W. (1999b). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24, 342-366.
23. Patz, R. J., Junker, B. W., Johnson, M. S., & Mariano, L. T. (2002). The hierarchical rater model for rated test items and its application to large-scale educational assessment data. Journal of Educational and Behavioral Statistics, 27, 341-384.
24. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Educational Research.
25. Samejima, F. (1969). Estimation of latent trait ability using a response pattern of graded scores. Psychometrika Monograph, No. 17.
26. Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461-464.
27. Tanner, M. A., & Wong, W. H. (1987). The calculation of posterior distributions by data augmentation. Journal of the American Statistical Association, 82, 528-549.
28. Verhelst, N., & Verstralen, H. (2001). IRT models for multiple raters. In A. Boomsma, M. A. J. van Duijn, & T. A. B. Snijders (Eds.), Essays on item response theory (pp. 89-108). New York: Springer-Verlag.
29. Wilson, M., & Hoskens, M. (2001). The rater bundle model. Journal of Educational and Behavioral Statistics, 26, 283-306.