Classification consistency and accuracy for complex assessments under the compound multinomial model

Applied Psychological Measurement

Volumn 33, Issue 5, 2009, Pages 374-390

  Lee, Won Chan ^a   Brennan, Robert L ^a   Wan, Lei ^b  

^a UNIVERSITY OF IOWA   (United States)

^b Pearson Educational Measurement ^*  (United States)

Author keywords

Bootstrap; Classification accuracy; Classification consistency; Complex assessment; Compound multinomial model

Indexed keywords

EID: 70449339776     PISSN: 01466216     EISSN: 15523497     Source Type: Journal    
DOI: 10.1177/0146621608321759     Document Type: Article

References (21)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: Author.
2. Brennan, R.L. (2001). Generalizability theory. New York: Springer-Verlag.
3. Brennan, R.L. (2004). BB-CLASS: A computer program that uses the beta-binomial model for classification consistency and accuracy (Version 1.0, CASMA Research Report No. 9). Iowa City: University of Iowa, Center for Advanced Studies in Measurement and Assessment. Available at http://www.education.uiowa.edu/casma.
4. Brennan, R.L., Harris, D.J., & Hanson, B.A. (1987). The bootstrap and other procedures for examining the variability of estimated variance components in testing contexts (American College Testing Research Report No. 87-7). Iowa City, IA: ACT, Inc. No. 87-7
5. Brennan, R.L., & Lee, W. (2006a). Correcting for bias in single-administration decision consistency indexes (CASMA Research Report No. 18). Iowa City: University of Iowa, Center for Advanced Studies in Measurement and Assessment. Available at http://www.education.uiowa.edu/casma.
6. Brennan, R.L., & Lee, W. (2006b). Some perspectives on KR-21 (CASMA Technical Note No. 2). Iowa City: University of Iowa, Center for Advanced Studies in Measurement and Assessment. Available at http://www.education.uiowa.edu/casma.
7. Breyer, F.J., & Lewis, C. (1994). Pass-fail reliability for tests with cut scores: A simplified method (ETS Research Report No. 94-39). Princeton, NJ: Educational Testing Service.
8. Efron, B. (1982). The jackknife, the bootstrap, and other resampling plans. Philadelphia: Society for Industrial and Applied Mathematics.
9. Efron, B., & Tibshirani, R. (1986). Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science, 1, 54-77.
10. Hanson, B.A., & Brennan, R.L. (1990). An investigation of classification consistency indexes estimated under alternative strong true score models. Journal of Educational Measurement, 27, 345-359.
11. Huynh, H. (1976). On the reliability of decisions in domain-referenced testing. Journal of Educational Measurement, 13, 253-264.
12. Kelley, T.L. (1947). Fundamentals of statistics. Cambridge, UK: Harvard University Press.
13. Lee, W. (2007). Multinomial and compound multinomial error models for tests with complex item scoring. Applied Psychological Measurement, 31, 255-274.
14. Lee, W. (2008). MULT-CLASS: A computer program for multinomial and compound-multinomial classification consistency and accuracy (Version 3.0). Iowa City: University of Iowa, Center for Advanced Studies in Measurement and Assessment. Available at http://www.education.uiowa.edu/casma.
15. Lee, W., Hanson, B.A., & Brennan, R.L. (2002). Estimating consistency and accuracy indices for multiple classifications. Applied Psychological Measurement, 26, 412-432.
16. Livingston, S.A., & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32, 179-197.
17. No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425 (2002).
18. Peng, C.J., & Subkoviak, M.J. (1980). A note on Huynh's normal approximation procedure for estimating criterion-referenced reliability. Journal of Educational Measurement, 17, 359-368.
19. Subkoviak, M.J. (1976). Estimating reliability from a single administration of a criterion-referenced test. Journal of Educational Measurement, 13, 265-276.
20. Wan, L., Brennan, R.L., & Lee, W. (2007). Estimating classification consistency for complex assessments (CASMA Research Report No. 22). Iowa City: University of Iowa, Center for Advanced Studies in Measurement and Assessment. Available at http://www.education.uiowa.edu/casma.
21. Woodruff, D.J., & Sawyer, R.L. (1989). Estimating measures of pass-fail reliability from parallel half-tests. Applied Psychological Measurement, 13, 33-43.