Classification consistency and accuracy for complex assessments using item response theory

Journal of Educational Measurement

Volumn 47, Issue 1, 2010, Pages 1-17

  Lee, Won Chan ^a  

^a UNIVERSITY OF IOWA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77953800600     PISSN: 00220655     EISSN: 17453984     Source Type: Journal    
DOI: 10.1111/j.1745-3984.2009.00096.x     Document Type: Article

References (39)

1. Brennan R L. Generalizability theory 2001a, New York, NY, Springer-Verlag
2. Brennan R L. An essay on the history and future of reliability from the perspective of replications. Journal of Educational Measurement 2001b, 38:295-317.
3. Brennan R L. BB-CLASS: A computer program that uses the beta-binomial model for classification consistency and accuracy (Version 1.0) (CASMA Research Report No. 9) [Computer software and manual. 2004, http://www.education.uiowa.edu/casma, Iowa City, IA, Center for Advanced Studies in Measurement and Assessment, University of Iowa, Available at
4. Brennan R L, Lee W. Correcting for bias in single-administration decision consistency indexes 2006a, CASMA Research Report No. 18, Iowa City, IA, Center for Advanced Studies in Measurement and Assessment, University of Iowa
5. Brennan R L, Lee W. Some perspectives on KR-21 2006b, CASMA Technical Note No. 2, Iowa City, IA, Center for Advanced Studies in Measurement and Assessment, University of Iowa
6. Brennan R L, Wan L. Bootstrap procedures for estimating decision consistency for single-administration complex assessments 2004, CASMA Research Report No. 7, Iowa City, IA, Center for Advanced Studies in Measurement and Assessment, University of Iowa
7. Breyer F J, Lewis C. Pass-fail reliability for tests with cut scores: A simplified method 1994, ETS Research Report No. 94-39, Princeton, NJ, Educational Testing Service
8. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960, 20:37-46.
9. Ercikan K, Schwarz R D, Julian M W, Burket G R, Weber M M, Link V. Calibration and scoring of tests with multiple-choice and constructed-response item types. Journal of Educational Measurement 1998, 35:137-154.
10. Hanson B A. An extension of the Lord-Wingersky algorithm to polytomous items 1994, Unpublished research note., ACT, Inc. Iowa City, IA
11. Hanson B A, Brennan R L. An investigation of classification consistency indexes estimated under alternative strong true score models. Journal of Educational Measurement 1990, 27:345-359.
12. Hoover H D, Hieronymus A N, Frisbie D A, Dunbar S B. Iowa Tests of Basic Skills: Form M: Levels 13-14 1996a, Itasca, IL, Riverside Publishing
13. Hoover H D, Hieronymus A N, Frisbie D A, Dunbar S B. Constructed-Response Supplement to the Iowa Tests Form 1: Levels 13-14 1996b, Itasca, IL, Riverside Publishing
14. Huynh H. On the reliability of decisions in domain-referenced testing. Journal of Educational Measurement 1976, 13:253-264.
15. Huynh H. Computation and statistical inference for decision consistency indexes based on the Rasch model. Journal of Educational Statistics 1990, 15:353-368.
16. Kang T, Cohen A S, Sung H-J. IRT model selection methods for polytomous items 2005, April, Montreal, Canada
17. Kolen M J, Brennan R L. Test equating, scaling, and linking: Methods and practices 2004, 2nd ed., New York, NY, Springer-Verlag
18. Lee W. Multinomial and compound multinomial error models for tests with complex item scoring. Applied Psychological Measurement 2007, 31:255-274.
19. Lee W. MULT-CLASS: A computer program for multinomial and compound multinomial classification consistency and accuracy (Version 3.0) [Computer software 2008, http://www.education.uiowa.edu/casma, Iowa City, IA, Center for Advanced Studies in Measurement and Assessment, University of Iowa, Available at
20. Lee W, Brennan R L, Kolen M J. Estimators of conditional scale-score standard errors of measurement: A simulation study. Journal of Educational Measurement 2000, 37:1-20.
21. Lee W, Brennan R L, Wan L. Classification consistency and accuracy for complex assessments under the compound multinomial model. Applied Psychological Measurement 2009, 33:374-390.
22. Lee W, Hanson B A, Brennan R L. Estimating consistency and accuracy indices for multiple classifications. Applied Psychological Measurement 2002, 26:412-432.
23. Lee W, Kolen M J. IRT-CLASS: A computer program for item response theory classification consistency and accuracy (Version 2.0) [Computer software 2008, http://www.education.uiowa.edu/casma, Iowa City, IA, Center for Advanced Studies in Measurement and Assessment, University of Iowa, Available at
24. Li H-H, Stout W F. Assessment of unidimensionality for mixed polytomous and dichotomous item data: Refinements of Poly-DIMTEST 1995, April, San Francisco, CA
25. Livingston S A, Lewis C. Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32 1995, 179-197.
26. Lord F M, Wingersky M S. Comparison of IRT true-score and equipercentile observed-score " equatings." Applied Measurement in Education 1984, 8:452-461.
27. Masters G N, Wright B D. The partial credit model. Handbook of modern item response theory 1997, 153-185. van der Linden W J, Hambleton R K. In, Eds., pp, New York, NY, Springer-Verlag
28. Muraki E. A generalized partial credit model. Handbook of modern item response theory 1997, 153-185. van der Linden W JHambleton R K. Eds., pp, New York, NY, Springer-Verlag
29. Muraki E, Bock R D. PARSCALE 4: IRT item analysis and test scoring for rating-scale data [Computer software and manual 2003, Chicago, IL, Scientific Software International
30. Orlando M, Thissen D. New item fit indices for dichotomous item response theory models. Applied Psychological Measurement 2000, 24:50-64.
31. Peng C J, Subkoviak M J. A note on Huynh's normal approximation procedure for estimating criterion-referenced reliability. Journal of Educational Measurement 1980, 17:359-368.
32. Rudner L M. Computing the expected proportions of misclassified examinees. Practical Assessment Research & Evaluation 2001, 7(14). http://PAREonline.net/getvn.asp?v=7&n=14, Retrieved November 1, 2006, from
33. Rudner L M. Expected classification accuracy. Practical Assessment Research & Evaluation 2005, 10(13). http://PAREonline.net/getvn.asp?v=10&n=13, Retrieved November 21, 2006, from
34. Samejima F. Graded response model. Handbook of modern item response theory 1997, 85-100. van der Linden W JHambleton R K. In, Eds., pp, New York, NY, Springer-Verlag
35. Schulz E M, Kolen M J, Nicewander W A. A rationale for defining achievement levels using IRT-estimated domain scores. Applied Psychological Measurement 1999, 23:347-362.
36. Stout W F. A new item response theory modeling approach with applications to unidimensionality assessment and ability estimation. Psychometrika 1990, 55:293-325.
37. Subkoviak M J. Estimating reliability from a single administration of a criterion-referenced test. Journal of Educational Measurement 1976, 13:265-276.
38. Thissen D, Pommerich M, Billeaud K, Williams V S L. Item response theory for scores on tests including polytomous items with ordered responses. Applied Psychological Measurement 1995, 19:39-49.
39. Wang T, Kolen M J, Harris D J. Psychometric properties of scale scores and performance levels for performance assessments using polytomous IRT. Journal of Educational Measurement 2000, 37:141-162.