Estimating consistency and accuracy indices for multiple classifications

Applied Psychological Measurement

Volumn 26, Issue 4, 2002, Pages 412-432

  Lee, Won Chan ^a   Hanson, Bradley A ^b   Brennan, Robert L ^c  

^a American College Testing Incorporated   (United States)

^b CTB MCGRAW HILL   (United States)

^c UNIVERSITY OF IOWA   (United States)

Author keywords

Beta binomial; Classification accuracy; Classification consistency; Contingency table; Cutoff scores; Item response theory; Multiple classifications

Indexed keywords

EID: 0036447794     PISSN: 01466216     EISSN: None     Source Type: Journal    
DOI: 10.1177/014662102237797     Document Type: Article

References (33)

1. ACT, Inc. (1998). Characteristics of the Work Keys assessments. Iowa City, IA: Author.
2. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: Author.
3. Berk, R. A. (1984). Selecting the index of reliability. In R. A. Berk (Ed.), A guide to criterion-referenced test construction. Baltimore: Johns Hopkins University Press.
4. Brennan, R. L. (1981). Some statistical procedures for domain-referenced testing: A handbook for practitioners (ACT Technical Bulletin No. 38). Iowa City, IA: ACT, Inc.
5. Brennan, R. L. (2001). An essay on the history and future of reliability from the perspective of replications. Journal of Educational Measurement, 38, 295-317.
6. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
7. Dayton, C. M. (1998). Latent class scaling analysis (Sage University Papers Series on Quantitative Applications in the Social Sciences, series No. 07-126). Thousand Oaks, CA: Sage.
8. Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), Educational measurement (3rd ed.). New York: American Council on Education and Macmillan.
9. Hanson, B. A. (1991). Method of moments estimates for the four-parameter beta compound binomial model and the calculation of classification consistency indexes (ACT Research Report 91-5). Iowa City, IA: ACT, Inc.
10. Hanson, B. A., & Brennan, R. L. (1990). An investigation of classification consistency indexes estimated under alternative strong true score models. Journal of Educational Measurement, 27, 345359.
11. Huynh, H. (1976). On the reliability of decisions in domain-referenced testing. Journal of Educational Measurement, 13, 253-264.
12. Huynh, H. (1978). Reliability of multiple classifications. Psychometrika, 43, 317-325.
13. Huynh, H. (1980). Statistical inference for false positive and false negative error rates in mastery testing. Psychometrika, 45, 107-120.
14. Huynh, H. (1990). Computation and statistical inference for decision consistency indexes based on the Rasch model. Journal of Educational Statistics, 15, 353-368.
15. Huynh, H., & Saunders, J. C. (1980). Accuracy of two procedures for estimating reliability of mastery tests. Journal of Educational Measurement, 77, 351-358.
16. Keats, J. A., & Lord, F. M. (1962). A theoretical distribution of mental test scores. Psychometrika, 27, 59-72.
17. Kolen, M. J., & Brennan, R. L. (1995). Test equating: Methods and practices. New York: Springer-Verlag.
18. Langeheine, R., & Rost, J. (Ed.). (1988). Latent trait and latent class models. New York: Plenum Press.
19. Lee, W., Brennan, R. L., & Kolen, M. J. (2000). Estimators of conditional scale-score standard errors of measurement: A simulation study. Journal of Educational Measurement, 37, 1-20.
20. Lee, W., Hanson, B. A., & Brennan, R. L. (2000). Procedures for computing classification consistency and accuracy indices with multiple categories (ACT Research Report 2000-10). Iowa City, IA: ACT, Inc.
21. Livingston, S. A., & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32, 179-197.
22. Lord, F. M. (1965). A strong true-score theory, with applications. Psychometrika, 30, 239-270.
23. Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
24. Lord, F. M. (1984). Standard errors of measurement at different score levels. Journal of Educational Measurement, 21, 239-243.
25. Lord, F. M., & Wingersky, M. S. (1984). Compari- son of IRT true-score and equipercentile observed-score "equatings." Applied Psychological Measurement, 8, 453-461.
26. Mislevy, R. J., & Bock, R. D. (1990). BILOG3. Item and test scoring with binary logistic models (2nd ed.). Mooresville, IN: Scientific Software.
27. Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1992). Numerical recipes in Fortran (2nd ed.). New York: Cambridge University Press.
28. Schulz, E. M., Kolen, M. J., & Nicewander, W. A. (1997). A study of modified-Guttman and IRT-based level scoring procedures for Work Keys assessments (ACT Research Report 97-7). Iowa City, IA: ACT, Inc.
29. Schulz, E. M., Kolen, M. J., & Nicewander, W. A. (1999). A rationale for defining achievement levels using IRT-estimated domain scores. Applied Psychological Measurement, 23, 347-362.
30. Stocking, M. L., & Lord, F. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7, 201-210.
31. Subkoviak, M. J. (1984). Estimating the reliability of mastery-nonmastery classifications. In R. A. Berk (Ed.), A guide to criterion-referenced test construction. Baltimore: Johns Hopkins University Press.
32. Wang, T., Kolen, M. J., & Harris, D. J. (2000). Psychometric properties of scale scores and performance levels for performance assessments using polytomous IRT. Journal of Educational Measurement, 57, 141-162.
33. Wilcox, R. R. (1977). Estimating the likelihood of false-positive and false-negative decisions in mastery testing: An empirical Bayes approach. Journal of Educational Statistics, 2, 289-307.