Combining CAT with cognitive diagnosis: A weighted item selection approach

Behavior Research Methods

Volumn 44, Issue 1, 2012, Pages 95-109

  Wang, Chun ^a   Chang, Hua Hua ^a   Douglas, Jeffery ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

a stratification; CAT; Cognitive diagnosis; Constraint weighted item selection

Indexed keywords

ARTICLE; COMPUTER PROGRAM; COMPUTER SIMULATION; EDUCATION; HUMAN; METHODOLOGY; STATISTICAL MODEL;

COMPUTER SIMULATION; EDUCATIONAL MEASUREMENT; HUMANS; MODELS, STATISTICAL; SOFTWARE;

EID: 84863147461     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-011-0143-3     Document Type: Article

References (45)

1. Chang, H. (2004). Understanding Computerized Adaptive Testing - From Robbins-Monro to Lord and beyond. In David Kaplan (Ed.) The SAGE handbook of quantitative methodology for the social sciences (pp. 117-133), Sage Publications. www. sagepub. com/book. aspx?pid=10048.
2. Chang, H.-H., & Ying, Z. (1999). a-stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23, 211-222.
3. Chang, H.-H., & Ying, Z. (2008). To weight or not to weight? Balancing influence of initial items in adaptive testing. Psychometrika, 73, 441-450.
4. Cheng, Y. (2009). When cognitive diagnosis meets computerized adaptive testing: CD-CAT. Psychometrika, 74, 619-632.
5. Cheng, Y., & Chang, H.-H. (2009). The maximum priority index method for severely constrained item selection in computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 62, 369-383.
6. Cheng, Y., Chang, H.-H., & Yi, Q. (2007). Two-phase item selection procedure for flexible content balancing in CAT. Applied Psychological Measurement, 31, 467-482.
7. Cheng, Y., Guo, F., Chang, H., & Douglas, J. (2009). Constraint weighted a-stratification for computerized adaptive testing with nonstatistical constraints: Balancing measurement efficiency and exposure control. Educational and Psychological Measurement, 69, 35-49.
8. de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115-130.
9. de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353.
10. DiBello, L. V., Stout, W. F., & Roussos, L. A. (1995). Unified cognitive/ psychometric diagnostic assessment likelihood-based classification techniques. In P. D. Nichols, D. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 361-389). Hillsdale, NJ: Erlbaum.
11. Gibbons, R. D., & Hedeker, D. (1992). Full-information item bi-factor analysis. Psychometrika, 57, 423-436.
12. Greeno, J. G. (1980). Trends in the theory of knowledge for problem solving. In D. T. Tuma & F. Reif (Eds.), Problem solving and education: Issues in teaching and research (pp. 9-23). Hillsdale, NJ: Erlbaum.
13. Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333-352.
14. Hau, K. T., & Chang, H. (2001). Item selection in computerized adaptive testing: Should more discriminating items be used first? Journal of Educational Measurement, 38, 249-266.
15. Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29, 262-277.
16. Henson, R., Roussos, L., Douglas, J., & He, X. (2008). Cognitive diagnostic attribute-level discrimination indices. Applied Psychological Measurement, 32, 275-288.
17. Holzinger, K. J., & Swineford, F. (1937). The bi-factor method. Psychometrika, 2, 41-54.
18. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.
19. Kingsbury, G. G. (2009). Adaptive item calibration: A process for estimating item parameters within a computerized adaptive test. In D. J. Weiss (Ed.), Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing.
20. Leung, C., Chang, H., & Hau, K. (2005). Computerized adaptive testing: A mixture item selection approach for constrained situations. British Journal of Mathematical and Statistical Psychology, 58, 239-257.
21. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
22. Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187-212.
23. McGlohen, M., & Chang, H.-H. (2008). Combining computer adaptive testing technology with cognitively diagnostic assessment. Behavior Research Methods, 40, 808-821.
24. Pardos, Z. A., Beck, J. E., & Heffernan, N. T. (2008). The composition effect: Conjunctive or compensatory? An analysis of multi-skill math questions in ITS. Retrieved from http://www. educationaldatamining. org/EDM2008/uploads/proc/15_Pardos_44. pdf.
25. Rupp, A. A., & Templin, J. (2008). The effects of Q-Matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68, 78-96.
26. Segall, D. O. (1996). Multidimensional adaptive testing. Psychometrika, 2, 331-354.
27. Stocking, M. L., & Swanson, L. (1993). A method for severely constrained item selection in adaptive testing. Applied Psychological Measurement, 17, 277-292.
28. Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.
29. Tatsuoka, K. K. (1985). A probabilistic model for diagnosing misconceptions in the pattern classification approach. Journal of Educational Statistics, 10, 55-73.
30. Tatsuoka, K. K., & Tatsuoka, M. M. (1987). Bug distribution and pattern classification. Psychometrika, 52(2), 193-206.
31. Tatsuoka, K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Safto (Eds.), Monitoring skills and knowledge acquisition (pp. 453-488). Hillsdale, NJ: Erlbaum.
32. Tatsuoka, K. K. (1991). Boolean algebra applied to determination of universal set of knowledge states (Tech. Rep. RR-91-44-ONR). Princeton, NJ: Educational Testing Service.
33. Tatsuoka, K. K., & Tatsuoka, M. M. (1997). Computerized cognitive diagnostic adaptive testing: Effect on remedial instruction as empirical validation. Journal of Educational Measurement, 34(1), 3-20.
34. Tatsuoka, C. (2002). Data-analytic methods for latent partially ordered classification models. Applied Statistics (JRSS-C), 51, 337-350.
35. Templin, J., & Henson, R. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305.
36. Templin, J., Henson, R., & Douglas, J. (2008). General theory and estimation of cognitive diagnosis models: Using Mplus to retrieve model estimates. Manuscript under review.
37. Templin, J., Henson, R., Templin, S., & Roussos, L. (2008). Robustness of hierarchical modeling of skill association in cognitive diagnosis models. Applied Psychological Measurement, 32, 559-574.
38. Timminga, E. (1998). Solving infeasibility problems in computerized test assembly. Applied Psychological Measurement, 22, 280-291.
39. van der Linden, W. J. (2000). Constrained adaptive testing with shadow tests. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 27-52). Dordrecht: Kluwer.
40. van der Linden, W. J., & Chang, H. (2003). Implementing content constraints in Alpha-Stratified adaptive testing using a shadow test approach. Applied Psychological Measurement, 27(2), 107-120.
41. von Davier, M. (2005). A general diagnostic model applied to language testing data (ETS Research Report RR-05-16). Princeton, NJ: Educational Testing Service.
42. Wainer, H. D., Dorans, N. J., Flaugher, R., Green, B. F., Mislevy, F. J., Steinberg, L., et al. (1990). Computerized adaptive testing: A primer. Hillsdale, NJ: Erlbaum.
43. Xu, X., Chang, H.-H., & Douglas, J. (2003). Computerized adaptive testing strategies for cognitive diagnosis. Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Canada.
44. Yamamoto, K. (1989). HYBRID model of IRT and latent class model. ETS research report series (RR 89-41). Princeton, NJ: Educational Testing Service.
45. Yi, Q., Zhang, J., & Chang, H. (2008). Severity of organized item theft in computerized adaptive testing: A Simulation study. Applied Psychological Measurement, 32(7), 543-558.