Reporting diagnostic scores in educational testing: Temptations, Pitfalls, and some solutions

Multivariate Behavioral Research

Volumn 45, Issue 3, 2010, Pages 553-573

  Sinharay, Sandip ^a   Puhan, Gautam ^a   Haberman, Shelby J ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77953519116     PISSN: 00273171     EISSN: None     Source Type: Journal    
DOI: 10.1080/00273171.2010.483382     Document Type: Article

References (48)

1. Ackerman, T., & Shu, Z. (2009, April). Using confirmatory MIRT modeling to provide diagnostic information in large scale assessment. Paper presented at the annual meeting of the National Council of Measurement in Education, San Diego, CA.
2. Almond, R. G., DiBello, L. V., Moulder, B., & Zapata-Rivera, J. (2007). Modeling diagnostic assessment with Bayesian networks. Journal of Educational Measurement, 44, 341-359.
3. American Educational Research Association (AERA), American Psychological Association, National Council on Measurement in Education. (1999). Standards for Educational and Psychological Testing. Washington, DC: AERA.
4. Beaton, A. E., & Allen, N. L. (1992). Interpreting scales through scale anchoring. Journal of Educational Statistics, 17, 191-204.
5. de la Torre, J., & Douglas, J. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353.
6. de la Torre, J., & Patz, R. J. (2005). Making the most of what we have: A practical application of multidimensional IRT in test scoring. Journal of Educational and Behavioral Statistics, 30, 295-311.
7. DiBello, L. V., Roussos, L., & Stout, W. F. (2007). Review of cognitive diagnostic assessment and a summary of psychometric models. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics, Volume 26 (pp. 45-79). Amsterdam: Elsevier Science B.V.
8. DiBello, L. V., Stout, W. F., & Roussos, L. A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 361-389). Hillsdale, NJ: Erlbaum.
9. Educational Testing Service. (2008). PraxisTM 2008-09 Information Bulletin. Princeton, NJ: Author.
10. Embretson, S. E. (1997). Multicomponent latent trait models. In W. van der Linden & R. Hambleton (Eds.), Handbook of modern item response theory (pp. 305-322). New York: Springer-Verlag.
11. Fu, J., & Li, Y. (2007, April). Cognitively diagnostic psychometric models: An integrative review. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
12. Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality Psychology in Europe (Vol. 7, pp. 7-28). Tilburg, The Netherlands: Tilburg University Press.
13. Haberman, S. J. (2008a). When can subscores have value? Journal of Educational and Behavioral Statistics, 33, 204-229.
14. Haberman, S. J. (2008b). Subscores and validity (ETS Research Report No. RR-08-64). Princeton, NJ: Educational Testing Service.
15. Haberman, S. J., & Sinharay, S. (in press). Reporting of subscores using multidimensional item response theory. Psychometrika.
16. Haberman, S. J., Sinharay, S., & Puhan, G. (2009). Reporting subscores for institutions. British Journal of Mathematical and Statistical Psychology, 62, 79-95.
17. Harris, D. J., & Hanson, B. A. (1991, April). Methods of examining the usefulness of subscores. Paper presented at the annual meeting of National Council on Measurement in Education, Chicago, IL.
18. Henson, R., Templin, J., & Douglas, J. (2007). Using efficient model based sum-scores for conducting skills diagnoses. Journal of Educational Measurement, 44, 361-376.
19. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.
20. Katz, I. R., Attali, Y., Rijmen, F., & Williamson, D. M. (2008, April). ETS's iSkillsTM assessment: Measurement of information and communication technology literacy. Paper presented at the twenty-third annual conference of the Society for Industrial and Organizational Psychology, San Francisco, CA.
21. Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The attribute hierarchy model for cognitive assessment: A variation on Tatsuoka's rule-space approach. Journal of Educational Measurement, 41,205-237.
22. Ling, G. (2009, April). Why the major field (business) test does not report subscores of individual testtakers-reliability and construct validity evidence. Paper presented at the annual meeting of the National Council of Measurement in Education, San Diego, CA.
23. Luecht, R. M., Gierl, M. J., Tan, X., & Huff, K. (2006, April). Scalability and the development of useful diagnostic scales. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
24. Lyren, P. (2009). Reporting subscores from college admission tests. Practical Assessment, Research, and Evaluation, 14, 1-10.
25. Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187-212.
26. National Research Council. (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: The National Academies Press.
27. Puhan, G., & Liang, L. (in press). Equating subscores under the non equivalent anchor test (NEAT) design (ETS Research Report). Princeton, NJ: Educational Testing Service.
28. Puhan, G., Sinharay, S., Haberman, S. J., & Larkin, K. (2008). Comparison of subscores based on classical test theory. ETS Research Report Series 08-54. Educational Testing Service. October, 2008.
29. Reckase, M. D. (1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 21, 25-36.
30. Roussos, L. A., DiBello, L. V., Stout, W. F., Hartz, S. M., Henson, R. A., & Templin, J. L. (2007). The fusion model skills diagnostic system. In J. Leighton & M. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 275-318). Cambridge, UK: Cambridge University Press.
31. Rupp, A. A., & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement, 6, 219-262.
32. Sinharay, S. (in press). How often do subscores have added value? Results from operational and simulated data. Journal of Educational Measurement.
33. Sinharay, S., & Haberman, S. J. (2008a). Reporting subscores: A survey (ETS Research Memoran- dum No. RR-08-18). Princeton, NJ: Educational Testing Service.
34. Sinharay, S., & Haberman, S. J. (2008b). How much can we reliably know about what students know? Measurement: Interdisciplinary Research and Perspectives, 6, 46-49.
35. Sinharay, S., Haberman, S. J., & Puhan, G. (2007). Subscores based on classical test theory: To report or not to report. Educational Measurement: Issues and Practice, 26(4), 21-28.
36. Stone, C. A., Ye, F., Zhu, X., & Lane, S. (2010). Providing subscale scores for diagnostic information: A case study when the test is essentially unidimensional. Applied Measurement in Education, 23(1),63-86.
37. Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617.
38. Tate, R. L. (2004). Implications of multidimensionality for total score and subscore performance. Applied Measurement in Education, 17, 89-112.
39. Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.
40. Tatsuoka, K. K. (2009). Cognitive assessment: An introduction to the rule space method. New York: Routledge Academic.
41. Thissen-Roe, A., Hunt, E., & Minstrell, I. (2004). The DIAGNOSER project. Combining assessment and learning. Behavioral Research Methods, Instruments, and Computers, 36, 234-240.
42. von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287-307.
43. von Davier, M., DiBello, L. V., & Yamamoto, K. (2008). Reporting test outcomes using models for cognitive diagnosis. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 151-176). Toronto: Hogrefe & Huber.
44. Wainer, H., Vevea, J. L., Camacho, F., Reeve, B. B., Rosa, K., Nelson, L.x,et-al (2001). Augmented scores-borrowing strength to compute scores based on small numbers of items. In D. Thissen & H. Wainer (Eds.), Test scoring (pp. 343-387). Mahwah, NJ: Erlbaum.
45. Yao, L., & Boughton, K. A. (2007). A multidimensional item response modeling approach for improving subscale proficiency estimation and classification. Applied PsychologicalMeasurement, 31,83-105.
46. Yen, W. M. (1987, June). A Bayesian/IRT index of objective performance. Paper presented at the annual meeting of the Psychometric Society, Montreal, Quebec, Canada.
47. Zhang, J., & Stout, W. (1999). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213-249.
48. Zwick, R., Senturk, D., Wang, J., & Loomis, S. C. (2001). An investigation of alternative methods for item mapping on the National Assessment of Educational Progress. Educational Measurement: Issues and Practice, 20, 15-25.