Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for Fusion Model application to LanguEdge assessment

Language Testing

Volumn 26, Issue 1, 2009, Pages 031-073

  Jang, Eunice Eunhee ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Cognitive diagnostic assessment (CDA); Diagnostic feedback; Fusion Model; Language testing; LanguEdge assessment; Reading comprehension; Skill profiles; Validity arguments

Indexed keywords

EID: 68049122890     PISSN: 02655322     EISSN: 14770946     Source Type: Journal    
DOI: 10.1177/0265532208097336     Document Type: Article

References (59)

1. Afflerbach, P., & Johnston, P. (1984). Research methodology: On the use of verbal reports in reading research. Journal of Reading Behavior, 16(4), 307-322.
2. Alderson, J. C. (2000). Assessing reading. Cambridge: Cambridge University Press.
3. Alderson, J. C. (2005). Diagnosing foreign language proficiency: the interface between learning and assessment. London: Continuum.
4. Alderson, J. C., Clapham, C., & Wall, D. (1995). Language testing construction and evaluation. Cambridge: Cambridge University Press.
5. Alderson, J. C., & Huhta, A. (2005). The development of a suite of computerbased diagnostic tests based on the Common European Framework. Language Testing, 22, 301-320.
6. Alderson, J. C., & Lukmani, Y. (1989). Cognition and reading: Cognitive levels as embodied in test questions. Reading in a Foreign Language, 5(2), 253-270.
7. Anderson, N. J., Bachman, L., Perkins, K., & Cohen, A. (1991). An exploratory study into the construct validity of a reading comprehension test: Triangulation of data sources. Language Testing, 8, 41-66.
8. Bachman, L. F., Davidson, F., & Milanovic, M. (1996). The use of test method characteristics in the content analysis and design of EFL proficiency tests. Language Testing, 13(2), 125-150.
9. Bailey, K. M. (1999). Washback in language testing. (TOEFL Monograph Series Report. No. 15). Princeton, NJ: Educational Testing Service.
10. Berkoff, N. A. (1979). Reading skills in extended discourse in English as a Foreign Language. Journal of Research in Reading, 2(2), 95-107.
11. Black, P. J., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education, 5, 7-74.
12. Brown, J. D., & Hudson, T. (2002). Criterion-referenced language testing. Cambridge: Cambridge University Press.
13. Buck, G., & Tatsuoka, K. (1998). Application of the rule-space procedure to language testing: Examining attributes of a free response listening test. Language Testing, 15(2), 119-157.
14. Carver, R. P. (1992). What do standardized tests of reading comprehension measure in terms of efficiency, accuracy, and rate? Reading Research Quarterly, 27(4), 347-359.
15. Cohen, A. D. (1987). Studying language learning strategies: How do we get the information? In A. L. Wenden & J. Rubin (Eds.), Learner strategies in language learning (pp. 31-40). Englewood Cliffs, NJ: Prentice-Hall.
16. Cohen, A. D., & Hosenfeld, C. (1981). Some uses of mentalistic data in second-language research. Language Learning, 31(2), 285-313.
17. Cohen, A. D., & Upton, T. A. (2007). 'I want to go back to the text': Response strategies on the reading subtest of the new TOEFL. Language Testing, 24, 209-50.
18. Collins, A. (1990). Reformulating testing to measure learning and thinking. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 75-88). Hillsdale, NJ: Lawrence Erlbaum.
19. Cordon, L. A., & Day, J. D. (1996). Strategy use on standardized reading comprehension tests. Journal of Educational Psychology, 88(2), 288-295.
20. Davidson, F., & Lynch, B. K. (2002). Testcraft: A teacher's guide to writing and using language test specifications. New Haven and London: Yale University Press.
21. DiBello, L. V., Stout, W. F., & Roussos, L. A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 361-390). Hillsdale, NJ: Lawrence Erlbaum.
22. DiBello, L. V., Roussos, L. A., & Stout, W. (2007). Review of cognitive diagnostic assessment and a summary of psychometric models. In C. R. Rao 68 Cognitive diagnostic assessment of L2 reading comprehension & S. Sinharay (Eds.), Handbook of statistics, Vol. 26: Psychometrics (pp. 45-79). Elsevier Science B.V.: The Netherlands.
23. Embretson, S. (1991). A multidimensional latent trait model for measuring learning and change. Psychometrika, 37, 359-374.
24. Embretson, S. (1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3(3), 380-396.
25. Faerch, C., & Kasper, G. (Eds.). (1987). Introspection in second language research. Philadelphia, PA: Multilingual Matters.
26. Farr, R., Pritchard, R., & Smitten, B. (1990). A description of what happens when an examinee takes a multiple-choice reading comprehension test. Journal of Educational Measurement, 27(3), 209-226.
27. Fischer, G. H. (1973). The linear logistic test model as an instrument in educational research. Acta Psychologia, 37, 359-374.
28. Frederiksen, N., Glaser, R., Lesgold, A., & Shafto, M. (Eds.). (1990). Diagnostic monitoring of skill and knowledge acquisition. Hillsdale, NJ: Lawrence Erlbaum.
29. Gorin, J. S. (2007). Test construction and diagnostic testing. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 173-201). New York: Cambridge University Press.
30. Grabe, W. (1991). Current developments in second language reading research. TESOL Quarterly, 25(3), 375-406.
31. Grabe, W. (Ed.). (2000). Developments in reading research and their implications for computer-adaptive tests of reading. Cambridge: Cambridge University Press.
32. Greene, J. C., & Caracelli, V. J. (1997). Defining and describing the paradigm issue in mixed method evaluation. In J. C. Greene & V. J. Caracelli (Eds.), Advances in mixed methods evaluation: The challenges and benefits of integrating diverse paradigms. New directions for evaluation No. 74 (pp. 5-17). San Francisco, CA: Jossey-Bass.
33. Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign.
34. Hartz, S. M., & Roussos, L. A. (2005). The Fusion Model for skills diagnosis: Blending theory with practice. ETS Research Report, Educational Testing Service, Princeton, NJ.
35. Jang, E. E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign.
36. Jang, E. E. (2008). A framework for cognitive diagnostic assessment. In C. A. Chapelle, Y.-R. Chung, & J. Xu (Eds.), Towards adaptive CALL: Natural Language Processing for Diagnostic Language Assessment (pp. 117-131). Amer, IA: Iowa Scaner University.
37. Kasai, M. (1997). Application of the rule space model to the reading comprehension section of the test of English as a foreign language (TOEFL). Unpublished doctoral dissertation. University of Illinois, Urbana Champaign.
38. Kunnan, A. J., & Jang, E. E. (forthcoming). Diagnostic feedback in language assessment. In M. Log & C. Doughty (Eds.), Handbook of second and foreign language teaching. Walden, MA: Wiley-Blackwell.
39. Li, W. (1992). What is a test testing? An investigation of the agreement between students' test-taking processes and test constructors' presumption. Unpublished MA thesis. Lancaster University.
40. Lumley, T. (1993). The notion of subskills in reading comprehension tests: An EAP example. Language Testing, 10(3), 211-234.
41. Mislevy, R. J. (1995). Probability-based inference in cognitive diagnosis. In P. Nichols, S. Chipman, & R. J. Brennan (1995), Cognitively diagnostic assessment (pp. 43-71). Hillsdale, NJ: Lawrence Erlbaum.
42. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-67.
43. Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2004). A brief introduction to Evidence-Centered Design. CSE Technical Report 632, The National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Center for the Study of Evaluation (CSE). LA, CA: University of California, Los Angeles.
44. Nevo, N. (1989). Test-taking strategies on a multiple-choice test of reading comprehension. Language Learning, 6, 199-215.
45. Nichols, P. D. (1994). A framework for developing cognitively diagnostic assessments. Review of Educational Research, 64(4), 575-603.
46. Nicols, P. D., Chipman, S. F., & Brennan, R. L. (Eds.). (1995). Cognitively diagnostic assessment. Hillsdale, NJ: Lawrence Erlbaum.
47. Patz, R. J. & Junker, B. W. (1999). A straightforward approach to Markov Chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146-178.
48. Pellegrino, J. W., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.
49. Rost, D. (1993). Assessing the different components of reading comprehension: Fact or fiction? Language Testing, 10(1), 79-82.
50. Roussos, L., DiBello, L., Henson, R., Jang, E. E., & Templin, J. (in press). Skills diagnosis for education and psychology with IRT-based parametric latent class models. In S. E. Embretson & J. Roberts (Eds.), New directions in psychological measurement with model-based approaches. Washington, DC: American Psychological Association.
51. Sheehan, K. M. (1997). Atree-based approach to proficiency scaling and diagnostic assessment. Journal of Educational Measurement, 34(4), 333-352.
52. Shohamy, E. (1992). Beyond performance testing: A diagnostic feedback testing model for assessing foreign language learning. Modern Language Journal, 76(4), 513-521.
53. Snow, R. E., & Lohman, D. F. (1989). Implications of cognitive psychology for educational measurement. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 263-332). New York: Macmillan.
54. Spolsky, B. (1990). Social aspects of individual assessment. In J. de Jong & D. K. Stevenson (Eds.), Individualizing the assessment of language abilities (pp. 3-15). Avon: Multilingual Matters.
55. Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage Publications.
56. Tatsuoka, K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345-354.
57. Tatsuoka, K. (1990). Toward an integration of Item-Response Theory and cognitive error diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 453-488). Hillsdale, NJ: Lawrence Erlbaum.
58. Weir, C. J., & Porter, D. (1996). The multi-divisible or unitary nature of reading: The language tester between Scylla and Charybdis. Reading in a Foreign Language, 10, 1-19.
59. Weir, C. J., Huizhong, Y., & Yan, J. (2000). An empirical investigation of the componentiality of L2 reading in English for academic purposes. Cambridge: Cambridge University Press.