Differential functioning of reading subskills on the OSSLT for L1 and ELL students: A multidimensionality model-based DBF/DIF approach

Language Learning

Volumn 59, Issue 4, 2009, Pages 825-865

  Kim, Youn Hee ^a   Jang, Eunice Eunhee ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Differential bundle functioning (DBF); Differential item functioning (DIF); L1 and ELL students; Literacy assessment; Multidimensionality based DBF DIF analysis; Ontario Secondary School Literacy Test (OSSLT); Reading comprehension; Test bias

Indexed keywords

EID: 70450270607     PISSN: 00238333     EISSN: 14679922     Source Type: Journal    
DOI: 10.1111/j.1467-9922.2009.00527.x     Document Type: Article

References (75)

1. Abbott, M. L. (2007). A confirmatory approach to differential item functioning on an ESL reading assessment. Language Testing, 24, 7 36.
2. Abedi, J. (2004). The no child left behind act and English language learners: Assessment and accountability issues. Educational Researcher, 33, 4 14.
3. Abedi, J., & Hejri, P. (2004). Accommodations for students with limited English proficiency in the national assessment of educational progress. Applied Measurement in Education, 17, 371 392.
4. Abedi, J., Leon, S., & Mirocha, J. (2003). Impact of students' language background on content-based assessment: Analyses of extant data (CSE Tech. Rep. No. 603). Los Angeles : University of California, National Center for Research on Evaluation, Standards, and Student Testing.
5. Ackerman, T. A. (1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29, 67 91.
6. Ackerman, T. A., Gierl, M. J., & Walker, C. M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement, Issues and Practice, 22, 37 53.
7. Alderson, J. C. (1990a). Testing reading comprehension skills (Part one). Reading in a Foreign Language, 6, 425 438.
8. Alderson, J. C. (1990b). Testing reading comprehension skills (Part two): Getting students to talk about taking a reading test (A pilot study). Reading in a Foreign Language, 7, 465 503.
9. Alderson, J. C. (2000). Assessing reading. Cambridge : Cambridge University Press.
10. Alderson, J. C., & Lukmani, Y. (1989). Cognition and reading: Cognitive levels as embodied in test questions. Reading in a Foreign Language, 5, 253 270.
11. Alderson, J. C., & Urquhart, A. H. (1983). The effect of student background discipline on comprehension: A pilot study. In A. Hughes & D. Porter (Ed. Current developments in language testing (pp. 121 127). London : Academic Press.
12. Alderson, J. C., & Urquhart, A. H. (1984). Student discipline and reading comprehension. In T. Culhane (Ed. Practice and problems in language testing. University of Essex Occasional Papers.
13. Alderson, J. C., & Urquhart, A. H. (1985). The effect of students' academic discipline on their performance on ESP reading tests. Language Testing, 2, 192 204.
14. AERA/APA/NCME (American Educational Research Association/American Psychological Association/National Council on Measurement in Education) 1999). Standards for educational psychological testing. Washington, DC : Author.
15. Anderson, R. C., & Freebody, P. (1981). Vocabulary knowledge. In J. T. Guthrie (Eds. Comprehension and teaching: Research reviews (pp. 77 117). Newark, DE : International Reading Association.
16. Block, E. L. (1992). See how they read: Comprehension monitoring of L1 and L2 readers. TESOL Quarterly, 26, 319 343.
17. Bullock, C. (2000). Ontario cuts to English as a Second Language. Retrieved January 30, 2008, from http://www.caledoninst.org/Publications/PDF/1- 894159-98-5.pdf
18. Carrell, P. L. (1983). Three components of background knowledge in reading comprehension. Language Learning, 33, 183 203.
19. Carrell, P. L. (1989). Metacognitive awareness and second language reading. Modern Language Journal, 73, 121 134.
20. Chang, H.-H., Mazzeo, J., & Roussos, L. (1996). Detecting DIF for polytomously scored items: An adaptation of the SIBTEST procedure. Journal of Educational Measurement, 33, 333 353. (Pubitemid 126318107)
21. Chen, Z., & Henning, G. (1985). Linguistic and cultural bias in language proficiency tests. Language Testing, 2, 155 163.
22. Cheng, L., Klinger, D., & Zheng, Y. (2007). The challenges of the Ontario Secondary School Literacy Test for second language students. Language Testing, 24, 185 208.
23. Clapham, C. (1998). The effect of language proficiency and background knowledge on EAP students' reading comprehension. In A. J. Kunnan (Ed. Validation in language assessment (pp. 141 168). Mahwah, NJ : Lawrence Erlbaum.
24. Cohen, A., Glasman, H., Rosenbaum-Cohen, P. R., Ferrara, J., & Fine, J. (1979). Reading English for specialized purposes: Discourse analysis and the use of student informants. TESOL Quarterly, 13, 551 564.
25. Cooper, M. (1984). Linguistic competence of practiced and unpracticed non-native readers of English. In J. C. Alderson & A. H. Urquhart (Eds. Reading in a foreign language (pp. 122 138). London : Longman.
26. Cummins, J. (1981). Age on arrival and immigrant second language learning in Canada: A reassessment. Applied Linguistics, 2, l32 l49.
27. Cummins, J. (1984). Bilingualism and special education: Issues in assessment and pedagogy. Clevedon, UK : Multilingual Matters.
28. Cummins, J. (2001). Negotiating identities: Education for empowerment in a diverse society. Los Angeles : California Association for Bilingual Education.
29. Davis, F. B. (1968). Research in comprehension in reading. Reading Research Quarterly, 3, 499 545.
30. Davis, F. B. (1972). Psychometric research on comprehension in reading. Reading Research Quarterly, 6, 628 678.
31. Dorans, N. J., & Holland, P. W. (1993). DIF detection and description: Mantel-Haenszel and standardization. In P. W. Holland & H. Wainer (Eds. Differential item functioning (pp. 35 66). Hillsdale, NJ : Lawrence Earlbaum.
32. Douglas, J., Kim, H.-R., Roussos, L., Stout, W., & Zhang, J. (1999). LSAT dimensionality analysis for December 1991, June 1992, and October 1992 administrations (Law School Admission Council Statistical Report 95-05). Newton, PA : LSAT.
33. Douglas, J., Roussos, L., & Stout, W. (1996). Item bundle DIF hypothesis testing: Identifying suspect bundles and assessing their differential functioning. Journal of Educational Measurement, 33, 465 484.
34. Elder, E. (1996). The effect of language background on "foreign" language test performance: The case of Chinese, Italian, and modern Greek. Language Learning, 46, 233 282.
35. EQAO (Education Quality and Accountability Office) 2006). Ontario secondary school literacy test (2005-2006): Provincial report. Retrieved June 01, 2007, from http://www.eqao.com/pdf-e/06/06P028e.pdf
36. Ferne, T., & Rupp, A. (2007). A synthesis of 15 years of research on DIF in language testing: Methodological advances, challenges, and recommendations. Language Assessment Quarterly, 4, 113 148.
37. Fox, J., & Cheng, L. (2007). Did we take the same test? Differing accounts of the Ontario Secondary School Literacy Test by first and second language test-takers. Assessment in Education, 14, 9 26.
38. Geranpayeh, A., & Kunnan, A. (2007). Differential item functioning in terms of age in the certificate in advanced English examination. Language Assessment Quarterly, 4, 190 220.
39. Gierl, M. J. (2005). Using dimensionality-based DIF analyses to identify and interpret constructs that elicit group differences. Educational Measurement: Issues and Practice, 24, 3 14.
40. Gierl, M. J., Bisanz, J., Bisanz, G., & Boughton, K. (2003). Identifying content and cognitive skills that produce gender differences in mathematics: A demonstration of the DIF analysis paradigm. Journal of Educational Measurement, 40, 281 306.
41. Gierl, M., Bisanz, J., Bisanz, G., Boughton, K., & Khaliq, S. (2001, April). Illustrating the procedures and content reviews for identifying translation DIF. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Montreal.
42. Gierl, M. J., & Khaliq, S. N. (2001). Identifying sources of differential item and bundle functioning on translated achievement tests: A confirmatory analysis. Journal of Educational Measurement, 38, 164 187.
43. Grabe, W. (1991). Current developments in second language reading research. TESOL Quarterly, 25, 375 406.
44. Hofstetter, C. (2003). Contextual and mathematics accommodation test effects for English-language learners. Applied Measurement in Education, 16, 159 188.
45. Hsueh-chao, M. H., & Nation, P. (2000). Unknown vocabulary density and reading comprehension. Reading in a Foreign Language, 13, 403 430.
46. Kim, M. (2001). Detecting DIF across the different language groups in a speaking test. Language Testing, 18, 89 114.
47. Koenig, J., & Bachman, L. (2004). Keeping score for all: The effects of inclusion and accommodation policies on large-scale educational assessments. Washington, DC : National Academies Press.
48. Laufer, B. (1992). How much lexis is necessary for reading comprehension In H. Béjoint & P. Arnaud (Eds. Vocabulary and applied linguistics (pp. 126 132). London : Macmillan.
49. Lee, Y.-W., Breland, H., & Muraki, E. (2005). Comparability of TOEFL CBT writing prompts for different native language groups. International Journal of Testing, 5, 131 158.
50. Li, H., & Stout, W. (1996). A new procedure for detection of crossing DIF. Psychometrika, 61, 647 677. (Pubitemid 126345242)
51. Lumley, T. (1993). The notion of subskills in reading comprehension tests: An EAP example. Language Testing, 10, 211 234.
52. Mezynski, K. (1983). Issues concerning the acquisition of knowledge: Effects of vocabulary training on reading comprehension. Review of Educational Research, 53, 253 279.
53. O'Neill, K. A., & McPeek, W. M. (1993). Item and test characteristics that are associated with differential item functioning. In P. W. Holland & H. Wainer (Eds. Differential item functioning (pp. 255 276). Hillsdale, NJ : Lawrence Erlbaum.
54. Ontario Ministry of Education and Training 1999). The Ontario Curriculum Grade 9 to 12: English as a Second Language and English Literacy Development. Toronto : Ontario Ministry of Education and Training.
55. Pae, T.-I. (2004). DIF for examinees with different academic backgrounds. Language Testing, 21, 53 73.
56. Pettit, N. T., & Cockriel, I. W. (1974). A factor study of the literal reading comprehension test and the inferential reading comprehension test. Journal of Reading Behavior, 6, 63 75.
57. Ramsay, J. (1989). TESTGRAF98 Computer software and manual Retrieved June, 01, 2007, from http://www.psych.mcgill.ca/faculty/ramsay/TestGraf.html
58. Roever, C. (2007). DIF in the assessment of second language pragmatics. Language Assessment Quarterly, 4, 165 189.
59. Rost, D. H. (1993). Assessing different components of reading comprehension: Fact or fiction Language Testing, 10, 79 92.
60. Roussos, L., & Stout, W. (1996a). A multidimensionality-based DIF analysis paradigm. Applied Psychological Measurement, 20, 355 371. (Pubitemid 126332317)
61. Roussos, L., & Stout, W. (1996b). Simulation studies of the effects of small sample size and studied item parameters on SIBTEST and Mantel-Haenszel Type I error performance. Journal of Educational Measurement, 33, 215 230.
62. Roussos, L., & Stout, W. (2004). Differential item functioning analysis: Detecting DIF items and testing DIF hypotheses. In D. Kaplan (Eds. The Sage handbook for social sciences (pp. 107 115). Newbury Park, CA : Sage.
63. Roussos, L., Stout, W., & Marden, J. (1998). Using new proximity measures with hierarchical cluster analysis to detect multidimensionality. Journal of Educational Measurement, 35, 1 30.
64. Ryan, K., & Bachman, L. (1992). Differential item functioning on two tests of EFL proficiency. Language Testing, 9, 12 29.
65. Salager-Meyer, F. (1991). Reading expository prose at the post-secondary level: The influence of textual variables on L2 reading comprehension (a genre-based approach). Reading in a Foreign Language, 8, 645 662.
66. Sasaki, M. (1991). A comparison of two methods for detecting differential item functioning in an ESL placement test. Language Testing, 8, 95 111.
67. Shealy, R., & Stout, W. (1993a). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58, 159 194.
68. Shealy, R., & Stout, W. (1993b). An item response theory model for test bias and differential test functioning. In P. W. Holland & H. Wainer (Eds. Differential item functioning (pp. 197 240). Hillsdale, NJ : Lawrence Erlbaum.
69. Snetzler, S., & Qualls, A. L. (2000). Examination of Differential Item Functioning on a standardized achievement battery with limited English proficient students. Educational and Psychological Measurement, 60, 564 577.
70. Stout, W., Froelich, A., & Gao, F. (2001). Using resampling methods to produce an improved DIMTEST procedure. In A. Boomsma, M. A. J. van Duijn, & T. A. B. Snijders (Eds. Essays on item response theory (pp. 357 376). New York : Springer-Verlag.
71. Thorndike, R. L. (1974). Reading as reasoning. Reading Research Quarterly, 9, 135 147.
72. Uiterwijk, H., & Vallen, T. (2005). Linguistic sources of item bias for second generation immigrants in Dutch tests. Language Testing, 22, 211 234.
73. Vernon, P. E. (1962). The determinants of reading comprehension. Educational and Psychological Measurement, 22, 269 286.
74. Zhang, J., & Stout, W. (1999). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213 249.
75. Zumbo, B. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, ON : Directorate of Human Resources Research and Evaluation, Department of National Defense.