Probabilistic Approaches to Examining Linguistic Features of Test Items and Their Effect on the Performance of English Language Learners

Applied Measurement in Education

Volumn 27, Issue 4, 2014, Pages 236-247

  Solano Flores, Guillermo ^a  

^a UNIVERSITY OF COLORADO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84907464546     PISSN: 08957347     EISSN: None     Source Type: Journal    
DOI: 10.1080/08957347.2014.944308     Document Type: Article

References (74)

1. Abedi, J. (2006). Language issues in item development. In. S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 377–398). Mahwah, NJ: Lawrence Erlbaum Associates.
2. Abedi, J., Hofstetter, C. H., & Lord, C. (2004). Assessment accommodations for English language learners: Implications for policy-based empirical research. Review of Educational Research, 74(1), 1–28.
3. Abedi, J., & Lord, C. (2001). The language factor in mathematics tests. Applied Measurement in Education, 14(3), 219–234.
4. Abedi, J., Lord, C., Hofstetter, C., & Baker, E. (2001). Impact of accommodation strategies on English language learners’ test performance. Educational Measurement: Issues and Practice, 19(3), 16–26.
5. Allalouf, A. (2003). Revising translated differential item functioning items as a tool for improving cross-lingual assessment. Applied Measurement in Education, 16(1), 55–73.
6. Allalouf, A., Hambleton, R., & Sireci, S. (1999). Identifying the causes of DIF on verbal items. Journal of Educational Measurement, 36, 18–198.
7. American Educational Research Association, American Psychological Association, & National Council of Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: AERA.
8. Bachman, L. F., Lynch, B. K., & Mason, M. (1995). Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing, 12(2), 238–257.
9. Bailey, A. L., & Butler, F. A. (2003). An evidentiary framework for operationalizing academic language for broad application to K–12 education: A design document. (CSE Tech. Rep. No. 611). Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing.
10. Basterra, M. R. (2012). Cognition, culture, language, and assessment: How to select culturally valid assessments in the classroom. In M. R. Basterra, E. Trumbull, & G. Solano-Flores (Eds.). Cultural validity in assessment: Addressing linguistic and cultural diversity (pp. 72–95). New York, NY: Routledge.
11. Bialystok, E. (2001). Bilingualism in development: Language, literacy, and cognition. Cambridge, UK: Cambridge University Press.
12. Bod, R., Hay, J., & Jannedy, S. (Eds.) (2003). Probabilistic linguistics. Cambridge, MA: MIT.
13. Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer-Verlag.
14. Cronbach, L. J., Glaser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements. New York, NY: Wiley.
15. De Corte, E., & Verschaffel, L. (1991). Some factors influencing the solution of addition and subtraction word problems. In.K. Durkin & B. Shire (Eds.), Language in mathematical education: Research and practice (pp. 17–30), Buckingham, UK: Open University Press.
16. Ercikan, K. (2002). Disentangling sources of differential item functioning in multi-language assessments. International Journal of Testing, 2, 199–215.
17. Ercikan, K., Arim, R. G., Law, D. M., Lacroix, S., Gagnon, F., & Domene, J. F. (2010). Application of think-aloud protocols in examining sources of differential item functioning. Educational Measurement: Issues and Practice, 29, 24–35.
18. Ercikan, K., Roth, W.-M., Simon, M., Sandilands, D., & Lyons-Thomas, J. (2014). Inconsistencies in DIF Detection for Sub-Groups in Heterogeneous Language Groups. Applied Measurement in Education, 27, 273–285.
19. General Accounting Office. (2006). No Child Left Behind Act: Assistance from Education could help states better measure progress of students with limited English proficiency (GAO Report to Congressional Requesters, Report No. GAO-355-06-815). Retrieved from http://www.gao.gov/assets/260/250866.pdf
20. Gibson, J. J. (1977). The theory of affordances. In R. E. Shaw & J. Bransford (Eds.), Perceiving, acting, and knowing: Toward an ecological psychology (pp. 67–82). Hillsdale, NJ: Lawrence Erlbaum Associates.
21. Gierl, M. J. (2000). Construct equivalence on translated achievement tests. Canadian Journal of Education/Revue canadienne de l’education, 25(4), 280–296.
22. Gierl, M. J., & Khaliq, S. N. (2001). Identifying sources of differential item and bundle functioning on translated achievement tests: A confirmatory analysis. Journal of Educational Measurement, 38, 16–187.
23. Gries, S. T. (2013). Sources of variability relevant to the cognitive sociolinguist, and corpus- as well as psycholinguistic methods and notions to handle them. Journal of Pragmatics, 52, 5–16.
24. Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. London, UK: Hodder Arnold.
25. Halliday, M. A. K. (2005). Computational and quantitative studies. New York, NY: Continuum.
26. Hambleton, R. K. (2005). Issues, designs, and technical guidelines for adapting tests into multiple languages and cultures. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 3–38). Mahwah, NJ: Lawrence Erlbaum Associates.
27. Hambleton, R. K., & Jones, R. W. (1992). Comparison of empirical and judgmental methods for detecting differential item functioning. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
28. Hudicourt-Barnes, J., Noble, T., O’Connor, M. C., Rosebery, A. S., Suarez, C., Warren, B., & Wright, C. G. (2008). Making sense of children’s performance on achievement tests: The case of the 5th grade science MCAS. Symposium paper presented at the annual meeting of the American Educational Research Association. New York, NY.
29. Kachchaf, R. R. (2011). Exploring problem solving strategies on multiple-choice science items comparing native Spanish-speaking English language learners and mainstream monolinguals. Doctoral dissertation. University of Colorado at Boulder, Department of Education.
30. Kachchaf, R. R., & Solano-Flores, G. (2012). Rater language background as a source of measurement error in the testing of English language learners. Applied Measurement in Education, 25, 167–172.
31. Kopriva, R. J. (2000). Ensuring accuracy in testing for English language learners: A practical guide for assessment development. Washington, DC: Council of Chief State School Officers.
32. Kopriva, R. J. (2008). Improving testing for English language learners: A comprehensive approach to designing, building, implementing and interpreting better academic assessments. New York, NY: Routledge.
33. Kopriva, R. J., Emick, J. E., Hipólito-Delgado, C. P., & Cameron, C. A. (2007). Do proper accommodation assignments make a difference? Examining the impact of improved decision making on scores for English language learners. Educational Measurement: Issues and Practice, 26(3), 11–20.
34. Kopriva, R., J., & Koran, J. (2008). Proper assignment of accommodations to individual students. In R. J. Kopriva (Ed.), Improving testing for English language learners: A comprehensive approach to designing, building, implementing and interpreting better academic assessments (pp. 217–254). New York, NY: Routledge.
35. Martiniello, M. (2009). Linguistic complexity, schematic representations, and differential item functioning for English language learners in math tests. Educational Assessment, 14(3), 160–179.
36. Menezes, V. (2011). Affordances for language learning beyond the classroom. In P. Benson & H. Renders (Eds.), Beyond the language classroom (pp. 59–71). Basingstoke, UK: Palgrave Macmillan.
37. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed.) (pp. 147–200). New York, NY: Macmillan Publishing.
38. Mislevy, R. J., Wilson, M. R., Ercikan, K., & Chudowsky, N. (2003). Psychometric principles in student assessment. In T. Kellaghan & D. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 489–531). Dordrecht, the Netherlands: Kluwer Academic Press. (Eds.).
39. Nguyen-Le, K. (2010). Personal and formal backgrounds as factors which influence linguistic and cultural competency in the teaching of mathematics. Doctoral dissertation, Educational Equity and Cultural Diversity Program, University of Colorado at Boulder, 2010.
40. Noble, T., Suarez, C., Rosebery, A., O’Connor, M. C., Warren, B., & Hudicourt-Barnes, J. (2012). “I never thought of it as freezing”: How students answer questions on large-scale science tests and what they know about science. Journal of Research in Science Teaching, 49(6), 778–803.
41. Norman, D. A. (1990). The design of everyday things. New York, NY: Doubleday.
42. O’Connor, M. C. (1998). Language socialization in the mathematics classroom: Discourse practices and mathematical thinking. In M. Lampert & M. Blunk (Eds.), Talking mathematics: Studies of teaching and learning in school (pp. 17–55). New York, NY: Cambridge University Press.
43. Penock-Roman, M., & Rivera, C. (2011). Mean effects of test accommodations for ELLs and non-ELLs: A meta-analysis of experimental studies. Educational Measurement: Issues and Practice, 30(3), 10–28.
44. Prosser, R. R., & Solano-Flores, G. (2010). Including English language learners in the process of test development: A study on instrument linguistic adaptation for cognitive validity. Paper presented at the annual conference of the National Council of Measurement in Education, Denver, CO.
45. Rivera, C., & Stansfield, C. W. (2004). The effect of linguistic simplification of science test items on score comparability. Educational Assessment, 9(3&4), 79–105.
46. Sato, E., Rabinowitz, S., Gallagher, C., & Huang, C.-W. (2010). Accommodations for English language learner students: The effect of linguistic modification of math test item sets (NCEE 2009-4079). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
47. Scarcella, R. C. (2003). Academic English: A conceptual framework (Rep. No. 2003-1). Santa Barbara, CA: University of California, Linguistic Minority Research Institute.
48. Shaftel, J., Belton-Kocher, E., Glasnapp, D., & Poggio, G. (2006). The impact of language characteristics in mathematics test items on the performance of English language learners and students with disabilities. Educational Assessment, 11(2), 105–126.
49. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
50. Sireci, S. G., Li, S., & Scarpati, S. (2003). The effects of test accommodation on test performance: A review of the literature. (Center for Educational Assessment, Research Report No. 485). Amherst, MA: University of Massachusetts at Amherst.
51. Solano-Flores, G. (2006). Language, dialect, and register: Sociolinguistics and the estimation of measurement error in the testing of English-language learners. Teachers College Record. 108(11), 2354–2379.
52. Solano-Flores, G. (2008). Who is given tests in what language by whom, when, and where? The need for probabilistic views of language in the testing of English language learners. Educational Researcher, 37(4), 189–199.
53. Solano-Flores, G. (2009). The testing of English language learners as a stochastic process: Population misspecification, measurement error, and overgeneralization. In K. Ercikan & W. M. Roth (Eds.), Generalizing from educational research (pp. 33–48). New York, NY: Routledge.
54. Solano-Flores, G. (2010). Function and form in research on language and mathematics education. In Moschkovich, J. (Ed.), Language and mathematics in education: Multiple perspectives and directions for research (pp. 113–149). Charlotte, NC: Information Age Publishing.
55. Solano-Flores, G. (2012). Translation accommodations framework for testing English language learners in mathematics. (Developed for Smarter Balanced Assessment Consortium [SBAC]). Retrieved from http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/09/Translation-Accommodations-Framework-for-Testing-ELL-Math.pdf
56. Solano-Flores, G., Backhoff, E., & Contreras-Niño, L. A. (2009). Theory of test translation error. International Journal of Testing, 9, 78–91.
57. Solano-Flores, G., Contreras-Niño, L. A., & Backhoff, E. (2012). The measurement of translation error in PISA-2006 items: An application of the theory of test translation error. In M. Prenzel, M. Kobarg, K. Schöps, & S. Rönnebeck (Eds.), Research in the context of the programme for international student assessment (pp. 71–85). Germany: Springer Verlag.
58. Solano-Flores, G., & Gustafson, M. (2012). Assessment of English language learners: A critical, probabilistic, systemic view. In M. Simon, K. Ercikan, & M. Rousseau (Eds.), Improving large scale assessment in education: Theory, issues, and practice. (pp. 87–109). New York, NY: Routledge.
59. Solano-Flores, G., & Li, M. (2006). The use of generalizability (G) theory in the testing of linguistic minorities. Educational Measurement: Issues and Practice 25(1), 13–22.
60. Solano-Flores, G., & Li, M. (2008). Examining the dependability of academic achievement measures for English-language learners. Assessment for Effective Intervention, 33(3), 135–144.
61. Solano-Flores, G., & Li, M. (2009). Language variation and score variation in the testing of English language learners, native Spanish speakers. Educational Assessment, 14, 1–15.
62. Solano-Flores, G., & Li, M. (2013). Generalizability theory and the fair and valid assessment of linguistic minorities. Educational Research and Evaluation, 19(2–3), 245–263.
63. Solano-Flores, G., Li, M., Speroni, C., Rodriguez, J., Basterra, M., & Dovholuk, G. (2007). Comparing the properties of teacher-adapted and linguistically-simplified test items for English language learners. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
64. Solano-Flores, G., & Trumbull, E. (2003). Examining language in context: The need for new research and practice paradigms in the testing of English-language learners. Educational Researcher, 32(2), 3–13.
65. Solano-Flores, G., & Trumbull, E. (2008). In what language should English language learners be tested? In R. J. Kopriva (Ed.), Improving testing for English language learners (pp. 169–200). New York, NY: Routledge.
66. Stansfield, C. W. (2003). Test translation and adaptation in public education in the USA. Language Testing, 20(2), 189–207.
67. Stansfield, C. W., & Bowles, M. (2006). Study 2: Test translation and state assessment policies for English language learners. In C. Rivera & E. Collum (Eds.), State assessment policy and practice for English language learners: A national perspective. (pp. 1–173). Mahwah, NJ: Lawrence Erlbaum Associates.
68. Steyvers, M., & Griffiths, T. (2007). Probabilistic topic models. In T. K. Landauer, D. S. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of latent semantic analysis (pp. 427–448). Mahwah, NJ: Lawrence Erlbaum Associates.
69. van Lier, L. (2004). The ecology and semiotics of language learning: A sociocultural perspective. Dordrecht, Netherlands: Kluwer Academic Publishers.
70. Wellington, J., & Osborne, J. (2001). Language and literacy in science education. Buckingham, UK: Open University Press.
71. Winter, P. C., Kopriva, R. J., Chen, C.-S., & Emick, J. E. (2006). Exploring individual and item factors that affect assessment validity for diverse learners: Results from a large-scale cognitive lab. Learning and Individual Differences, 16(4), 267–276.
72. Wolf, M. K., Kao, J., Herman, J., Bachman, L. F., Bailey, A., Bachman, P. L., … Chang. S. M. (2008). Issues in assessing English language learners: English language proficiency measures and accommodation uses: Literature review (Part 1 of 3). (CRESST Report 731). Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing and Graduate School of Education & Information Studies. University of California.
73. Wolf, M. K., & Leon, S. (2009). An investigation of the language demands in content assessments for English language learners. Educational Assessment, 14(3), 139–159.
74. Wolfram, W., Adger, C. T., & Christian, D. (1999). Dialects in schools and communities. Mahwah, NJ: Lawrence Erlbaum Associates.