Generalizability theory and the fair and valid assessment of linguistic minorities

Educational Research and Evaluation

Volumn 19, Issue 2-3, 2013, Pages 245-263

  Solano Flores, Guillermo ^a   Li, Min ^b  

^a UNIVERSITY OF COLORADO   (United States)

^b UNIVERSITY OF WASHINGTON   (United States)

Author keywords

emergent bilinguals; English language learners; fairness; generalizability theory; measurement error; validity

Indexed keywords

EID: 84875830545     PISSN: 13803611     EISSN: 17444187     Source Type: Journal    
DOI: 10.1080/13803611.2013.767632     Document Type: Article

References (61)

1. Abedi, J. 2007. "English language proficiency assessment and accountability under the NCLB Title III: An overview". In English language proficiency in the nation: Current status and future practice, Edited by: Abedi, J. 3-10. Davis, CA: University of California, Davis.
2. Abedi, J. 2008. Classification system for English language learners: Issues and recommendations. Educational Measurement: Issues and Practice, 27(3): 17-31. (doi:10.1111/j.1745-3992.2008.00125.x)
3. Abedi, J., Hofstetter, C. H. and Lord, C. 2004. Assessment accommodations for English language learners: Implications for policy-based empirical research. Review of Educational Research, 74: 1-28. (doi:10.3102/00346543074001001)
4. Abedi, J., Lord, C., Hofstetter, C. and Baker, E. 2001. Impact of accommodation strategies on English language learners' test performance. Educational Measurement: Issues and Practice, 19(3): 16-26. (doi:10.1111/j.1745-3992.2000.tb00034.x)
5. Bachman, L. F., Lynch, B. K. and Mason, M. 1995. Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing, 12: 239-257. (doi:10.1177/026553229501200206)
6. Backhoff, E., Solano-Flores, G. and Contreras-Niño, L. A. 2012, October. Aplicación piloto de 30 reactivos del Excale-00/PM en poblaciones mayas: Quinto reporte de investigación. Proyecto: "Pertinencia de la Traducción y Adaptación de las Pruebas Excale de Preescolar a la Lengua Maya" [Analysis of a 30-item test administered to pre-school Mayan students: Fifth research report. Project: "Feasibility of translating and adapting EXCALE pre-school tests into the Mayan language"]. Submitted to the National Institute for Educational Evaluation, Universidad Autónoma de Baja California, Ensenada, Baja California, Mexico
7. Baker, C. 2001. Foundations of bilingual education and bilingualism, 4, Clevedon, UK: Multilingual Matters.
8. Bialystok, E. 2001. Bilingualism in development: Language, literacy, and cognition., Cambridge, UK: Cambridge University Press.
9. Bialystok, E., Craik, F. I.M., Green, D. W. and Gollan, T. H. 2009. Bilingual minds. Psychological Science in the Public Interest, 10: 80-129. (doi:10.1177/1529100610387084)
10. Brennan, R. L. 2000. (Mis) conception about generalizability theory. Educational Measurement: Issues and Practice, 19(1): 5-10. (doi:10.1111/j.1745-3992.2000.tb00017.x)
11. Brennan, R. L. 2001. Generalizability theory, New York, NY: Springer Verlag.
12. Brennan, R. L., Gao, X. and Colton, D. A. 1995. Generalizability analyses of Work Keys listening and writing tests. Educational and Psychological Measurement, 55: 157-176. (doi:10.1177/0013164495055002001)
13. Brennan, R. L., Harris, D. J. and Hanson, B. A. 1987. The bootstrap and other procedures for examining the variability of estimated variance components in testing contexts, Iowa City, IA: ACT. (ACT Research Report 87-7
14. Brown, H. D. 2007. Principles of language learning and teaching, 5, Boston, MA: Addison Wesley & Longman.
15. Cardinet, J., Tourneur, Y. and Allal, L. 1976. The symmetry of generalizability theory: Applications to educational measurement. Journal of Educational Measurement, 13: 119-135. (doi:10.1111/j.1745-3984.1976.tb00003.x)
16. Cronbach, L. J., Gleser, G. C., Nanda, H. and Rajaratnam, N. 1972. The dependability of behavioral measurements, New York, NY: Wiley.
17. Cronbach, L. J., Linn, R. L., Brennan, R. L. and Haertel, E. H. 1997. Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57: 373-399. (doi:10.1177/0013164497057003001)
18. Crystal, D. 1997. The Cambridge encyclopedia of language, 2, Cambridge, UK: Cambridge University Press.
19. Cummins, J. 2000. Language, power and pedagogy, Clevedon, UK: Multilingual Matters.
20. Darling-Hammond, L. and Sykes, G. 2003. Wanted: A national teacher supply policy for education: The right way to meet the "Highly Qualified Teacher" challenge. Education Policy Analysis Archives, 11(33) Retrieved from http://epaa.asu.edu/epaa/v11n33/
21. García, O. and Kleifgen, J. A. 2010. Educating emergent bilinguals: Policies, programs, and practices of English language learners, New York, NY: Teachers College Press.
22. Haertel, E. H. 2006. "Reliability". In Educational measurement, 4, Edited by: Brennan, R. L. 65-110. Wesport, CT: American Council on Education.
23. Hakuta, K. 2000. How long does it take English learners to attain proficiency?, Santa Barbara, CA: Linguistic Minority Research Institute. (University of California Linguistic Minority Research Institute. Policy Reports, Retrieved from http://repositories.cdlib.org/lmri/pr/hakuta
24. Kachchaf, R. and Solano-Flores, G. 2012. Rater language background as a source of measurement error in the testing of English language learners. Applied Measurement in Education, 25: 167-172. (doi:10.1080/08957347.2012.660366)
25. Kane, M. T. 1982. A sampling model of validity. Applied Psychological Measurement, 6: 125-160. (doi:10.1177/014662168200600201)
26. Kopriva, R., Koran, J. and Hedgspeth, C. 2007. "Addressing the importance of systematically addressing student needs and test accommodations". In Large-scale assessment and accommodations: What works?, Edited by: Cahalan Laitusis, C. and Cook, L. L. 145-165. Arlington, VA: Council for Exceptional Children.
27. Lee, Y. 2006. Dependability of scores for a new ESL speaking assessment consisting of integrated and independent tasks. Language Testing, 23: 131-166. (doi:10.1191/0265532206lt325oa)
28. Li, D. and Brennan, R. L. 2007. A multi-group generalizability analysis of a large-scale reading comprehension test (Center for Advanced Studies in Measurement and Assessment Research. Report No. 25, August). College of Education Iowa City, IA, University of Iowa. Retrieved fromhttp://www.education.uiowa.edu/centers/docs/casma-research/25casmareport.pdf?sfvrsn=0
29. Lynch, B. K. and McNamara, T. F. 1998. Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing, 15: 158-180.
30. Oller, D. K., Pearson, B. Z. and Cobo-Lewis, A. B. 2007. Profile effects in early bilingual language and literacy. Applied Psycholinguistics, 28: 191-230. (doi:10.1017/S0142716407070117)
31. Pennock-Roman, M. and Rivera, C. 2011. Mean effects of test accommodations for ELLs and non-ELLs: A meta-analysis of experimental studies. Educational Measurement: Issues and Practice, 30(3): 10-28. (doi:10.1111/j.1745-3992.2011.00207.x)
32. Prosser, R. R., & Solano-Flores, G. (2010, April). Including English language learners in the process of test development: A study on instrument linguistic adaptation for cognitive validity. Paper presented at the Annual Conference of the National Council of Measurement in Education, Denver, CO.
33. Rivera, C., Collum, E., Willner, L. N. and Sia, J. K. Jr. 2006. "Study 1: An analysis of state assessment policies regarding the accommodation of English language learners". In State assessment policy and practice for English language learners: A national perspective, Edited by: Rivera, C. and Collum, E. 1-136. Mahwah, NJ: Erlbaum.
34. Ruiz-Primo, M. A., Baxter, G. P. and Shavelson, R. J. 1993. On the stability of performance assessments. Journal of Educational Measurement, 30: 41-53. (doi:10.1111/j.1745-3984.1993.tb00421.x)
35. Shavelson, R. J., Ruiz-Primo, M. A. and Wiley, E. W. 1999. Notes on sources of sampling variability in science performance assessments. Journal of Educational Measurement, 36: 61-71. (doi:10.1111/j.1745-3984.1999.tb00546.x)
36. Shavelson, R. J. and Webb, N. M. 1981. Generalizability theory: 1973-1980. British Journal of Mathematical and Statistical Psychology, 34: 133-166. (doi:10.1111/j.2044-8317.1981.tb00625.x)
37. Shavelson, R. J. and Webb, N. M. 1991. Generalizability theory: A primer, Newbury Park, CA: Sage.
38. Shavelson, R. J. and Webb, N. M. 2009. "Generalizability theory and its contribution to the discussion of the generalizability of research findings". In Generalizing from educational research, Edited by: Ercikan, K. and Roth, W. M. 13-32. New York, NY: Routledge.
39. Sireci, S. G., Li, S. and Scarpati, S. 2003. The effects of test accommodation on test performance: A review of the literature, Amherst, MA: University of Massachusetts at Amherst. (Center for Educational Assessment Research Report No. 485
40. Solano-Flores, G. 2006. Language, dialect, and register: Sociolinguistics and the estimation of measurement error in the testing of English-language learners. Teachers College Record, 108: 2354-2379. (doi:10.1111/j.1467-9620.2006.00785.x)
41. Solano-Flores, G. 2008. Who is given tests in what language by whom, when, and where? The need for probabilistic views of language in the testing of English language learners. Educational Researcher, 37: 189-199. (doi:10.3102/0013189X08319569)
42. Solano-Flores, G. 2009. "The testing of English language learners as a stochastic process: Population misspecification, measurement error, and overgeneralization". In Generalizing from educational research, Edited by: Ercikan, K. and Roth, W. M. 33-48. New York, NY: Routledge.
43. Solano-Flores, G. 2011a. Assessing the cultural validity of assessment practices: An introduction., Edited by: Basterra, M. R., Trumbull, E. and Solano-Flores, G. 3-21. New York, NY: Routledge. Cultural validity in assessment: Addressing linguistic and cultural diversity
44. Solano-Flores, G. (2011b, April). Development of illustrations as image supports for English language learners in large-scale testing: A report on the procedure for designing vignette illustrations. Paper presented at the CADRE ELL-STEM Roundtable session "Advancing English Language Learners in Science and Math: Realizing the Promise" at the Annual Meeting of the American Educational Research Association, New Orleans, LA.
45. Solano-Flores, G. and Gustafson, M. in press. "Assessment of English language learners: A critical, probabilistic, systemic view". In Improving large scale assessment in education: Theory, issues, and practice, Edited by: Simon, M., Ercikan, K. and Rousseau, M. 87-109. New York, NY: Taylor & Francis Routledge.
46. Solano-Flores, G., Lara, J., Sexton, U. and Navarrete, C. 2001. Testing English language learners: A sampler of student responses to science and mathematics test items, Washington, DC: Council of Chief State School Officers.
47. Solano-Flores, G. and Li, M. 2006. The use of generalizability (G) theory in the testing of linguistic minorities. Educational Measurement: Issues and Practice, 25(1): 13-22. (doi:10.1111/j.1745-3992.2006.00048.x)
48. Solano-Flores, G. and Li, M. 2008. Examining the dependability of academic achievement measures for English-Language Learners. Assessment for Effective Intervention, 33: 135-144. (doi:10.1177/1534508407313238)
49. Solano-Flores, G. and Li, M. 2009a. Generalizability of cognitive interview-based measures across cultural groups. Educational Measurement: Issues and Practice, 28(2): 9-18. (doi:10.1111/j.1745-3992.2009.00143.x)
50. Solano-Flores, G. and Li, M. 2009b. Language variation and score variation in the testing of English language learners, native Spanish speakers. Educational Assessment, 14: 1-15.
51. Solano-Flores, G., Li, M., Speroni, C., Rodriguez, J., Basterra, M., & Dovholuk, G. (2007, April). Comparing the properties of teacher-adapted and linguistically-simplified test items for English language learners. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL.
52. Solano-Flores, G., Nguyen-Le, K., & Wang, C. (2012, July). Linking generalizability and effectiveness in the evaluation of testing accommodations for English language learners. Paper presented at the symposium "Limits of Generalizing in Psychological Research" organized by Kadriye Ercikan, International Congress of Psychology in Cape Town, South Africa.
53. Solano-Flores, G. and Trumbull, E. 2008. "In what language should English language learners be tested? In". In Improving testing for English language learners, Edited by: Kopriva, R. J. 169-200. New York, NY: Routledge.
54. Valdés, G. and Figueroa, R. A. 1994. Bilingualism and testing: A special case of bias, Norwood, NJ: Ablex.
55. Van Weeren, J. and Theunissen, T. J. J. M. 1987. Testing pronunciation: An application of generalizability theory. Language Learning, 37: 109-122. (doi:10.1111/j.1467-1770.1968.tb01314.x)
56. Wardhaugh, R. 2002. An introduction to sociolinguistics, (4th ed., Malden: MA: Blackwell.
57. Webb, N. M., Shavelson, R. J. and Haertel, E. H. 2007. "Reliability coefficients and generalizability theory". In Handbook of statistics, Volume 26: Psychometrics, Edited by: Rao, C. R. and Sinharay, S. 81-124. Amsterdam, The Netherlands: Elsevier.
58. Wei, L. 2000. "Dimensions of bilingualism". In The bilingualism reader, Edited by: Wei, L. 3-25. New York, NY: Routledge.
59. Wiley, E. W. 2001. Bootstrap strategies for variance component estimation: Theoretical and empirical results, Stanford: Stanford University. CA., (Unpublished doctoral dissertation
60. Wolfram, W., Adger, C. T. and Christian, D. 1999. Dialects in schools and communities, Mahwah, NJ: Lawrence Erlbaum Associates.
61. Young, J. W. and King, T. C. 2008. Testing accommodations for English language learners: A review of state and district policies, New York, NY: The College Board.