Rater Language Background as a Source of Measurement Error in the Testing of English Language Learners

Applied Measurement in Education

Volumn 25, Issue 2, 2012, Pages 162-177

  Kachchaf, Rachel ^a   Solano Flores, Guillermo ^b  

^a TERC   (United States)

^b UNIVERSITY OF COLORADO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84859630872     PISSN: 08957347     EISSN: None     Source Type: Journal    
DOI: 10.1080/08957347.2012.660366     Document Type: Article

References (47)

1. Abedi, J. 2007. "English language proficiency assessment and accountability under NCLB Title III: An overview". In English language proficiency in the nation, Edited by: Abedi, J. 3-10. Davis, CA: University of California, Davis.
2. Abedi, J. and Gándara, P. 2006. Performance of English language learners as a subgroup in large-scale assessment: Interaction of research and policy. Educational Measurement: Issues and Practice, 25(4): 36-46.
3. American Educational Research Association, American Psychological Association and National Council on Measurement in Education. 1999. Standards for educational and psychological testing, Washington, DC: American Educational Research Association.
4. August, D. and Hakuta, K. 1997. Improving schooling for language minority students: A research agenda, Washington, DC: National Academy Press. Committee on Developing a Research Agenda on the Education of Limited-English-proficient and Bilingual Students, Board on Children, Youth and Families, Commission on Behavioral and Social Sciences and Education, National Research Council, Institute of Medicine
5. Bachman, L. F. 2002. Alternative interpretations of alternative assessments: Some validity issues in educational performance assessments. Educational Measurement: Issues and Practice, 21(3): 5-18.
6. Bachman, L. F., Lynch, B. K. and Mason, M. 1995. Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing, 12(2): 238-257.
7. Baxter, G. P., Shavelson, R. J., Goldman, S. R. and Pine, J. 1992. Evaluation of procedure based scoring for hands-on science assessment. Journal of Educational Measurement, 29(1): 1-17.
8. Bialystok, E. 1997. Effects of bilingualism and biletracy on children's emerging concepts of print. Developmental Psychology, 3(3): 429-440.
9. Brennan, R. L. 1992. Elements of generalizability theory, Iowa City, IA: The American College Testing Program.
10. Brennan, R. L. 2001. Generalizability theory, New York, NY: Springer-Verlag.
11. Brown, A. 1995. The effect of rater variables in the development of an occupation-specific language performance test. Language Testing, 12(1): 1-15.
12. Coulmas, F. 2005. Sociolinguistics: The study of speakers' choices, New York, NY: Cambridge University Press.
13. Cronbach, L. J., Gleser, G. C., Nanda, H. and Rajaratnam, N. 1972. The dependability of behavioral measurements, New York, NY: Wiley.
14. Cronbach, L. J., Linn, R. L., Brennan, R. L and Haertel, E. H. 1997. Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57(3): 373-399.
15. Cummins, J. 2001. "Assessment and intervention with culturally and linguistically diverse learners". In Literacy assessment of second language learners, Edited by: Hurley, S. R. and Tinajero, J. V. 115-129. Boston, MA: Allyn and Bacon.
16. Eckes, T. 2005. Examining rater effects in TestDaF Writing and Speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly, 2(3): 197-221.
17. Engelhard, G. Jr. 2002. "Monitoring raters in performance assessments". In Large-scale assessment programs for all students: Validity, technical adequacy, and implementation, Edited by: Tindal, G. and Haladyna, T. M. 261-287. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
18. Fitzpatrick, R. and Morrison, E. J. 1971. "Performance and product evaluation". In Educational measurement, Edited by: Thorndike, R. L. 237-270. Washington, DC: American Council on Education.
19. Fry, R. 2007. How far behind are English language learners in math and reading, Washington, DC: Pew Hispanic Center.
20. Koretz, D., Stecher, B., Klein, S. and McCaffrey, D. 1994. The Vermont portfolio assessment program: Findings and implications. Educational Measurement: Issues and Practice, 13(3): 5-16.
21. Mackey, W. 1962. The description of bilingualism. Canadian Journal of Linguistics, 7: 51-85.
22. National Assessment of Educational Progress. 1996. Mathematics items public release, Washington, DC: Author.
23. National Assessment of Educational Progress. 2000. Mathematics items public release, Washington, DC: Author.
24. National Center for Education Statistics. (2009). NAEP inclusion policy (updated April 20, 2009). Retrieved from http://nces.ed.gov/nationsreportcard/about/inclusion.asp#accom_table
25. Pierce, C. A., Block, R. A. and Aguinis, H. 2004. Cautionary note on reporting eta-squared values from multifactor ANOVA designs. Educational and Psychological Measurement, 64(6): 916-924.
26. Porter, S. G. and Vega, G. 2007. "Overview of existing English language proficiency tests". In English language proficiency in the nation, Edited by: Abedi, J. 93-102. Davis, CA: University of California, Davis.
27. Rivera, C. and Collum, E. 2006. State assessment policy and practice for English language learners: A national perspective, Mahwah, NJ: Lawrence Erlbaum Associates.
28. Ruiz-Primo, M. A., Baxter, G. P. and Shavelson, R. J. 1993. "On the stability of performance assessments". In Journal of Educational Measurement Vol. 30, 41-53.
29. Shavelson, R. J., Baxter, G. P. and Pine, J. 1992. Performance assessments: Political rhetoric and measurement reality. Educational Researcher, 21(4): 22-27.
30. Shavelson, R. J. and Webb, N. M. 1991. Generalizability theory: A primer, Newbury Park, CA: Sage.
31. Shohamy, E., Gordon, C. M. and Kraemer, R. 1992. The effect of raters' background and training on the reliability of direct writing tests. The Modern Language Journal, 76(1): 27-33.
32. Sleeter, C. E. 2001. Preparing teachers for culturally diverse schools: Research and the overwhelming presence of whiteness. Journal of Teacher Education, 52: 94-106.
33. Solano-Flores, G. 2006. Language, dialect, and register: Sociolinguistics and the estimation of measurement error in the testing of English-language learners. Teachers College Record., 108(11): 2354-2379.
34. Solano-Flores, G. 2008. Who is given tests in what language by whom, when, and where? The need for probabilistic views of language in the testing of English language learners. Educational Researcher, 37(4): 189-199.
35. Solano-Flores, G. 2009. "The testing of English language learners as a stochastic process: Population misspecification, measurement error, and overgeneralization". In Generalizing from educational research (pp. 33-45), Edited by: Ercikan, K. and Roth, W. M. New York, NY: Routledge.
36. Solano-Flores, G., Jovanovic, J., Shavelson, R. J. and Bachman, M. 1999. On the development and evaluation of a shell for generating science performance assessments. International Journal of Science Education, 21(3): 293-315.
37. Solano-Flores, G. and Li, M. 2006. The use of generalizability (G) theory in the testing of linguistic minorities. Educational Measurement: Issues and Practice, 25(1): 13-22.
38. Solano-Flores, G. and Li, M. 2008. Examining the dependability of academic achievement measures for English-Language Learners. Assessment for Effective Intervention, 33(3): 135-144.
39. Solano-Flores, G. and Li, M. 2009. Language variation and score variation in the testing of English language learners, native Spanish Speakers. Educational Assessment, 14: 1-15.
40. Solano-Flores, G. and Trumbull, E. 2003. Examining language in context: The need for new research and practice paradigms in the testing of English-language learners. Educational Researcher, 32(2): 3-13.
41. Solano-Flores, G. and Trumbull, E. 2008. "In what language should English language learners be tested?". In Improving testing for English language learners, Edited by: Kopriva, R. J. 169-200. New York, NY: Routledge.
42. Stuhlmann, J., Daniel, C., Dellinger, A., Kenton Denny, R. and Powers, T. 1999. A generalizability study of the effects of training on teachers' abilities to rate children's writing using a rubric. Journal of Reading Psychology, 20(2): 107-127.
43. The University of Iowa Center for Advanced Studies in Measurement and Assessment. (2004). Computer programs. Retrieved from http://www.education.uiowa.edu/casma/computer_programs.htm
44. Valdés, G. and Figueroa, R. A. 1994. Bilingualism and testing: A special case of bias, Norwood, NJ: Albex.
45. Weigle, S. C. 1999. Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing, 6: 145-178.
46. Weigle, S. C., Bold, H. and Valsecchi, M. I. 2003. Effects of task and rater background on the evaluation of ESL student writing: A pilot study. TESOL Quarterly, 37(2): 345-354.
47. Wolfe, E. 2004. Identifying rater effects using latent trait models. Psychology Science, 46(1): 35-51.