Promoting Validity in the Assessment of English Learners

Review of Research in Education

Volumn 39, Issue 1, 2015, Pages 215-252

  Sireci, Stephen G ^a   Faulkner Bond, Molly ^a  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84924765148     PISSN: 0091732X     EISSN: 19351038     Source Type: Journal    
DOI: 10.3102/0091732X14557003     Document Type: Article

References (110)

1. Abedi J.Language accommodation for large-scale assessment in science: Assessing English language learners. Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing, University of California Los Angeles; 2001:
2. English language proficiency assessment in the nation: Current status and future practice. Abedi J., ed. Davis: University of California, Davis, School of Education; 2007:
3. AbediJ.EwersN. (2013). Accommodations for English learners and students with disabilities: A research based decision algorithm. Retrieved from http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/08/Accomodations-for-under-represented-students.pdf
4. AbediJ.LinquantiR. (2012, January). Issues and opportunities in improving the quality of large scale assessment systems for ELLs. Paper presented the Understanding Language conference, Palo Alto, CA. Retrieved from http://ell.stanford.edu/publication/issues-and-opportunities-improving-quality-large-scale-assessment-systems-ells
5. Abedi J.,Lord C.,Hofstetter C.,Baker E.Impact of accommodation strategies on English language learners’ test performance.Educational Measurement: Issues and Practice. 2000;19 (3): 16-26
6. Abedi J.,Hofstetter C. H.,Lord C.Assessment accommodations for English language learners: Implications for policy-based empirical research.Review of Educational Research. 2004;74:1-28
7. Standards for educational and psychological testing. Washington, DC: American Educational Research Association.; 1999:
8. Standards for educational and psychological testing. Washington, DC: American Educational Research Association.; 2014:
9. American Institutes for Research. (2013). ELPA-to-ELPA Look Back. Washington DC. Retrieved from http://www.ode.state.or.us/teachlearn/standards/contentperformance/elpa-to-elpa_look-back_report_02052013.pdf
10. Anstrom K.,DiCerbo P.,Katz A.,Millet J.,Rivera C.A review of the literature on academic English: Implications for K-12 English language learners. Arlington, VA: The George Washington University Center for Equity and Excellence in Education; 2010:
11. Bailey A. L.,Butler F.The language demands of school: Putting academic English to the test. Bailey A. L., ed. New Haven, CT: Yale University Press; 2007:68-102.
12. Bailey A. L.,Carroll P. E.Assessment of English language learners in the era of new academic content standards.Review of Research in Education. 39:253-294
13. Bailey A. L.,Huang B. H.Do current English language development/proficiency standards reflect the English needed for success in school?.Language Testing. 2011;28:343-365
14. Bennett R. E.Cognitively based assessment of, for, and as learning (CBAL): A preliminary theory of action for summative and formative assessment.Measurement: Interdisciplinary Research and Perspectives. 2010;8:70-91
15. Bhola D. S.,Impara J. C.,Buckendahl C. W.Aligning tests with states’ content standards: Methods and issues.Educational Measurement: Issues and Practice. 2003;22 (3): 21-29
16. Bunch M. B.Testing English language learners under No Child Left Behind.Language Testing. 2011;28:323-341
17. California Department of Education. (2012, November). Overview of the California English language development standards and proficiency level descriptors. Retrieved from http://www.cde.ca.gov/sp/el/er/documents/sbeoverviewpld.pdf
18. Campbell D. T.,Fiske D. W.Convergent and discriminant validation by the multitrait-multimethod matrix.Psychological Bulletin. 1959;56:81-105
19. Carroll P.,Bailey A..Paper presented at the annual meeting of the National Council on Measurement in Education; 2014Philadelphia, PA; 2014.
20. Cook H. G.,Wilmes C.,Boals T.,Santos M.Issues in the development of annual measurable achievement objectives for WIDA Consortium states. Madison: Wisconsin Center for Education Research; 2008:
21. English language proficiency (ELP) standards with correspondences to K-12 English language arts (ELA), mathematics, and science practices, K-12 ELA standards, and 6-12 literacy standards. Washington, DC: Author; 2013:
22. Crocker L. M.,Miller D.,Franks E. A.Quantitative methods for assessing the fit between test and curriculum.Applied Measurement in Education. 1989;2:179-194
23. Crotts K.,Sireci S. G..Paper presented at the annual meeting of the National Council on Measurement in Education; 2014Philadelphia, PA; 2014.
24. Currie M.,Chiramanee T.The effect of the multiple-choice item format on the measurement of knowledge of language structure.Language Testing. 2010;27:471-491
25. De la Torre J.,Song H.,Hong Y.A comparison of four methods of IRT subscoring.Applied Psychological Measurement. 2011;35:296-316
26. Duncan G. D.,del Rio Parant L.,Chen W.-H.,Ferrara S.,Johnson E.,Oppler S.,Shieh Y.-Y.Study of a dual-language test booklet in eighth-grade mathematics.Applied Measurement in Education. 2005;18:129-161
27. Guidelines for the assessment of English learners. Princeton, NJ: Author; 2009:
28. Forte E.Examining the assumptions underlying the NCLB federal accountability policy on School Improvement.Educational Psychologist. 2010;45:76-88
29. Forte E.,Faulkner-Bond M.,Waring S.,Kuti L.,Fenner D. S.The administrator’s guide to federal programs for English learners. Washington, DC: Thompson; 2010:
30. Geisinger K. F.Psychological testing at the end of the millennium: A brief historical review.Professional Psychology: Research and Practice. 2000;31:117-118
31. GrissomJ. B. (2004). Reclassification of English Learners. Education Policy Analysis Archives, 12(36). Retrieved from http://isla.tamucc.edu/files/esl_reclassificationenglishlearners.pdf
32. Haladyna T. M.,Rodriguez M. C.Developing and validating test items. New York, NY: Routledge; 2014:
33. Hauger J. B.,Sireci S. G.Detecting differential item functioning across examinees tested in their dominant language and examinees tested in a second language.International Journal of Testing. 2008;8:237-250
34. Differential item functioning. Holland P. W.Wainer H., ed. Hillsdale, NJ: Lawrence Erlbaum; 1993:
35. In’nami Y.,Koizumi R.A meta-analysis of test format effects on reading and listening test performance: Focus on multiple-choice and open-ended formats.Language Testing. 2009;26:219-244
36. In’nami Y.,Koizumi R.Language Testing. Test: A multiple-sample analysis; 2012:131-152.
37. International Test Commission (2010). Guidelines for translating and adapting tests. Retrieved from http://www.intestcom.org. Accessed October 12, 2014.
38. Kachchaf R.,Solano-Flores G.Rater language background as a source of measurement error in the testing of English language learners.Applied Measurement in Education. 2012;25:162-177
39. Kane M.Validating the performance standards associated with passing scores.Review of Educational Research. 1994;64:425-461
40. Kane M.Educational measurement. (2006). Validation. In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 17–64). Washington, DC: American Council on Education/Praeger.Brennan R. L., ed. Washington, DC: American Council on Education/Praeger; 2006:17-64.
41. Kane M.Validating the interpretations and uses of test scores.Journal of Educational Measurement. 2013;50:1-73
42. Kane M. T.An argument-based approach to validity.Psychological Bulletin. 1992;112:527-535
43. Kieffer M. J.,Lesaux N. K.,Rivera M.,Francis D. J.Accommodations for English language learners taking large-scale assessments: A meta-analysis on effectiveness and validity.Review of Educational Research. 2009;79:1168-1201
44. Kim J.,Herman J. L.A three-state study of English learner progress.Educational Assessment. 2009;14:212-231
45. Kim J.,Herman J. L.Understanding patterns and precursors of ELL success subsequent to reclassification. Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing; 2012:
46. Koenig J. A.,Bachman L. F.Keeping score for all: The effects of inclusion and accommodation policies on large-scale educational assessments. Washington, DC: National Academies Press; 2004:
47. Kopriva R. J.,Hedgspeth C.Technical manual, selection taxonomy for English language learner accommodation (STELLA) decision-making systems. College Park: University of Maryland, C-SAVE; 2005:
48. Kopriva R. J.,Emick J. E.,Hipolito-Delgado C. P.,Cameron C. A.Do proper accommodation assignments make a difference? Examining the impact of improved decision making on scores of English language learners.Educational Measurement: Issues and Practice. 2007;26 (3): 11-20
49. KuriakoseA. (2011, January1). The factor structure of the English language development assessment: A confirmatory factor analysis (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (ED535975)
50. Lakin J. M.,Young J. W.Evaluating growth for ELL students: Implications for accountability policies.Educational Measurement: Issues and Practice. 2013;32 (3): 11-26
51. Lane S.Validity evidence based on testing consequences.Psicothema. 2014;26:127-135.
52. Lane S.,Leventhal B.Psychometric challenges in assessing English language learners and students with disabilities.Review of Research in Education. 2015;39:165-214
53. Lee W.Classification consistency and accuracy for complex assessments using item response theory. Iowa City: University of Iowa; 2008:
54. Li H.,Suen H. K.The effects of test accommodations for English learners: A meta-analysis.Applied Measurement in Education. 2012;25:327-346
55. Linquanti R.The redesignation dilemma: Challenges and choices in fostering meaningful accountability for English learners. Santa Barbara: University of California Linguistic Minority Research Institute; 2001:
56. Livingston S. A.,Lewis C.Estimating the consistency and accuracy of classifications based on test scores.Journal of Educational Measurement. 1995;32:179-197
57. Luecht R. M.,Ackerman T.Oregon English Language Proficiency Examination (EPLA) dimensionality analysis for blended-domain locator blocks. Greensboro, NC: Center for Assessment Research and Technology; 2007:
58. Martiniello M.Language and the performance of English-language learners in math word problems.Harvard Educational Review. 2008;78:333-368
59. Martone A.,Sireci S. G.Evaluating alignment between curriculum, assessments, and instruction.Review of Educational Research. 2009;4:1332-1361
60. Massachusetts Department of Elementary and Secondary Education. (2012). Transitioning English language learners in Massachusetts: An exploratory data review. Malden, MA: Author. Retrieved from http://www.nciea.org/publication_PDFs/Transitioning%20ELL_CD12.pdf
61. McNamara T.,Knoch U.The Rasch wars: The emergence of Rasch measurement in language testing.Language Testing. 2012;29:555-576
62. Messick S.Educational measurement. (1989). Validity. In LinnR. (Ed.), Educational measurement (3rd ed., pp. 13–100). Washington, DC: American Council on Education.Linn R., ed. Washington, DC: American Council on Education; 1989:13-100.
63. Mosher F. A.The role of learning progressions in standards-based education reform. Philadelphia, PA: Consortium for Policy Research in Education; 2011:
64. National Center for Education Statistics. (n.d.). Common core of data (CCD). Retrieved from http://nces.ed.gov/ccd/index.asp
65. National Research Council. (2011). Allocating federal funds for state programs for English language learners. Washington, DC: National Academies Press. Retrieved from http://www.nap.edu/openbook.php?record_id=13090
66. O’ConnerR.AbediJ.TungS. (2012a). A descriptive analysis of enrollment and achievement among English language learner students in Pennsylvania: Summary (Issues & Answers, REL 2012-No. 127). Retrieved from http://files.eric.ed.gov/fulltext/ED531429.pdf
67. O’ConnerR.AbediJ.TungS. (2012b). A descriptive analysis of enrollment and achievement among limited English proficient students in New Jersey (Issues & Answers, REL 2012-No. 108). Retrieved from http://files.eric.ed.gov/fulltext/ED531432.pdf
68. Padilla J.-L.,Benitez I.Validity evidence based on response processes.Psicothema. 2014;26:110-117
69. ParkerC.LouieJ.O’DwyerL. (2009). New measures of English language proficiency and their relationship to performance on large-scale content assessments (Issues & Answers, REL 2009 No. 066). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northeast and Islands. Retrieved from http://ies.ed.gov/ncee/edlabs
70. Accessibility features and accommodations manual. Washington, DC: Author; 2013:
71. Pennock-Roman M.Relative effects of English proficiency on general admissions tests versus subject tests.Research in Higher Education. 2002;43:601-623
72. Pennock-Roman M.,Rivera C.Mean effects of test accommodations for ELLs and non-ELLs: A meta-analysis of experimental studies.Educational Measurement: Issues and Practice. 2011;30 (3): 10-28
73. Ragan A.,Lesaux N.Federal, state, and district level English language learner program entry and exit requirements: Effects on the education of language minority learners.Education Policy Analysis Archives. 2006;14 (20): 1-32
74. Ramsey P. A.Differential item functioning. Holland P. W.Wainer H., ed. Hillsdale, NJ: Erlbaum; 1993:367-388.
75. Rivera C.,Collum E.,Willner L. S.,Sia J. K.A national review of state assessment policy and practice for English language learners. Rivera C.Collum E., ed. Mahwah, NJ: Lawrence Erlbaum; 2006:1-173.
76. Robinson J. P.Evaluating criteria for English learner reclassification: A causal-effects approach using a binding-score regression discontinuity design with instrumental variables.Educational Evaluation and Policy Analysis. 2011;33:267-292
77. Römhild A.,Kenyon D.,MacGregor D.Exploring domain-general and domain-specific linguistic knowledge in the assessment of academic English language proficiency.Language Assessment Quarterly. 2011;8:213-228
78. Rudner L. M.Computing the expected proportions of misclassified examinees.Practical Assessment, Research & Evaluation. 2001;7 (14):
79. Rudner L. M.Expected classification accuracy.Paper presented at the annual meeting of the National Council on Measurement in Education; 2004San Diego, CA; 2004.
80. Saunders W. M.,Goldenberg C.Improving education for English learners: Research-based approaches. Sacramento: California Department of Education; 2010:20-81.
81. Sawaki Y.,Stricker L. J.,Oranje A. H.Factor structure of the TOEFL Internet-based test.Language Testing. 2009;26:5-30
82. Schleppegrell M. J.,O’Hallaron C. L.Teaching academic language in L2 secondary settings.Annual Review of Applied Linguistics. 2011;31:3-18
83. Shepard L.,Taylor G.,Betebenner D.Los Angeles: University of California, Center for the Study of Evaluation/National Center for Research on Evaluation, Standards, and Student Testing; 1998:
84. Sireci S. G.Gathering and analyzing content validity data.Educational Assessment. 1998;5:299-321
85. Sireci S. G.Agreeing on validity arguments.Journal of Educational Measurement. 2013;50:99-104
86. Sireci S. G.,Li S.,Scarpati S.The effects of tests accommodations on test performance: A review of the literature. Commissioned paper by the National Academy of Sciences/National Research Council’s Board on Testing and Assessment. Washington, DC: National Research Council; 2003:
87. Sireci S. G.,Mullane L. A.Evaluating test fairness in licensure testing: The sensitivity review process.CLEAR Exam Review. 1994;5 (2): 22-28
88. Sireci S. G.,Rios J. A.,Powers S.Fairness. Dorans N.Cook L., ed. New York, NY: Routledge; in press:
89. Sireci S. G.,Wells C.,Hu H..Paper presented at the annual meeting of the National Council on Measurement in Education; 2014Philadelphia, PA; 2014.
90. Usability, accessibility, and accommodations guidelines. San Francisco, CA: WestEd; 2013:
91. Solano-FloresG. (2012). Translation accommodations framework for testing English language learners in mathematics. Retrieved from http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/09/Translation-Accommodations-Framework-for-Testing-ELL-Math.pdf
92. Solano-Flores G.,Li M.Language variation and score variation in the testing of English language learners, native Spanish speakers.Educational Assessment. 2009;14:180-194
93. Solano-Flores G.,Trumbull E.,Nelson-Barber S.Concurrent development of dual language assessments: An alternative to translating tests for linguistic minorities.International Journal of Testing. 2002;2:107-129
94. Stansfield C. W.Test translation and adaptation in public education in the USA.Language Testing. 2003;20:188-206
95. Swanson C. B.Perspectives on a population: English-language learners in American Schools. Bethesda, MD: Editorial Projects in Education Research Center; 2009:
96. ThompsonS.BlountA.ThurlowM. (2002). A summary of research on the effects of test accommodations: 1999 through 2001 (Technical Report No. 34). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Retrieved from http://education.umn.edu/NCEO/OnlinePubs/Technical34.htm
97. UCLA. (n.d.). Dynamic Language Learning Progression Project: Resources on learning progressions. Retrieved from http://www.dllp.org/index.php/resources/
98. U.S. Department of Education. (2011). Consolidated state performance reports: Parts I and II school years 2009-2010. Washington, DC: Office of Elementary and Secondary Education. Retrieved from http://www2.ed.gov/admins/lead/account/consolidated/index.html#sy09-10
99. National evaluation of Title III implementation supplemental report: Exploring approaches to setting English language proficiency performance criteria and monitoring English learner progress. Washington, DC: Author; 2012:
100. Standards and assessments peer review guidance: Information and examples for meeting the requirements of the No Child Left Behind Act of 2001. Washington, DC: Author; 2009:
101. 2012 Amplification of the English language development standards, Kindergarten-Grade 12. Madison: Board of Regents of the University of Wisconsin System; 2012:
102. Willner L.,Rivera C.,Acosta B.Descriptive study of state assessment policies for accommodating English language learners. Arlington, VA: George Washington University Center for Equity and Excellence in Education; 2008:
103. Wilson M.,Moore S.Building out a measurement model to incorporate complexities of testing in the language domain.Language Testing. 2011;28:441-462
104. Wolf M. K.,Kim J.,Kao J.The effects of glossary and read-aloud accommodations on English language learners’ performance on a mathematics assessment.Applied Measurement in Education. 2012;25:347-374
105. Wolf M. K.,Leon S.An investigation of the language demands in content assessments for English language learners.Educational Assessment. 2009;14:139-159
106. Wolf M. K.,Wang Y.,Holtzman S..Paper presented at the annual meeting of the American Educational Research Association; 2011New Orleans, LA; 2011.
107. Working Group on ELL Policy. (2010). Improving educational outcomes for English language learners: Recommendations for the reauthorization of the Elementary and Secondary Education Act. Retrieved from http://ellpolicy.org/wp-content/uploads/ESEAFinal.pdf
108. YoungJ. W.PitoniakM. J.KingT. C.AyadE. (2012). Guidelines for accessibility for English language learners. Retrieved from http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/05/TaskItemSpecifications/Guidelines/AccessibilityandAccommodations/GuidelinesforAccessibilityforELL.pdf
109. Zhang B.Assessing the accuracy and consistency of language proficiency classification under competing measurement models.Language Testing. 2010;27:119-140
110. Zwick R.,Schlemer L.SAT validity for linguistic minorities at the University of California, Santa Barbara.Educational Measurement: Issues and Practice. 2004;23:6-16