A framework for test validity research on content assessments taken by English language learners

Educational Assessment

Volumn 14, Issue 3-4, 2009, Pages 122-138

  Young, John W ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 74249099463     PISSN: 10627197     EISSN: None     Source Type: Journal    
DOI: 10.1080/10627190903422856     Document Type: Article

References (63)

1. Abedi, J. (2002). Standardized achievement tests and English language learners: Psychometric issues. Educational Assessment, 8, 231-257.
2. Abedi, J. (2003). Impact of student language background on content-based performance: Analyses of extant data (CSE Rep. No. 603). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.
3. Abedi, J. (2006). Language issues in item development. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 377-398). Mahwah, NJ: Erlbaum.
4. Abedi, J., Courtney, M., & Leon, S. (2003). Research-supported accommodations for English language learners in NAEP (CSE Tech. Rep. No. 586). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.
5. Abedi, J., & Gandara, P. (2006). Performance of English language learners as a subgroup in large-scale assessment: Interaction of research and policy. Educational Measurement: Issues and Practice, 25(4), 36-46.
6. Abedi, J., & Lord, C. (2001). The language factor in mathematics tests. Applied Measurement in Education, 14, 219-234.
7. Abedi, J., Lord, C., & Plummer, J. (1995). Language background as a variable in NAEP mathematics performance (CSE Tech. Rep. No. 429). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.
8. August, D., Carlo, M., Dressler, C., & Snow, C. (2005). The critical role of vocabulary development for English language learners. Learning Disability Research and Practice, 20(1) 50-57.
9. Bailey, A. L., Huang, B. H., Shin, H. W., Farnsworth, T., & Butler, F. A. (2007). Developing academic English language proficiency prototypes for 5th-grade reading: Psychometric and linguistic profiles of tasks (CSE Tech. Rep. No. 727). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.
10. Bernhardt, E. (2005). Progress and procrastination in second language reading. Annual Review of Applied Linguistics, 25,133-150.
11. Borgioli, G. (2008). Equity for English language learners in mathematics classrooms. Teaching Children Mathematics, 15,185-191.
12. Camilli, G., & Shepard, L. (1994). Methods for identifying biased test items. Newbury Park, CA: Sage.
13. Castellon-Wellington, M. (2000). The impact of preference for accommodations: The performance of ELLs on largescale academic achievement tests (Tech. Rep. No. 524). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.
14. Clauser, B. E., & Mazor, K. M. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues and Practice, 17, 31-44.
15. Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 443-507). Washington, DC: American Council on Education.
16. Cummins, D. D., Kintsch, W., Reusser, K., & Weimer, R. (1988). The role of understanding in solving word problems. Cognitive Psychology, 20, 405-438.
17. Dodd, B. G. (Ed.). (2004). Assessing the population sensitivity of equating functions [Special issue]. Journal of Educational Measurement, 41(1).
18. Dorans, N. J., & Kulick, E. (1986). Demonstrating the utility of the standardization approach to assessing unexpected differential item performance on the Scholastic Aptitude Test. Journal of Educational Measurement, 23, 355-368.
19. Duncker, K. (1945). On problem-solving. In J. F. Dashiell (Ed.), Psychological monographs (pp. 1-114). Washington, DC: American Psychological Association.
20. Embretson, S., & Gorin, J. (2001). Improving construct validity with cognitive psychology principles. Journal of Educational Measurement, 38, 343-368.
21. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Revised edition). Cambridge, MA: MIT Press.
22. Fidalgo, A. M., Ferreres, D., & Muniz, J. (2004). Utility of the Mantel-Haenszel procedure for detecting differential item functioning in small samples. Educational and Psychological Measurement, 64, 925-936.
23. Fidalgo, A. M., Mellenbergh, G. J., & Muniz, J. (1998). Comparison of the Mantel-Haenszel procedure versus the log linear models for detecting differential item functioning. Psicotherma, 10, 219-228.
24. Forster, K. I., & Olbrei, I. (1973). Semantic heuristics and syntactic trial. Cognition, 2, 319-347.
25. Francis, D., Lesaux, N., Kieffer, M., & Rivera, H. (2006). Research-based recommendations for the use of accommodations in large-scale assessments. Houston, TX: Center on Instruction.
26. Hamilton, L. S., Nussbaum, E. M., & Snow, R. E. (1997). Interview procedures for validating science assessments. Applied Measurement in Education, 10, 181-200.
27. Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Erlbaum.
28. Johnstone, C. J., Bottsford-Miller, N. A., & Thompson, S. J. (2006). Using the think aloud method (cognitive labs) to evaluate test designs for student with disabilities and English language learners (Tech. Rep. No. 44). Minneapolis: University of Minnesota, National Center on Educational Outcomes.
29. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). Westport, CT: American Council on Education/Praeger.
30. Kindler, A. L. (2002). Survey of the states' limited English proficient students and available educational programs and services: 2000-2001 summary report. Washington, DC: National Clearinghouse for English Language Acquisition and Language Instruction Educational Programs.
31. Kopriva, R. J. (2000). Ensuring accuracy in testing for English language learners: A practical guide for assessment development. Washington, DC: Council of Chief State School Officers.
32. Kopriva, R. J. (2001, June). ELL validity research designs for state academic assessments: An outline of five research designs evaluating the validity of large-scale assessments for English language learners and other test takers. Paper presented at the Council of Chief State School Officers Meeting, Houston, TX.
33. Kopriva, R. J. (2008). Improving testing for English language learners. New York: Routledge.
34. Kopriva, R. J., Emick, J. E., Hipolito-Delgado, C. P., & Cameron, C. A. (2008). Do proper accommodation assignments make a difference? Examining the impact of improved decision making on scores for English language learners. Educational Measurement: Issues and Practice, 27(3), 11-20
35. Leighton, J. P. (2004). Avoiding misconceptions, misuse, and missed opportunities: The collection of verbal reports in educational achievement testing. Educational Measurement: Issues and Practice, 23(4), 6-15.
36. Leighton, J. P., Rogers, W. T., & Mcguire, T. O. (1999). Assessment of student problem solving on ill-defined tasks. Alberta Journal of Educational Research, 45, 409-427.
37. Linn, R. L. (1978). Single-groupvalidity, differential validity, and differential prediction. Journal of Applied Psychology, 63,507-512.
38. Mapuranga, R., Dorans, N. J., & Middleton, K. (2008). A review of recent development in differential item functioning (ETS RR-08-43). Princeton, NJ: Educational Testing Service.
39. Martiniello, M. (2006, October). Sources of differential item functioning for English learners in word math problems. Paper presented at the annual meeting of the Northeastern Educational Research Association, Ellenville, NY.
40. Martiniello, M. (2007). Linguistic complexity and Differential Item Functioning (DIF) for English Language Learners (ELL) in math word problems. Dissertation Abstracts International Section A: Humanities and Social Sciences, Vol 68(6-A), 2007. p. 2422. Publisher: US: ProQuest Information & Learning. ISSN: 0419-4209 (Print).
41. Martiniello, M. (2008). Language and the performance of English-language learners in math word problems. Harvard Educational Review, 78, 333-368.
42. Mazor, K. M., Clauser, B. E., & Hambleton, R. K. (1992). The effect of sample size on the functioning of the Mantel-Haenszel statistic. Educational and Psychological Measurement, 52, 443-452.
43. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). Washington, DC: American Council on Education/Macmillan.
44. Millsaps, R. E., & Everson, H. T. (1993). Methodology review: Statistical approaches for assessment measurement bias. Applied Psychological Measurement, 17, 297-334.
45. Muniz, J., Hambleton, R. K., & Xing, D. (2001). Small sample studies to detect flaws in item translations. International Journal of Testing, 1(2), 115-135.
46. National Clearinghouse for English Language Acquisition. (2007). The growing numbers of limited English proficient students: 1995-96-2005-06. Retrieved from http://www.ncela.gwu.edu/stats/2_nation.htm
47. Parshall, C. G., & Miller, T. R. (1995). Exact versus asymptotic Mantel-Haenszel DIF statistics: A comparison of performance under small-sample conditions. Journal of Educational Measurement, 32, 302-316.
48. Penfield, R. D., & Lam, T. C. M. (2000). Assessing differential item functioning in performance assessment: Review and recommendations. Educational Measurement: Issues and Practice, 19, 5-15.
49. Pennock-Roman, M., & Rivera, C. (2007). Test validity and mean effects of test accommodations for ELLs and non-ELLs: A meta-analysis. Arlington, VA: George Washington University Center for Equity and Excellence in Education.
50. Rivera, C., Collum, E., Willner, L. S., & Sia, J. K. (2006). An analysis of state assessment policies regarding the accommodation of English language learners. In C. Rivera & E. Collum (Eds.), State assessment policy and practice for English language learners: A national perspective (pp. 1-174). Mahwah, NJ: Erlbaum.
51. Ryan, K. E. (1991). The performance of the Mantel-Haenszel procedure across samples and matching criteria. Journal of Educational Measurement, 28, 325-337.
52. Schachter, P. (1983). On syntactic categories. Bloomington: Indiana University Linguistics Club.
53. Sherman, S., & Robinson, N. (Eds.). (1982). Ability testing of handicapped people: Dilemma for government, science, and the public. Washington, DC: National Academy Press.
54. Sireci, S., Han, K. T., & Wells, C. S. (2008). Methods for evaluating the validity of test scores for English language learners. Educational Assessment, 13(3), 108-131.
55. Thompson, S. J., Johnstone, C. J., & Thurlow, M. L. (2002). Universal design applied to large scale assessments (Synthesis Rep. No. 44). Minneapolis: National Center on Educational Outcomes.
56. Willingham, W. W., Ragosta, M., Bennett, R. E., Braun, H., Rock, D. A., & Powers, D. E. (1988). Testing handicapped people. Needham, MA: Allyn & Bacon.
57. Willner, L. S., Rivera, C., & Acosta, B. D. (2008). Descriptive study of state assessment policies for accommodating English language learners. Arlington, VA: George Washington University Center for Equity and Excellence in Education.
58. Winter, P., Kopriva, R. J., Chen, S., & Emick, J. (2006). Exploring individual and item factors that affect assessment validity for diverse learners: Results from a large-scale cognitive lab. Learning and Individual Differences, 16, 267-276.
59. Wolf, M. K., Farnsworth, T., & Herman, J. (2008). Validity issues in assessing English language learners' language proficiency. Educational Assessment, 13(3), 80-107.
60. Young, J. W. (2001). Differential validity, differential prediction, and college admission testing: A comprehensive review and analysis (College Board Research Rep. No. 2001-6). New York: College Entrance Examination Board.
61. Young, J. W. (2008, December). Ensuring valid content tests for English language learners (R&D Connections, No. 8). Princeton, NJ: Educational Testing Service.
62. Young, J. W., Cho, Y., Ling, G., Cline, F., Steinberg, J., & Stone, E. (2008). Validity and fairness of state standards-based assessments for English language learners. Educational Assessment, 13(3), 170-192.
63. Young, J. W., & King, T. C. (2008). Testing accommodations for English language learners: A review of state and district policies (College Board Research Rep. No. 2008-6 and ETS RR-08-48). New York: College Entrance Examination Board.