Investigating uniform and non-uniform gender DIF in computer-based ESL writing assessment

Applied Measurement in Education

Volumn 20, Issue 4, 2007, Pages 377-403

  Breland, Hunter ^a   Lee, Yong Won ^b  

^a Seoul National University ^*

^b Seoul National University   (South Korea)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 41449105940     PISSN: 08957347     EISSN: None     Source Type: Journal    
DOI: 10.1080/08957340701429652     Document Type: Article

References (56)

1. AERA/APA/NCME (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education). (1999). Standards for educational and psychological testing. Washington., DC: AERA.
2. Agresti, A. (1990). Categorical data analysis. New York: Wiley.
3. Bermudez, A. B., & Prater, D. L. (1994). Examining the effects of gender and second language proficiency on Hispanic writers' persuasive discourse. Bilingual Research Journal, 18(3-4), 47-62.
4. Breland, H. M., & Griswold, P. A. (1982). Use of a performance test as a criterion in a differential validity study. Journal of Educational Psychology, 74(5), 713-721.
5. Breland, H. M., & Jones, R. J. (1982). Perceptions of writing skill (College Board Report No. 88-3, ETS RR-82-47). New York: The College Board.
6. Breland, H., Lee, Y.-W., & Muraki, E. (2005). Comparability of TOEFL CBT writing prompts: Response mode analyses. Educational and Psychological Measurement, 65(4), 577-595.
7. Bridgeman, B., & Bonner, M. (1994). SAT-W as a predictor of grades in freshman English composition courses. Unpublished manuscript.
8. Bridgeman, B., & McHale, F. (1996). Potential impact of the addition of writing assessment on admissions decisions. Research in Higher Education, 39, 663-677.
9. Broer, M., Lee, Y.-W., Rizavi, S., & Powers, D. (2005). Ensuring the fairness of GRE writing prompts: Assessing differential difficulty (GRE Research Report No. 02- 07R; ETS RR-05-11). Princeton, NJ: ETS.
10. Clauser, B. E., & Major, K. M. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues and Practice, 17(1), 31-44.
11. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
12. Cohen, J. (1992). Statistical power analysis. Current Directions in Psychological Science, 1(3), 98-101.
13. College Board (2000). An overview of computer-based testing (Research Notes RN-09, April 2000). New York: Author.
14. Cox, D. R., & Snell, E. J. (1989). Analysis of binary data (2nd ed.). London: Chapman & Hall.
15. Davis, K. A., & Skilton-Sylvester, E. (2004). Looking back, taking stock, moving forward: Investigating gender in TESOL. TESOL Quarterly, 38(3), 381-404.
16. Educational. Testing Service. (1998). Computer-based TOEFL score user guide. Princeton, NJ: Author.
17. Englehard, G., Gordon, B., & Gabrielson, S. (1991). The influences of mode of discourse, experiential demand, and gender on the quality of school writing. Research in the Teaching of English, 26(3), 315-336.
18. Ehrlich, S. (1997). Gender as social practice. Implications for second language acquisition. Studies in Second Language Acquisition, 19, 421-446.
19. Ehrman, M. E., & Oxford, R. L. (1990). Adult learning styles and strategies in an intensive training setting. Modern Language Journal, 73, 1-13.
20. Ellis, R. (1994). The study of second language acquisition. Oxford, England: Oxford University Press.
21. French, A. W., & Miller, T. R. (1996). Logistic regression and its use in detecting differential item. functioning in polytoinous items. Journal of Educational Measurement, 33(3), 315-332.
22. Folk, V. G., & Smith, R. L. (2002). Models for delivery for CBTs. In C. Mills, M. Potenza, J. Fremer, & W. Ward (Eds.), Computer-based testing: Building the foundations of future assessments (pp. 41-66). Mahwah, NJ: Lawrence Erlbaum.
23. Gibson, W. M., & Weiner, J. A. (1998). Generating random parallel test forms using CTT in a computer-based environment. Journal of Educational Measurement, 35, 297-310.
24. Ginther, A., & Stevens, J. (1998). Language background, ethnicity, and the internal, construct validity of the Advanced Placement Spanish Language Examination. In A. J. Kunnan (Ed.), Validation, in language assessment (pp. 169-194). Mahwah, NJ: Lawrence Erlbaum.
25. Gierl, M. J., Bisanz, J., Bisanz, G., & Boughton, K. (2003). Identifying content and cognitive skills that produce gender differences in mathematics: A demonstration of the DIF analysis framework. Journal of Educational Measurement, 40, 281-306.
26. Golub-Smith, M., Reese, C., & Steinhaus, K. (1993). Topic and topic type comparability on the Test of Written English (TOEFL Research. Report No. 42; ETS RR-93-10). Princeton, NJ: Educational Testing Service.
27. Halpern, D. (1992). Sex differences in cognitive abilities. Hillside, NJ: Lawrence Erlbaum.
28. Heck, R. H., & Crislip, M. (2001). Direct and indirect writing assessments: Examining issues of equity and utility. Educational Evaluation and Policy Analysis, 23(1), 19-36.
29. Hosmer, D. W., & Lemeshow, S. (1989). Applied logistic regression. New York: Wiley.
30. Jodoin, M. G., & Gierl, M. J. (2001). Evaluating Type I error and power rates using an effect size measure with, the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(A), 329-349.
31. Kolen, M. J., & Brennan, R. L. (1995). Test equating: Methods and practices. New York: Springer-Verlag.
32. Lee, Y.-W., Breland, H., & Muraki, E. (2005). Comparability of TOEFL CBT essay prompts for different native language groups. International Journal of Testing, 5(2), 133-160.
33. Mazor, K. M., Kanjee, A., & Clauser, B. E. (1995). Using logistic regression and the Mantel-Haenszel with multiple ability estimates to detect differential item functioning. Journal of Educational Measurement, 32(2), 131-144.
34. Mazzeo, J., Schmitt, A. P., & Bleistein, C. A. (1993). Sex-related performance differences on constructed response and multiple-choice sections of Advanced Placement examinations (College Board Report No. 92-7; ETS RR-93-5). New York: The College Board.
35. Mellenbergh, G. J. (1982). Contingency table models for assessing item bias. Journal of Educational Statistics, 7, 105-107.
36. Nagelkerke, N. J. D. (1991). A note one general, definition of the coefficient of determination. Biometrika, 78(3), 691-692.
37. National. Assessment of Educational Progress. (1994). NAEP 1992 trends in academic progress. Washington, DC: National. Center for Education. Statistics.
38. Oltman, P. K., Stricker, L. J., & Barrows, T. (1988). Native language, English, proficiency, and the structure of the Test of English as a Foreign Language for several language groups (TOEFL Research Report no. 27). Princeton, NJ: ETS.
39. Penfield, R. D., & Lam, T. C. M. (2000). Assessing differential item functioning in performance assessment: Review and recommendations. Educational Measurement: Issues and Practice, 19(3), 5-15.
40. Pomplun, M., & Sundbye, N. (1999). Gender differences in constructed response reading items. Applied Measurement in Education, 12(1), 95-109.
41. Pomplun, M., Wright, D., Oleka, N., & Sudlow, M. (1992). An analysis of English Composition Test prompts for differential difficulty (College Board Report 92-4; ETS RR-92-34). New York: The College Board.
42. Roussos, L., & Stout, W. (1996). A multidimensionality-based DIF analysis paradigm. Applied Psychological Measurement, 20, 355-371.
43. Samejima, F. (1969). Estimation of ability using a response pattern of graded scores. Psychometrika Monograph, 17, 1-100.
44. SAS Institute (1990). SAS/STAT user's guide, version 6 (4th ed.). Cary, NC: SAS Institute, Inc.
45. Schaeffer, G. A., Briel, J. B., & Fowles, M. E. (2001). Psychometric evaluation of the new GRE Writing Assessment (GRE Board Profession Report No. 96-11P, ETS RR-01-08). Princeton, NJ: Educational. Testing Service.
46. Schmenk, B. (2004). Language Learning: A feminine domain? The role of stereo typing in constructing gendered learner identities. TESOL Quarterly, 38(3), 514-524.
47. Sheppard, L. A. (1982). Definition of bias. In R. Berk (Ed.), Handbook of methods for detecting test bias (pp. 9-30). Baltimore, MD: Johns Hopkins University.
48. Sincoff, J. B., & Sternberg, R. J. (1987). Two faces of verbal ability. Intelligence, 11, 263-276.
49. Sincoff, J. B., & Sternberg, R. J. (1988). Development of verbal fluency abilities and strategies in elementary-school-age children. Developmental Psychology, 24, 646-653.
50. Stocking, M. L., & Eignor, D. R. (1986). The impact of different ability distributions on IRT preequating (ETS Research No. 86-49). Princeton, NJ: Educational Testing Service.
51. Sunderland, J. (2000). Issues of language and gender in second and foreign language education. Language Teaching, 33, 203-223.
52. Swaminathan, H., & Rogers, H. J. (1990). Detecting differential, item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361-370.
53. Swinton, S. S., & Powers, D. E. (1980). Factor analysis of the Test of English as a Foreign Language for several language groups (TOEFL Research Report no. 6). Princeton, NJ: ETS.
54. Welch, C., & Miller, T. (1995). Assessing differential item functioning in direct writing assessments: Problems and an example. Journal of Educational Measurement, 32, 163-178.
55. Willinghain, W. W., & Cole, N. S. (1997). Gender and fair assessment. Mahwah, NJ: Erlbaum.
56. Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, Ontario, Canada: Directorate of Human Resources Research and Evaluation, Department of National Defense.