AN ANALYSIS OF TOEFL CBT WRITING PROMPT DIFFICULTY AND COMPARABILITY FOR DIFFERENT GENDER GROUPS

ETS Research Report Series

Volumn 2004, Issue 1, 2004, Pages i-54

  Breland, Hunter ^a   Lee, Yong Won ^a   Najarian, Michelle ^a   Muraki, Eiji ^b  

^a EDUCATIONAL TESTING SERVICE   (United States)

^b TOHOKU UNIVERSITY   (Japan)

Author keywords

comparability; computer based writing assessment; essay prompts; fairness; gender; logistic regression; polytomous DIF (differential item functioning); proportional odds ratio model

Indexed keywords

EID: 85164501392     PISSN: None     EISSN: 23308516     Source Type: Journal    
DOI: 10.1002/j.2333-8504.2004.tb01932.x     Document Type: Article

References (33)

1. Agresti, A. (1990). Categorical data analysis. New York: Wiley.
2. Bermudez, A. B., & Prater, D. L. (1994). Examining the effects of gender and second language proficiency on Hispanic writers' persuasive discourse. Bilingual Research Journal, 18(3–4), 47–62.
3. Berninger, V. W., & Fuller, F. (1992). Gender differences in orthographic, verbal, and compositional fluency—Implications for assessing writing disabilities in primary grade children. Journal of School Psychology, 30(4), 363–382.
4. Breland, H. M., & Griswold, P. A. (1982). Use of a performance test as a criterion in a differential validity study. Journal of Educational Psychology, 74(5), 713–721.
5. Breland, H. M., & Jenkins, L. M. (1997). English word frequency statistics: Analysis of a corpus of 14 million tokens. New York: College Entrance Examination Board.
6. Breland, H. M., & Jones, R. J. (1982). Perceptions of writing skill (College Board Report No. 88–3, ETS RR-82-47). New York: The College Board.
7. Bridgeman, B., & Bonner, M. (1994). SAT-W as a predictor of grades in freshman English composition courses. Unpublished manuscript.
8. Bridgeman, B., & McHale, F. (1996). Potential impact of the addition of writing assessment on admissions decisions. Research in Higher Education, 39, 663–677.
9. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
10. Cohen, J. (1992). Statistical power analysis. Current Directions in Psychological Science, 1(3), 98–101.
11. Cumming, A. (1990a). Expertise in evaluating second language composition. Language Testing, 7, 31–51.
12. Cumming, A. (1990b). Metalinguistic and ideational thinking in second language composing. Written Communication, 7, 482–511.
13. Cumming, A., & Mellow, D. (1996). An investigation into the validity of written indicators of second language proficiency. In A. Cumming & R. Berwick (Eds.), Validation in language testing (pp. 72–93). Clevedon, UK: Multilingual Matters.
14. Englehard, G., Gordon, B., & Gabrielson, S. (1991). The influences of mode of discourse, experiental demand, and gender on the quality of school writing. Research in the Teaching of English, 26(3), 315–336.
15. ETS. (1998). Computer-based TOEFL test score user guide. Princeton, NJ: Author.
16. French, A. W., & Miller, T. R. (1996). Logistic regression and its use in detecting differential item functioning in polytomous items. Journal of Educational Measurement, 33(3), 315–332.
17. Golub-Smith, M., Reese, C., & Steinhaus, K. (1993). Topic and topic type comparability on the Test of Written English (TOEFL Research Report No. 42, ETS RR-93-10). Princeton, NJ: ETS.
18. Heck, R. H., & Crislip, M. (2001). Direct and indirect writing assessments: Examining issues of equity and utility. Educational Evaluation and Policy Analysis, 23(1), 19–36.
19. Hosmer, D. W., & Lemeshow, S. (1989). Applied logistic regression. New York: Wiley.
20. Jodoin, M. G., & Gierl, M. J. (2001). Evaluating Type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329–349.
21. Mazzeo, J., Schmitt, A. P., & Bleistein, C. A. (1993). Sex-related performance differences on constructed response and multiple-choice sections of Advanced Placement Examinations (College Board Report No. 92-7, ETS RR-93-5). New York: The College Board.
22. National Assessment of Educational Progress. (1994). NAEP 1992 trends in academic progress. Washington, DC: National Center for Education Statistics.
23. Pomplun, M., Wright, D., Oleka, N., & Sudlow, M. (1992). An analysis of English Composition Test prompts for differential difficulty (College Board Report No. 92-4, ETS RR-92-34). New York: The College Board.
24. Powers, D. E., & Fowles, M. E. (1999). Test-takers' judgments of essay prompts: Perceptions and performance. Educational Assessment, 6(1), 3–22.
25. Samejima, F. (1969). Estimation of ability using a response pattern of graded scores. Psychometrika Monograph, 17.
26. SAS Institute. (1990). SAS/STAT user's guide, version 6 (4th ed.). Cary, NC: SAS Institute, Inc.
27. Schaeffer, G. A., Briel, J. B., & Fowles, M. E. (2001). Psychometric evaluation of the new GRE Writing Assessment (GRE Board Profession Report No. 96–11P, ETS RR-01-08). Princeton, NJ: ETS.
28. Sincoff, J. B., & Sternberg, R. J. (1987). Two faces of verbal ability. Intelligence, 11, 263–276.
29. Sincoff, J. B., & Sternberg, R. J. (1988). Development of verbal fluency abilities and strategies in elementary-school-age children. Developmental Psychology, 24, 646–653.
30. Stansfield, C. W., & Ross, J. (1988). A long-term research agenda for the Test of Written English. Language Testing, 5, 160–186.
31. Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370.
32. Willingham, W. W., & Cole, N. S. (1997). Gender and fair assessment. Mahwah, NJ: Erlbaum.
33. Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, Ontario, Canada: Directorate of Human Resources Research and Evaluation, Department of National Defense.