COMPARABILITY OF TOEFL CBT WRITING PROMPTS FOR DIFFERENT NATIVE LANGUAGE GROUPS

ETS Research Report Series

Volumn 2004, Issue 1, 2004, Pages i-56

  Lee, Yong Won ^a   Breland, Hunter ^a   Muraki, Eiji ^b  

^a EDUCATIONAL TESTING SERVICE   (United States)

^b TOHOKU UNIVERSITY   (Japan)

Author keywords

computer based writing assessment; essay prompt comparability; fairness; logistic regression; native languages; polytomous DIF (differential item functioning); proportional odds ratio model

Indexed keywords

EID: 85164495607     PISSN: None     EISSN: 23308516     Source Type: Journal    
DOI: 10.1002/j.2333-8504.2004.tb01951.x     Document Type: Article

References (55)

1. Angoff, W. H. (1989). Context bias in the Test of English as a Foreign Language (TOEFL Research Report No. 29). Princeton, NJ: ETS.
2. Angoff, W. H. (1993). Perspectives on differential item functioning methodology. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 3–23). Hillsdale, NJ: Lawrence Erlbaum.
3. American Educational Research Association, American, Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
4. Agresti, A. (1990). Categorical data analysis. New York: Wiley.
5. Breland, H., Muraki, E., & Lee, Y. (2001, April). Comparability of TOEFL CBT writing prompts for different response modes. Paper presented at the annual meeting of National Council on Measurement in Education (NCME) in Seattle, WA.
6. Breland, H., Muraki, E., Lee, Y., Najarian, M., & Beyer, J. (2000, April). Comparability in TOEFL CBT essay prompts using the logistic regression method. Paper presented at the annual conference of National Council on Measurement in Education (NCME) in New Orleans, LA.
7. Brown, A., & Iwashita, N. (1998). The role of language background in the validation of a computer-adaptive test. In A. Kunnan (Ed.), Validation in language assessment (pp. 195–207). Mahwah, NJ: Lawrence Erlbaum.
8. Chang, H., Mazzeo, J., & Roussos, L. A. (1995). Detecting DIF for polytomously scored items: An adaptation of the SIBTEST procedure (ETS RR-95-5). Princeton, NJ: ETS.
9. Chen, Z., & Henning, G. (1985). Linguistic and cultural bias in language proficiency tests. Language Testing, 2, 155–163.
10. Clauser, B. E., & Mazor, K. M. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues and Practice, 17(1), 31–44.
11. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
12. Connor, U. (2002). New directions in contrastive rhetoric. TESOL Quarterly, 36(4), 493–510.
13. DeMauro, G. (1992). An investigation of the appropriateness of the TOEFL test as matching variable to equate TWE topics (TOEFL Research Report No. 37). Princeton, NJ: ETS.
14. Dorans, N. J., & Schmitt, A. J. (1991). Constructed response and differential item functioning: A pragmatic approach (ETS RR-91-47). Princeton, NJ: ETS.
15. ETS. (1998). Computer-based TOEFL score user guide. Princeton, NJ: Author.
16. ETS. (1999). TOEFL tips: Preparing students for the computer-based test. Princeton, NJ: Author.
17. French, A. W., & Miller, T. R. (1996). Logistic regression and its use in detecting differential item functioning in polytomous items. Journal of Educational Measurement, 33, 315–332.
18. Gentile, C., Riazantseva, A., & Cline, F. (2001). A comparison of handwritten and word-processed TOEFL essays. Manuscript submitted for publication.
19. Ginther, A., & Stevens, J. (1998). Language background, ethnicity, and the internal construct validity of the Advanced Placement Spanish Language Examination. In Kunnan, A. J. (Ed.), Validation in language assessment (pp. 169–94). Mahwah, NJ: Lawrence Erlbaum.
20. Hale, G. A., Rock, D. A., & Jirele, T. (1989). Confirmatory factor analysis of the Test of English as a Foreign Language (TOEFL Research Report No. 32). Princeton, NJ: ETS.
21. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.
22. Hauck, W. W. (1983). A note on confidence bands for the logistic response curve. The American Statistician, 37, 158–160.
23. Henning, G. (1996). Accounting for nonsystematic error in performance testing. Language Testing, 13, 53–61.
24. Hinds, J. (1990). Inductive, deductive, quasi-inductive: Expository writing in Japanese, Korean, Chinese, and Thai. In U. Connor & A. Johns (Eds.), Coherence in writing (pp. 87–110). Alexandria, VA: TESOL.
25. Hinkel, E. (1997). Indirectness in L1 and L2 academic writing. Journal of Pragmatics, 27(3), 360–386.
26. Hinkel, E. (2002). Second language writers' text: Linguistic and rhetorical features. Mahwah, NJ: Lawrence Erlbaum.
27. Holland, P., & Thayer, D. (1988). Differential item performance. In H. Wainer & H. Braun (Eds.), Test validity (pp. 129–145). Hillsdale, NJ: Lawrence Erlbaum.
28. Hosmer, D. W., & Lemeshow, S. (1989). Applied logistic regression. New York: Wiley.
29. Kim, M. (2001). Detecting DIF across the different language groups in a speaking test. Language Testing, 18, 89–114.
30. Kincaid, L. (Ed.). (1987). Communication theory: Eastern and Western perspectives. San Diego, CA: Academic Press.
31. Jodoin, M. G., & Gierl, M. J. (2001). Evaluating Type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329–349.
32. Lynch, B. K., & McNamara, T. F. (1998). Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing, 15(2), 158–80.
33. Miller, M. D., & Linn, R. L. (2000). Validation of performance-based assessments. Applied Psychological Measurement, 24, 367–378.
34. Miller, T., & Spray, J. A. (1993). Logistic discriminant function analysis for DIF identification of polytomously scored items. Journal of Educational Measurement, 30, 107–122.
35. Muraki, E. (1999). Stepwise analysis of differential item functioning based on multiple group partial credit model. Journal of Educational Measurement, 36(3), 217–232.
36. Oltman, P. K., Stricker, L. J., & Barrows, T. (1988). Native language, English proficiency, and the structure of the Test of English as a Foreign Language for several language groups. (TOEFL Research Report No. 27). Princeton, NJ: ETS.
37. Penfield, R. D. (2001). Assessing differential item functioning among multiple groups: A comparison of three Mantel-Haenzel Procedures. Applied Measurement in Education, 14(3), 235–259.
38. Penfield, R. D., & Lam, T. C. M. (2000). Assessing differential item functioning in performance assessment: Review and recommendations. Educational Measurement: Issues and Practice, 19(3), 5–15.
39. Potenza, M. T., & Dorans, N. J. (1995). DIF assessment for polytomously scored items: A framework for classification and evaluation. Applied Psychological Measurement, 19, 23–37.
40. Powers, D. E., & Fowles, M. E. (1998). Test takers' judgments about GRE writing test prompts (ETS RR-98-36). Princeton, NJ: ETS.
41. Prentice, D. A., & Miller, D. T. (1992). When small effects are impressive. Psychological Bulletin, 112(1), 160–164.
42. Roussos, L. A., & Stout, W. F. (1996). Simulation studies of the effects of small sample size and studied item parameters on SIBTEST and Mantel-Haenzel Type I error performance. Journal of Educational Measurement, 33, 215–230.
43. Ryan, K. E., & Bachman, L. (1992). Differential item functioning on two tests of EFL proficiency. Language Testing, 9(1), 12–29.
44. Samejima, F. (1997). Estimation of ability using a response pattern of graded scores. Psychometrika Monograph, 17.
45. th ed., Vol. 2). Cary, NC: Author.
46. Sasaki, M. (1991). A comparison of two methods for detecting differential item functioning in an ESL placement. Language Testing, 8(2), 95–111.
47. Scollon, R., & Scollon, S. W. (1991). Topic confusion in English-Asian discourse. World Englishes, 10(2), 113–125.
48. Sheppard, L. A. (1982). Definition of bias. In R. Berk (Ed.), Handbook of methods for detecting test bias (pp. 9–30). Baltimore, MD: John Hopkins University.
49. Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361–370.
50. Swinton, S. S., & Powers, D. E. (1980). Factor analysis of the Test of English as a Foreign Language for several language groups (TOEFL Research Report No. 6). Princeton, NJ: ETS.
51. Taylor, I. (1995). Writing and literacy in Chinese, Korean, and Japanese. Amsterdam: John Benjamins.
52. Wainer, H., Serici, S. G., & Thissen, D. (1991). Differential testlet functioning: Definitions and detection. Journal of Educational Measurement, 28, 197–219.
53. Wolfe, E. W., & Manalo, J. R. (2001). An investigation of the impact of composition medium on the quality of scores from the TOEFL writing section: A report from the broad-based study. Manuscript submitted for publication.
54. Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, Ontario, Canada: Directorate of Human Resources Research and Evaluation, Department of National Defense.
55. Zwick, R., Donoghue, J. R., & Grima, A. (1993). Assessment of differential item functioning for performance tasks. Journal of Educational Measurement, 30, 233–251.