Comparability of TOEFL CBT essay prompts: Response-mode analyses

Educational and Psychological Measurement

Volumn 65, Issue 4, 2005, Pages 577-595

  Breland, Hunter ^a   Lee, Yong Won ^a   Muraki, Eiji ^b  

^a EDUCATIONAL TESTING SERVICE   (United States)

^b TOHOKU UNIVERSITY   (Japan)

Author keywords

Comparability; Computer based writing assessment; Essay prompts; Fairness; Logistic regression; Proportional odds ratio model; Response mode

Indexed keywords

EID: 23444454362     PISSN: 00131644     EISSN: None     Source Type: Journal    
DOI: 10.1177/0013164404272504     Document Type: Article

References (34)

1. Agresti, A. (1990). Categorical data analysis. New York: John Wiley.
2. Bridgeman, B., & Cooper, P. (1998, April). Comparability of scores on word-processed and handwritten essays on the Graduate Management Admissions Test. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
3. Cochran-Smith, M., Paris, C. L., & Kahn, J. (1991). Learning to write differently. Norwood, NJ: Ablex.
4. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
5. Dalton, D., & Hannafin, M. (1987). The effects of word processing on written composition. Journal of Educational Research, 50, 223-228.
6. Educational Testing Service. (1998). Computer-based TOEFL score user guide. Princeton, NJ: Author.
7. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (2nd ed.). Cambridge, MA: MIT Press.
8. Gentile, C., Riazantseva, A., & Cline, F. (2001). A comparison of handwritten and word-processed TOEFL essays. Princeton, NJ: Educational Testing Service.
9. Green, A. J. K. (1997). Verbal protocol analysis in language testing research. Cambridge, UK: Cambridge University Press and University of Cambridge Local Examination Syndicate.
10. French, A. W., & Miller, T. R. (1996). Logistic regression and its use in detecting differential item functioning in polytomous items. Journal of Educational Measurement, 33, 315-332.
11. Hawisher, G. (1987). The effects of word processing on the revision strategies of college freshmen. Research in the Teaching of English, 21, 145-159.
12. Hollenbeck, K., Tindal, G., Stieber, S., & Harniss, M. (1999). Handwritten vs. word processed statewide compositions: Do judges rate them differently? Eugene, OR: University of Oregon, BRT.
13. Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression (2nd ed.). New York: John Wiley.
14. Jodoin, M. G., & Gierl, M. J. (2001). Evaluating Type I error and power rates using an effect size measure with the logistic regression procedure for DIP detection. Applied Measurement in Education, 14(4), 329-349.
15. Kirsch, I., Taylor, C., Jamieson, J., & Eignor, D. (1998). Computer familiarity among TOEFL examinees (TOEFL Research Report 59). Princeton, NJ: Educational Testing Service.
16. Kurth, R. J. (1987). Using word processing to enhance revision strategies during student writing activities. Educational Technology, 727(1), 13-19.
17. Lee, Y.-W., Breland, H., & Muraki, E. (2004). Comparability of TOEFL CBT writing prompts for different native language groups (TOEFL Research Report 77). Princeton, NJ: Educational Testing Service.
18. Linacre, J. M. (1989). Many-facet Rasch measurement. Chicago: Mesa.
19. Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246-276.
20. Lutz, J. A. (1987). A study of professional and experienced writers revising and editing at the computer and with pen and paper. Research in the Teaching of English, 21, 398-421.
21. Porter, R. (1986). Writing and word processing in year one. Australian Educational Computing, 1, 18-23.
22. Powers, D. E., Fowles, M. E., Farnum, M., & Ramsey, P. (1994). Will they think less of my hand-written essay if others word process theirs? Effects on essay scores of intermingling hand-written and word-processed essays. Journal of Educational Measurement, 31(3), 220-233.
23. Roussos, L. A., & Stout, W. F. (1996). Simulation studies of the effects of small sample size and studied item parameters on SIBTEST and Mantel-Haenzel Type I error performance. Journal of Educational Measurement, 33(2), 215-230.
24. Russell, M., & Haney, W. (1997). Testing writing on computers: An experiment comparing student performance on tests conducted via computer and via paper-and-pencil. Education Policy Analysis Archives, 5(3), 1-18.
25. SAS Institute. (1990). SAS/STAT user's guide (Ver. 6, 4th ed., Vol. 2). Cary, NC: Author.
26. Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361-370.
27. Taylor, C., Jamieson, J., Eignor, D., & Kirsch, I. (1998). The relationship between computer familiarity and performance on computer-based TOEFL test tasks (TOEFL Research Report 61). Princeton, NJ: Educational Testing Service.
28. Welch, C., & Miller, T. (1995). Assessing differential item functioning in direct writing assessments: Problems and an example. Journal of Educational Measurement, 32, 163-178.
29. Wiegle, S. C. (1994). Effects of training on raters of ESL compositions. Language Testing, 11, 197-223.
30. Wiegle, S. C. (1998). Using FACETS to model rater training effects. Language Testing, 15, 263-287.
31. Wolfe, E. W., Bolton, S., Feltovich, B., & Welch, C. (1993). A comparison of word-processed and handwritten essays from a standardized writing assessment (ACT Research Report 93-8). Iowa City, IA: American College Testing Program.
32. Wolfe, E. W., Bolton, S., Feltovich, B., & Bangert, A. W. (1996). A study of word processing experience and its effects on student essay writing. Journal of Educational Computing Research, 14(3), 269-283.
33. Wolfe, E. W., & Manalo, J. R. (2001). An investigation of the impact of composition medium on the quality of scores from the TOEFL writing section: A report from the broad-based study (Draft paper). Princeton, NJ: Educational Testing Service.
34. Zumbo, B. D. (1999). A handbook on the theory and methods of differential item Functioning (DIP): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, Canada: Directorate of Human Resources Research and Evaluation, Department of National Defense.