COMPARABILITY OF TOEFL CBT WRITING PROMPTS: RESPONSE MODE ANALYSES

ETS Research Report Series

Volumn 2004, Issue 1, 2004, Pages i-39

  Breland, Hunter ^a   Lee, Yong Won ^a   Muraki, Eiji ^b  

^a EDUCATIONAL TESTING SERVICE   (United States)

^b TOHOKU UNIVERSITY   (Japan)

Author keywords

comparability; computer based writing assessment; essay prompts; fairness; logistic regression; proportional odds ratio model; response mode

Indexed keywords

EID: 85164518151     PISSN: None     EISSN: 23308516     Source Type: Journal    
DOI: 10.1002/j.2333-8504.2004.tb01950.x     Document Type: Article

References (36)

1. Agresti, A. (1990). Categorical data analysis. New York: Wiley.
2. Bridgeman, B., & Cooper, P. (1998). Comparability of scores on word-processed and handwritten essays on the Graduate Management Admissions Test. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
3. Cochran-Smith, M., Paris, C. L., & Kahn, J. (1991). Learning to write differently. Norwood, NJ: Ablex.
4. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
5. Dalton, D., & Hannafin, M. (1987). The effects of word processing on written composition. Journal of Educational Research, 50, 223–228.
6. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis. Verbal reports as data (2nd ed.). Cambridge, MA: MIT Press.
7. ETS. (1998). Computer-based TOEFL score user guide. Princeton, NJ: Author.
8. Gentile, C. (1999). An investigation of the impact of composition medium on the quality of scores from the TOEFL writing section: A report from the pilot focused study. Manuscript in preparation.
9. Gentile, C., Riazantseva, A., & Cline, F. (2001). A comparison of handwritten and word-processed TOEFL essays. Princeton, NJ: Educational Testing Service.
10. Green, A. J. K. (1997). Verbal protocol analysis in language testing research. Cambridge: Cambridge University Press.
11. French, A. W., & Miller, T. R. (1996). Logistic regression and its use in detecting differential item functioning in polytomous items. Journal of Educational Measurement, 33, 315–332.
12. Hawisher, G. (1987). The effects of word processing on the revision strategies of college freshmen. Research in the Teaching of English, 21, 145–159.
13. Hollenbeck, K., Tindal, G., Stieber, S., & Harniss, M. (1999). Handwritten vs. word processed statewide compositions: Do judges rate them differently? Retrieved January 1, 2003, from the University of Oregon Behavioral Research and Teaching Program Web site: http://brt.uoregon.edu/upload/HdwrtnVsTyped.pdf.
14. Hosmer, D. W., & Lemeshow, S. (1989). Applied logistic regression. New York: Wiley.
15. Jodion, M. G., & Gierl, M. J. (2001). Evaluating Type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329–349.
16. Kirsch, I., Taylor, C., Jamieson, J., & Eignor, D. (1998). Computer familiarity among TOEFL examinees (TOEFL Research Report No. 59). Princeton, NJ: ETS.
17. Kurth, R. J. (1987). Using word processing to enhance revision strategies during student writing activities. Educational Technology, 127(1), 13–19.
18. Linacre, J. M. (1989). Many-facet Rasch measurement. Chicago: MESA Press.
19. Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246–276.
20. Lutz, J. A. (1987). A study of professional and experienced writers revising and editing at the computer and with pen and paper. Research in the Teaching of English, 21, 398–421.
21. Manalo, J. R., & Wolfe, E. W. (2000a). A comparison of word-processed and hand-written essays written for the Test of English as a Foreign Language. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
22. Manalo, J. R., & Wolfe, E. W. (2000b). The impact of composition medium on essay raters in foreign language testing. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
23. Porter, R. (1986). Writing and word processing in year one. Australian Educational Computing, 1, 18–23.
24. Powers, D. E., Fowles, M. E., Farnum, M., & Ramsey, P. (1994). Will they think less of my handwritten essay if others word process theirs? Effects on essay scores of intermingling handwritten and word-processed essays. Journal of Educational Measurement, 31(3), 220–233.
25. Roussos, L. A., & Stout, W. F. (1996). Simulation studies of the effects of small sample size and studied item parameters on SIBTEST and Mantel-Haenzel Type I error performance. Journal of Educational Measurement, 33(2), 215–230.
26. Russell, M., & Haney, W. (1997). Testing writing on computers: An experiment comparing student performance on tests conducted via computer and via paper-and-pencil. Education Policy Analysis Archives, 5(3), 1–18. Retrieved on December 1, 2003, from http://epaa.asu.edu/epaa/v5n3.html
27. Samejima, F. (1997). Estimation of ability using a response pattern of graded scores. Psychometrika Monograph, 17.
28. SAS Institute. (1990). SAS/STAT user's guide, version six (4th ed., Vol. 2). Cary, NC: Author.
29. Swaminathan, H. & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361–370.
30. Taylor, C., Jamieson, J., Eignor, D., & Kirsch, I. (1998). The relationship between computer familiarity and performance on computer-based TOEFL test tasks (TOEFL Research Report No. 61). Princeton, NJ: ETS.
31. Wiegle, S. C. (1994). Effects of training on raters of ESL compositions. Language Testing, 11, 197–223.
32. Wiegle, S. C. (1998). Using FACETS to model rater training effects. Language Testing, 15, 263–287.
33. Wolfe, E. W., Bolton, S., Feltovich, B., & Welch, C. (1993). A comparison of word-processed and handwritten essays from a standardized writing assessment (ACT Research Report No. 93-8). Iowa City, IA: American College Testing Program.
34. Wolfe, E. W., Bolton, S., Feltovich, B., & Bangert, A. W. (1996). A study of word processing experience and its effects on student essay writing. Journal of Educational Computing Research, 14(3), 269–283.
35. Wolfe, E. W., & Manalo, J. R. (2001). An investigation of the impact of composition medium on the quality of scores from the TOEFL writing section: A report from the broad-based study. Princeton, NJ: ETS.
36. Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, Ontario, Canada: Directorate of Human Resources Research and Evaluation, Department of National Defense.