Evaluating analytic scoring for the TOEFL® Academic Speaking Test (TAST) for operational use

Language Testing

Volumn 24, Issue 2, 2007, Pages 251-286

  Xi, Xiaoming ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 34248676846     PISSN: 02655322     EISSN: 14770946     Source Type: Journal    
DOI: 10.1177/0265532207076365     Document Type: Article

References (49)

1. Bachman, L.F. 1988: Problems in examining the validity of the ACTFL oral proficiency interview. Studies in Second Language Acquisition 10(2): 149-64.
2. Bachman, L.F. and Savignon, S.J. 1986: The evaluation of communicative language proficiency: A critique of the ACTFL Oral Interview. The Modern Language Journal 70(4): 380-90.
3. Bachman, L.F., Lynch, B.K., and Mason, M. 1995: Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing 12: 238-57.
4. Bolus, R.E., Hinofotis, F.B., and Bailey, K.M. 1982: An introduction to generalizability theory in second language research. Language Learning 32: 245-58.
5. Brennan, R.L. 1992: Elements of generalizability theory, revised edition. Iowa City, IA: American College Testing.
6. Brennan, R.L. 2000: (Mis)conceptions about generalizability theory. Educational Measurement: Issues and Practice 19(1): 5-10.
7. Brennan, R.L. 2001: Generalizability theory. New York: Springer-Velag.
8. Brennan, R.L. 2002: Performance assessments from the perspective of generalizability theory. Applied Psychological Measurement 24(4): 339-53.
9. Brennan, R.L., Gao, X., and Colton, D.A. 1995: Generalizability analyses of Work Keys Listening and Writing tests. Educational and Psychological Measurement 55: 157-76.
10. Brown, A. 1995: The effect of rater variables in the development of an occupation-specific language performance test. Language Testing 12: 1-15.
11. Brown, J.D. 1984: A norm-referenced engineering reading test. In Pugh, A.K. and J.M. Ulijn, editors, Reading for professional purposes: Studies and practices in native and foreign language. London: Heinemann Educational Books.
12. Brown, J.D. 1990: Short-cut estimators of criterion-referenced test consistency. Language Testing 7: 77-97.
13. Brown, J.D. 1993: A comprehensive criterion-referenced testing project. In Douglas, D. and C. Chapelle A new decade of language testing research. Alexandra, VA: TESOL
14. Brown, J.D. 1999: The relative importance of persons, items, subtests and languages to TOEFL test variance. Language Testing 16: 217-38.
15. Brown, J.D. and Bailey, K.M. 1984: A categorical instrument for scoring second language writing skills. Language Learning 34: 21-42.
16. Brown, J.D. and Hudson, T. 2002: Criterion-referenced language testing. Cambridge: Cambridge University Press.
17. Brown, J.D. and Ross, J.A. 1996: Decision dependability of item types, sections, tests, and the overall TOEFL test battery. In Milanovic, M. and Saville, N., editors, Performance testing, cognition and assessment. Cambridge: Cambridge University Press.
18. Butler, F.A., Eignor, D., Jones, S., McNamara, T., and Suomi, B.K. 2000: TOEFL 2000 speaking framework: A working paper (TOEFL Monograph Series No. 20). Princeton, NJ: Educational Testing Service.
19. Chalhoub-Deville, M. 2003: Second language interaction: Current perspectives and future trends. Language Testing 20: 369-83.
20. Chiu, C.W.T. 1999: Scoring performance assessments based on judgments: Utilizing meta-analysis to estimate variance components in generalizability theory for unbalanced situations. Unpublished doctoral dissertation, Michigan State University.
21. Chiu, C.W.T. 2000, April: A subdividing method for generalizability theory: Precision of measurement errors and patterns of missing data. Paper presented at the annual meeting of the American Educational Research Association (AERA), New Orleans, LA.
22. Chiu, C.W.T. and Wolfe, E.W. 2002: A method for analyzing sparse data matrices in the generalizability theory framework. Applied Psychological Measurement 26: 321-38.
23. Cronbach, L.J., Gleser, G.C., Nanda, H., and Rajaratnam, N. 1972: The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
24. Derwing, T.M. and Munro, M.J. 1997: Accent, intelligibility, and comprehensibility: Evidence from four L1s. Studies in Second Language Acquisition 19: 1-16.
25. Douglas, D. 1997: Testing speaking ability in academic contexts: Theoretical considerations (TOEFL Monograph Series No. 8). Princeton, NJ: Educational Testing Service.
26. Douglas, D. and Smith, J. 1997: Theoretical underpinnings of the Test of Spoken English revision project (TOEFL Monograph Series No. 9). Princeton, NJ: Educational Testing Service.
27. Fulcher, G. 1997: The testing of speaking in a second language. In Clapham, C. and D. Corson, editors, Encyclopedia of Language and Education: Vol 7. Language Testing and Assessment. Dordrecht, the Netherlands: Kluwer, 75-85.
28. Gass, S. and Varonis, E.M. 1984: The effect of familiarity on the comprehensibility of non-native speech. Language Learning 34: 65-89.
29. Ingram, D. and Wylie, E. 1993: Assessing speaking proficiency in the International English Language Testing System. In Douglas, D. and C. Chapelle, editors, A new decade of language testing research: Selected papers from the 1990 Language Testing Research Colloquium. Alexandria, VA: TESOL.
30. Joe, G.W. and Woodward, J.A. 1976: Some developments in multivariate generalizability. Psychometrica 41: 205-17.
31. Kunnan, A.J. 1992: An investigation of a criterion-referenced test using G-theory, and factor analysis. Language Testing 9: 30-49.
32. Lee, Y-W. 2005: Dependability of scores for a new ESL speaking test: Evaluating prototype tasks (TOEFL Research Monograph No. 28). Princeton, NJ: Educational Testing Service.
33. Lee, Y-W. and Kantor, R. 2005: Dependability of new ESL writing test scores: Evaluating prototype tasks and alternative rating schemes (TOEFL Research Monograph No. 31). Princeton, NJ: Educational Testing Service.
34. Linn, R.L. and Burton, E. 1994: Performance-based assessment: Implications of task specificity. Educational Measurement: Issues and Practice 13: 5-15.
35. Marcoulides, G.A. 1994: Selecting weighting schemes in multivariate generalizability studies. Educational and Psychological Measurement 54(91): 3-7.
36. Rosenfeld, M., Leung, S., and Oltman, P.K. 2001: The reading, writing, speaking, and listening tasks important for academic success at the undergraduate and graduate levels (TOEFL Monograph Series No. 21). Princeton, NJ: Educational Testing Service.
37. Sawaki, Y. 2002, December: Multivariate analytic approaches to validation of a speaking test. Paper presented at the Language Testing Research Colloquium, Hong Kong.
38. Sawaki, Y. 2003: A comparison of summarization and free recall as reading comprehension tasks in Web-based assessment of Japanese as a foreign language. Unpublished doctoral dissertation, University of California, Los Angeles.
39. Shavelson, R.J. and Webb, N.M. 1981: Generalizability theory: 1973-1980. British Journal of Mathematical and Statistical Psychology 34: 133-66.
40. Smith, P.L. 1980, April: Some approaches to determining the stability of estimated variance components. Paper presented at the Annual Meeting of the American Educational Research Association (AERA), Boston, MA.
41. Stansfield, C.W. and Kenyon, D.M. 1992: Research of the compatibility of the oral proficiency interview and the simulated oral proficiency interview. System 20: 347-64.
42. Underhill, N. 1987: Testing spoken English. Cambridge: Cambridge University Press.
43. Waters, A. 1996: A review of research into needs in English for academic purposes of relevance to the North American higher education context (TOEFL Monograph Series No. 6). Princeton, NJ: Educational Testing Service.
44. Weir, C.J. 1990: Communicative language testing. Englewood Cliffs, NJ: Prentice-Hall.
45. Webb, N.M. and Shavelson, R.J. 1981: Multivariate generalizability of general educational development ratings. Journal of Educational Measurement 18: 13-22.
46. Webb, N.M., Shavelson, R.J. and Maddahian, E. 1983: Multivariate generalizability theory. In Fyans, L.J., Jr., editor, Generalizability theory: Inferences and practical applications: No. 18. New directions for testing and measurement, San Francisco: Jossey-Bass.
47. Wiley, D.E. 1992: Pooling variance component estimates from random effects analyses of variance used to evaluate the generalizability of test tasks and scores. Studies of the California Assessment Program Spring 1992 Field Trials.
48. Xi, X. 2003: Investigating language performance on the graph description task in a semi-direct oral test. Unpublished doctoral dissertation, University of California, Los Angeles.
49. Xi, X. and Mollaun, P. 2006: Investigating the utility of analytic scoring for TOEFL Academic Speaking Test (TAST) (TOEFL iBT Research Report No. 1). Princeton, NJ: Educational Testing Service.