An investigation into native and non-native teachers' judgments of oral English performance: A mixed methods approach

Language Testing

Volumn 26, Issue 2, 2009, Pages 187-217

  Kim, Youn Hee ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Many faceted rasch measurement; Mixed methods; NS and NNS; Oral english performance assessment

Indexed keywords

EID: 68149161812     PISSN: 02655322     EISSN: 14770946     Source Type: Journal    
DOI: 10.1177/0265532208101010     Document Type: Article

References (42)

1. Bachman, L. F. & Palmer, A. S. (1996). Language testing in practice. Oxford: Oxford University Press.
2. Barnwell, D. (1989). 'Naïve' native speakers and judgments of oral proficiency in Spanish. Language Testing, 6, 152-163.
3. Brown, A. (1995). The effect of rater variables in the development of an occupation-specific language performance test. Language Testing, 12, 1-15.
4. Caracelli, V. J. & Greene, J. C. (1993). Data analysis strategies for mixedmethod evaluation designs. Educational Evaluation and Policy Analysis, 15, 195-207.
5. Caracelli, V. J. & Greene, J. C. (1997). Crafting mixed method evaluation designs. In Greene, J. C. & Caracelli, V. J., editors, Advances in mixedmethod evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Evaluation no. 74 (pp. 19-32). San Francisco: Jossey-Bass.
6. Chalhoub-Deville, M. (1995). Deriving oral assessment scales across different tests and rater groups. Language Testing, 12, 16-33.
7. Chalhoub-Deville, M. & Wigglesworth, G. (2005). Rater judgment and English language speaking proficiency. World Englishes, 24, 383-391.
8. Clark, J. L. D. & Swinton, S. S. (1979). An exploration of speaking proficiency measures in the TOEFL context (TOEFL Research Report No. RR-04). Princeton, NJ: Educational Testing Service.
9. Clark, J. L. D. & Swinton, S. S. (1980). The test of spoken English as a measure of communicative ability in English-medium instructional settings (TOEFL Research Report No. RR-07). Princeton, NJ: Educational Testing Service.
10. Crystal, D. (2003). English as a global language. Cambridge: Cambridge University Press.
11. Fayer, J. M. & Krasinski, E. (1987). Native and nonnative judgments of intelligibility and irritation. Language Learning, 37, 313-326.
12. Galloway, V. B. (1980). Perceptions of the communicative efforts of American students of Spanish. Modern Language Journal, 64, 428-433.
13. Graddol, D. (1997). The future of English?: A guide to forecasting the popularity of English in the 21st century. London, UK: The British Council.
14. Greene, J. C., Caracelli, V. J. & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation design. Educational Evaluation and Policy Analysis, 11, 255-274.
15. Hadden, B. L. (1991). Teacher and nonteacher perceptions of second-language communication. Language Learning, 41, 1-24.
16. Hill, K. (1997). Who should be the judge?: The use of non-native speakers as raters on a test of English as an international language. In Huhta, A., Kohonen, V., Kurki-Suonio, L., & Luoma, S., editors, Current developments and alternatives in language assessment: Proceedings of LTRC 96 (pp. 275-290). Jyväskylä: University of Jyväskylä and University of Tampere.
17. Jenkins, J. (2003). World Englishes: A resource book for students. New York: Routledge.
18. Johnson, B. & Turner, L. A. (2003). Data collection strategies in mixed methods research. In Tashakkori, A. & Teddlie, C., editors, Handbook of mixed methods in social and behavioral research (pp. 297-319). Thousand Oaks, CA: Sage.
19. Kachru, B. B. (1985). Standards, codification and sociolinguistic realism: The English language in the outer circle. In Quirk, R. & Widdowson, H., editors, English in the world: Teaching and learning the language and literatures (pp. 11-30). Cambridge: Cambridge University Press.
20. Kachru, B. B. (1992). The other side of English. In Kachru, B. B., editors, The other tongue: English across cultures (pp. 1-15). Urbana, IL: University of Illinois Press.
21. Kim, Y-H. (2005). An investigation into variability of tasks and teacher-judges in second language oral performance assessment. Unpublished master's thesis, McGill University, Montreal, Quebec, Canada.
22. Lee, H-K. (2007). A study on the English teacher quality as an English instructor and as an assessor in the Korean secondary school. English Teaching, 62, 309-330.
23. Linacre, J. M. (1989). Many-facet Rasch measurement. Chicago, IL: MESA Press.
24. Linacre, J. M. (2005). A user's guide to facets: Rasch-model computer programs. [Computer software and manual]. Retrieved April 10, 2005, from www.winsteps.com.
25. Linacre, J. M. & Williams, J. (1998). How much is enough? Rasch Measurement: Transactions of the Rasch Measurement SIG, 12, 653.
26. Lowenberg, P. H. (2000). Assessing English proficiency in the global context: The significance of non-native norms. In Kam, H. W., editor, Language in the global context: Implications for the language classroom (pp. 207-228). Singapore: SEAMEO Regional Language Center.
27. Lowenberg, P. H. (2002). Assessing English proficiency in the Expanding Circle. World Englishes, 21, 431-435.
28. Lunz, M. E. & Stahl, J. A. (1990). Judge severity and consistency across grading periods. Evaluation and the health professions, 13, 425-444.
29. McNamara, T. F. (1996). Measuring second language performance. London: Longman.
30. Myford, C. M. & Wolfe, E. W. (2000). Monitoring sources of variability within the test of spoken English Assessment System (TOEFL Research Report No. RR-65). Princeton, NJ: Educational Testing Service.
31. Myford, C. M. & Wolfe, E. W. (2004a). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. In Smith, Jr., E. V. & Smith, R. M., editors, Introduction toRasch measurement (pp. 460-517). Maple Grove, MN: JAM Press.
32. Myford, C. M. & Wolfe, E. W. (2004b). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. In Smith, Jr., E. V. & Smith, R. M., editors, Introduction to Rasch measurement. Maple Grove, MN: JAM Press, 518-574.
33. O'Loughlin, K. (1995). Lexical density in candidate output on direct and semi-direct versions of an oral proficiency test. Language Testing, 12, 217-237.
34. Stansfield, C. W. & Kenyon, D. M. (1992a). The development and validation of a simulated oral proficiency interview. The Modern Language Journal, 72, 129-141.
35. Stansfield, C. W. & Kenyon, D. M. (1992b). Research on the comparability of the oral proficiency interview and the simulated oral proficiency interview. System, 20, 347-364.
36. Stansfield, C. W., Kenyon, D. M., Paiva, R., Doyle, F., Ulsh, I., & Antonia, M. (1990). The development and validation of the Portuguese Speaking Test, Hispania, 73, 641-651.
37. Tashakkori, A. & Teddlie, C. editors (2003). Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage.
38. Taylor, L. B. (2006). The changing landscape of English: Implications for language assessment, ELT Journal, 60, 51-60.
39. Teddlie, C. & Yu, F. (2007). Mixed methods sampling: A typology with examples. Journal of Mixed Methods Research, 1, 77-100.
40. Underhill, N. (1987). Testing spoken language: A handbook of oral testing techniques. Cambridge: Cambridge University Press.
41. Widdowson, H. G. (1994). The ownership of English. TESOL Quarterly, 28, 377-388.
42. Wright, B. D. & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement: Transactions of the Rasch Measurement SIG, 8, 370.