Impact of rater characteristics and prosodic features of speaker accentedness on ratings of international teaching assistants' oral performance

Language Assessment Quarterly

Volumn 9, Issue 3, 2012, Pages 249-269

  Kang, Okim ^a  

^a NORTHERN ARIZONA UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84865700995     PISSN: 15434303     EISSN: 15434311     Source Type: Journal    
DOI: 10.1080/15434303.2011.642631     Document Type: Article

References (106)

1. Anderson-Hsieh, J., Johnson, R., & Koehler, K. (1992). The relationship between native speaker judgments of nonnative pronunciation and deviance in segmentals, prosody and syllable structure. Language Learning, 42, 529-555.
2. Anderson-Hsieh, J., & Koehler, K. (1988). The effect of foreign accent and speaking rate on native speaker comprehension. Language Learning, 38, 561-613.
3. Bachman, L. F., Lynch, B. K., &Mason, M. (1995). Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing, 12, 238-247.
4. Barnwell, D. Naïve native speakers and judgments of oral proficiency in Spanish. Language Testing, 6, 152-163.
5. Bhat, S., Hasegawa-Johnson, M., & Sproat, R. (2010). Automatic fluency assessment by signal-level measurement of spontaneous speech. 2010 INTERSPEECH Satellite Workshop on Second Language Studies: Acquisition, Learning, Education and Technology, Makuhari, Japan.
6. Boersma, P., & Weenink, D. (2007). PRAAT for Windows. Available from http://www.fon. hum. uva. nl/praat/download-win.html
7. Bond, Z. (1999). Slips of the ear: Errors in the perception of casual conversation. San Diego, CA: Academic Press.
8. Borjas, G. J. (2000). Foreign-born teaching assistants and the academic performance of undergraduates. American Economic Review, 90, 355-359.
9. Boulet, J. R., van Zanten, M., McKinley, D. W., & Gary, N. E. (2001). Evaluating the spoken English proficiency of graduates of foreign medical schools. Medical Education, 35, 767-773.
10. Brazil, D. (1997). The communicative value of intonation in English. New York, NY: Cambridge University Press.
11. Brennan, E., & Brennan, J. (1981). Accent scaling and language attitudes: Reactions to Mexican-American English speech. Language and Speech, 24, 207-221.
12. Bresnahan, M. J., Ohashi, R., Nebashi, R., Liu, W. Y., & Shearman, S. M. (2002). Attitudinal and affective response toward accented English. Language and Communication, 22, 171-185.
13. Brown, A. (1995). The effect of rater variables in the development of an occupation-specific language performance test. Language Testing, 12, 1-15.
14. Brown, A. (2007). An investigation of the rating process in the IELTS oral interview. In L. Taylor & P. Falvey (Eds.), Research in speaking and writing assessment (IELTS collected papers; (pp. 98-141). Cambridge, UK: Cambridge University Press.
15. Brown, A., Iwashita, N., & McNamara, T. (2005). An examination of rater orientations and test-Taker performance on English-for-Academic-Purposes speaking tasks (TOEFL Research Rep. No. RR-05-05). Princeton, NJ: Educational Testing Service.
16. Brown, G. (1977). Listening to spoken English. London, UK: Longman.
17. Brown, G., & Yule, G. (1983). Discourse analysis. Cambridge, UK: Cambridge University Press.
18. Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.
19. Chalhoub-Deville, M. (1995). Deriving oral assessment scales across different tests and rater groups. Language Testing, 12, 16-33.
20. Chalhoub-Deville, M., & Wigglesworth, G. (2005). Rater judgment and English language speaking proficiency. World Englishes, 24, 389-391.
21. Clarke, C. M., & Garrett, M. F. (2004). Rapid adaptation to foreign accented English. Journal of the Acoustical Society of America, 116, 3647-3658.
22. Crocker, L., & Algina, J. (1986). Introduction to classical & modern test theory. Orlando, FL: Holt, Rinehart andWinston.
23. Crystal, D. (2003). A dictionary of linguistics and phonetics. Malden, MA: Blackwell.
24. Cutler, A., Dahan, D., & Donselaar, W. (1997). Prosody in the comprehension of spoken language: A literature review. Language and Speech, 40, 141-201.
25. Cumming, A. (1990). Expertise in evaluating second language compositions. Language Testing, 7, 31-51.
26. Davis, W. (1981). Comparing language proficiency scores and student evaluations to determine policy for international teaching assistants. College Student Journal, 25, 489-495.
27. De Jong, N. H., & Wempe, T. (2007). Automatic measurement of speech rate in spoken Dutch. ACLC Working Papers, 2, 51-60.
28. Derwing, T. M. (1990). Speech rate is no simple matter. Studies in Second Language Acquisition, 12, 303-313.
29. Derwing, T. M., & Munro, M. J. (1997). Accent, intelligibility, and comprehensibility: Evidence from four L1s. Studies in Second Language Acquisition, 19, 1-16.
30. Derwing, T., & Munro, M. (2005). Second language accent and pronunciation teaching: A research-based approach. TESOL Quarterly, 39, 379-397.
31. Derwing, T. M., Rossiter, M. J., Munro, M. J., & Thomson, R. I. (2004). Second language fluency: Judgments on different tasks. Language Testing, 54, 655-679.
32. Deterding, D., & Kirkpatrick, A. (2006). Intelligibility and an emerging ASEAN English lingua franca, World Englishes, 25, 391-410.
33. Erdosy, M. U. (2004). Exploring variability in judging writing ability in a second language: A study of four experienced raters of ESL composition (ETS Research Reports RR-03-17). Princeton, NJ: Educational Testing Service.
34. Fayer, J. M., & Krasinski, E. (1987). Native and nonnative judgments of intelligibility and irritation. Language Learning, 37, 313-326.
35. Field, J. (2005). The fuzzy notion of 'intelligibility': A headache for pronunciation teachers and oral testers. IATEFL Special Interest Groups Newsletter, (pp. 34-38).
36. Fitch, F., & Morgan, S. E. (2003). "Not a lick of English": Constructing the ITA identity through student narratives. Communication Education, 52, 297-310.
37. Flege, J. E. (1991). Age of learning affects the authenticity of voice-onset time (VOT) in stop consonants produced in a second language. Journal of Acoustical Society of America, 89(1), 395-411.
38. Fleisher, B., Hashimoto, M., & Weinberg, B. (2002). Foreign GTAs can be effective teachers of economics. Journal of Economic Education33, 299-325.
39. Fox, W., & Gay, G. (1994). Functions and effects of international teaching assistants. Review of Higher Education, 18, 1-24.
40. Freed, B. F. (2000). Is fluency, like beauty, the eyes, of the beholder? In H. Riggenbach (Ed.), Perspectives on fluency (pp. 243-265). Ann Arbor: University of Michigan Press.
41. Fulcher, G., 1996. Does thick description lead to smart tests? A data-based approach to rating scale construction. Language Testing, 13, 208-238.
42. Gallego, J. C. (1990). The intelligibility of three non-native English speaking teaching assistants: An analysis of student reported communication. Issues in Applied Linguistics, 1, 219-237.
43. Galloway, V. B. (1980). Perceptions of the communicative efforts of American students of Spanish. Modern Language Journal, 64, 428-433.
44. Gass, S. M., & Varonis, E. M. (1984). The effect of familiarity on the comprehensibility of nonnative speech. Language Learning, 34, 65-89.
45. Gayle, G. (1980). A descriptive analysis of second language teaching styles in the oral approach. Ottawa, Canada: University of Ottawa Press.
46. Griffiths, R. (1992). Speech rate and listening comprehension: Further evidence of the relationship. TESOL Quarterly, 26, 385-391.
47. Hadden, B. (1991). Teacher and nonteacher perceptions of second-language communication. Language Learning, 41, 1-24.
48. Hair, J. F., Jr., & Black, W. C. (2000). Cluster analysis. In L. G. Grimm & P. R. Yarnold (Eds.), Reading and understanding more multivariate statistics (pp. 147-205). Washington, DC: American Psychological Association.
49. Inglis, M. (1993). The communicator style measure applied to nonnative speaking teaching assistants. International Journal of Intercultural Relations, 17, 89-105.
50. Isaacs, T. (2008). Towards defining a valid assessment criterion of pronunciation proficiency in non-native Englishspeaking graduate students. The Canadian Modern Language Review, 64, 555-580.
51. Isaacs, T., & Trofimovich, P. (2010). Falling on sensitive ears? The influence of musical ability on extreme raters' judgments of L2 pronunciation. TESOL Quarterly, 44, 375-386.
52. Iwashita, N., Brown, A., McNamara, T., & O'Hagan, S. (2008). Assessed levels of second language speaking proficiency: How distinct? Applied Linguistics, 29, 24-49.
53. Juffs, A. (1990). Tone, syllable structure and interlanguage phonology: Chinese learners' stress errors. International Review of Applied Linguistics, 28, 99-117.
54. Kang, O. (2008). Ratings of L2 oral performance in English: Relative impact of rater characteristics and acoustic measures of accentedness (Unpublished doctoral dissertation). University of Georgia, Athens.
55. Kang, O. (2010). Relative salience of suprasegmental features on judgments of L2 comprehensibility and accentedness. System, 38, 301-315.
56. Kang, O., & Rubin, D. L. (2009). Reverse linguistic stereotyping: Measuring the effect of listener expectations on speech evaluation. Journal of Language and Social Psychology, 28, 441-456.
57. Kang, O., Rubin, D. L., & Pickering L. (2010). Suprasegmental measures of accentedness and judgments of English language learner proficiency in oral English. Modern Language Journal, 94, 554-566.
58. Kennedy, S., & Trofimovich, P. (2008). Intelligibility, comprehensibility, and accentedness of L2 speech: The role of listener experience and semantic context. Canadian Modern Language Review, 64, 459-490.
59. Kim, Y-H. (2009). An investigation into native and non-native teachers' judgments of oral English performance: A mixed methods approach. Language Testing, 26, 187-217.
60. Kirkpatrick, A. (2007). World Englishes. Cambridge, UK: Cambridge University Press.
61. Kormos, J., &Dénes, M. (2004). Exploringmeasures and perceptions of fluency in the speech of second language learners. System, 32, 145-164.
62. Levis, J., & Pickering, L. (2004). Teaching intonation in discourse using speech visualization technology. System, 32, 505-524.
63. Lindemann, S. (2002). Listening with an attitude: A model of native-speaker comprehension of non-native speakers in the United States. Language in Society, 31, 419-441.
64. Lindemann, S. (2003). Koreans, Chinese, or Indians? Attitudes and ideologies about non-native English speakers in the United States. Journal of Sociolinguistics, 7, 348-364
65. Lippi-Green, R. (1997). English with an Accent: Language, ideology, and discrimination in the United States. NewYork, NY: Routledge.
66. Littell, R., Milliken, G., Stroup, W., Wolfinger, R., & Schabenberger, O. (2007). SAS for mixed models. Cary, NC: SAS Institute.
67. Lumley, T., & McNamara, T. F. (1995). Rater characteristics and rater bias: implications for training. Language Testing, 12, 54-71.
68. Lynch, B. K., & McNamara, T. F. (1998). Using g-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing, 15, 158-180.
69. Major, R., Fitzmaurice, S., Bunta, F., & Balasubramanian, C. (2002). The effects of nonnative accents on listening comprehension: Implications for ESL assessment. TESOL Quarterly, 36, 173-190.
70. Mattran, K. J. (1977). Native speaker reactions to speakers of ESL: Implications for adult basic education oral English proficiency testing. TESOL Quarterly, 11, 407-414.
71. McNamara, T. (1996). Measuring second language performance. London, UK: Longman.
72. Moineddin, R., Matheson, F. I., & Glazier, R. H. (2007). A simulation study of sample size for multilevel logistic regression models. BMC Med Res Methodology, 16, 7-34.
73. Munro, M. J. (1995). Nonsegmental factors in foreign accent: Rating of filleted speech. Studies in Second Language Acquisition, 17, 17-34.
74. Munro, M. T. D., & Derwing, T. M. (1995). Foreign accent, comprehensibility, and intelligibility in the speech of second language learners. Language Learning, 45, 73-97.
75. Munro, M. J., Derwing, T. M., & Morton, S. L. (2006). The mutual intelligibility of L2 speech. Studies in Second Language Acquisition, 28, 111-131.
76. Myford, C. M., &Wolfe, E. (2000). Monitoring sources of variability within the Test of Spoken English assessment system (TOEFL Research Rep. No. RR-00-6). Princeton, NJ: Educational Testing Service.
77. O'Loughlin, K. (2007). An investigation into the role of gender in the IELTS oral interview. In L. Taylor & P. Falvey (Eds.), IELTS collected papers: Research in speaking and writing assessment (pp. 63-97). Cambridge, UK: Cambridge University Press.
78. Pickering, L. (1999). An analysis of prosodic systems in the classroom discourse of native speaker and nonnative speaker teaching assistants (Unpublished doctoral dissertation). University of Florida, Gainesville, FL.
79. Pickering, L. (2001). The role of tone choice in improving ITA communication in the classroom. TESOL Quarterly, 35, 233-255.
80. Pickering, L. (2004). The structure and function of intonational paragraphs in native and nonnative speaker instructional discourse. English for Specific Purposes, 23, 19.
81. Plakans, B. (1997). Undergraduates' experience with and attitudes toward international teaching assistants. TESOL Quarterly, 31, 95-119.
82. Powers, D. E., Schedl, M. A., Wilson Leung, S. W., & Butler, F. A. (1999). Validating the revised Test of Spoken English against a criterion of communicative success. Language Testing, 16, 339-425.
83. Riggenbach, H. (1991). Towards an understanding of fluency: A microanalysis of nonnative speaker conversation. Discourse Processes, 14, 423-441.
84. Riggenbach, H. (2000). Perspectives on fluency. Ann Arbor: University of Michigan Press.
85. Ross, S. (2001). Program-defining evaluation in a decade of eclecticism. In J. C. Alderson & A. Beretta (Eds.), Evaluating second language education (pp. 167-196). Cambridge, UK: Cambridge University Press.
86. Rounds, P. (1987). Characterizing successful classroom discourse for NNS teaching assistant training. TESOL Quarterly, 21, 643-672.
87. Rubin, D. L. (1992). Nonlanguage factors affecting undergraduate's judgments of nonnative English-speaking teaching assistants. Research in Higher Education, 33, 511-531.
88. Rubin, D. L. (2002). Help! My professor (or doctor or boss) doesn't talk English! In J. Martin, T. Nakayama, & L. Flores (Eds.), Readings in intercultural communication: Experiences and contexts (pp. 127-137). Boston, MA: McGraw-Hill.
89. Sarwark, S., Smith, J., MacCallum, R., & Cascallar, E. C. (1995). A study of characteristics of the SPEAK Test (Research Rep. No. RR 94-47). Princeton, NJ: Educational Testing Service.
90. Schmidt, A. M., & Flege, J. E. (1996). Speaking rate effects on stops produced by Spanish and English monolinguals and Spanish/English bilinguals. Phonetica, 53, 162-179
91. Schuetze-Coburn, S., Shapley, M., & Weber, E. G. (1991). Units of intonation in discourse: A comparison of acoustic and auditory analyses. Language and Speech, 34, 207-234.
92. Singer, J. D. (1998). Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. Journal of Educational and Behavioral Statistics, 24, 323-355.
93. Smith, R. A., Strom, R. E., & Muthuswamy, N. (2005). Undergraduates' rating of domestic and international teaching assistants: Timing of data collection and communication intervention. Journal of Intercultural Communication Research, 34, 3-21.
94. Snijders, T. A. B., & Bosker, R. J. (1994). Modeled variance in two-level models. Sociological Methods and Research, 22, 342-363.
95. Stansfield, C. W., & Kenyon, D. M. (1992). Research on the comparability of the oral proficiency interview and the simulated oral proficiency interview. System, 20, 347-364.
96. Tench. P. (1996). The intonation systems of English. London, UK: Cassell.
97. Thompson, I. (1991). Foreign accents revisited: the English pronunciation of Russian immigrants, Language Learning, 41, 177-204.
98. Thompson, W. F. (2003). Is music unique among human activities? Bulletin of Psychology and the Arts, 4, 38-39.
99. Towell, R., Hawkins, R., & Bazergui, N. (1996). The development of fluency in advanced learners of French. Applied Linguistics, 17, 84-119.
100. Vanderplank, R. (1993). Pacing and spacing as predictors of difficulty in speaking and understanding English. English Language Teaching Journal, 47, 117-125.
101. Weigle, S. (1998). Using FACETS to model rater training effects. Language Testing, 15, 263-287.
102. Wennerstrom, A. (1994). Intonational meaning in English discourse: A study of nonnative speakers. Applied Linguistics, 15, 399-421.
103. Wennerstrom, A. (2000). The role of intonation in second language fluency. In H. Riggenbach (Ed.), Perspectives on fluency (pp. 102-127). Ann Arbor: University of Michigan.
104. Wennerstrom, A. (2001). The music of everyday speech. New York, NY: Oxford University Press.
105. Wigglesworth, G. (2001). Influences on performance in task-based oral assessments. In M. Bygate, P. Skehan, & M. Swain (Eds.), Researching pedagogic tasks second language learning, teaching and testing (pp. 186-209). Essex, UK: Longman.
106. Williams, G. (2006). Cultural, professional and personal influences on the teaching identity development of international teaching assistants (Unpublished doctoral dissertation). University of Georgia, Athens.