Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions

Language Assessment Quarterly

Volumn 10, Issue 2, 2013, Pages 135-159

  Isaacs, Talia ^a   Thomson, Ron I ^b  

^a UNIVERSITY OF BRISTOL   (United Kingdom)

^b BROCK UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84878804138     PISSN: 15434303     EISSN: 15434311     Source Type: Journal    
DOI: 10.1080/15434303.2013.769545     Document Type: Article

References (57)

1. Alderson, J. C. (1991). Bands and scores. In J. C. Alderson & B. North (Eds.), Language testing in the 1990s: The communicative legacy (pp. 71-86). London, UK: Macmillan.
2. Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford, UK: Oxford University Press.
3. Barnwell, D. (1989). Proficiency and the native speaker. ADFL Bulletin, 20, 42-46.
4. Bendig, A. W. (1953). The reliability of self-ratings as a function of the amount of verbal anchoring and of the number of categories on the scale. Journal of Applied Psychology, 37, 38-41.
5. Bongaerts, T., van Summeren, C., Planken, B., & Schils, E. (1997). Age and ultimate attainment in the pronunciation of a foreign language. Studies in Second Language Acquisition, 19, 447-465.
6. Bridgeman, B., Powers, D., Stone, E., & Mollaun, P. (2012). TOEFL iBT speaking test scores as indicators of communicative language proficiency. Language Testing, 29, 91-108.
7. Brindley, G. (1991). Defining language ability: The criteria for criteria. In S. Anivan (Ed.), Current developments in language testing (pp. 139-164). Singapore: SEAMEO.
8. Brown, A., Iwashita, N., & McNamara, T. (2005). An examination of rater orientations and test-taker performance on English for academic purposes speaking tasks (TOEFL Monograph 29). Princeton, NJ: Educational Testing Service.
9. Calloway, D. R. (1980). Accent and the evaluation of ESL oral proficiency. In J. W. Oller, Jr. & K. Perkins (Eds.), Research in language testing (pp. 102-115). Rowley, MA: Newbury House.
10. Chalhoub-Deville, M. (1995). Deriving oral assessment scales across different tests and rater groups. Language Testing, 12, 62-70.
11. Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage.
12. Cucchiarini, C., Strick, H., & Boves, L. (2002). Quantitative assessment of second language learners' fluency: Comparisons between read and spontaneous speech. Journal of the Acoustical Society of America, 111, 2862-2873.
13. Davidson, F. (1991). Statistical support for training in ESL composition rating. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 155-164). Norwood, NJ: Ablex.
14. Davidson, F., & Lynch, B. K. (2002). Testcraft: A teacher's guide to writing and using language test specifications. New Haven, CT: Yale University.
15. Derwing, T. M., & Munro, M. J. (2009). Putting accent in its place: Rethinking obstacles to communication. Language Teaching, 42, 1-15.
16. Derwing, T. M., Munro, M. J., Thomson, R. I., & Rossiter, M. J. (2009). The relationship between L1 fluency and L2 fluency development. Studies in Second Language Acquisition, 31, 533-557.
17. Derwing, T. M., Munro, M. J., & Wiebe, G. E. (1998). Evidence in favor of a broad framework for pronunciation instruction. Language Learning, 48, 393-410.
18. Derwing, T. M., Rossiter,M. J., Munro, M. J., & Thomson, R. I. (2004). Second language fluency: Judgments on different tasks. Language Learning, 54, 665-679.
19. Derwing, T. M., Thomson, R. I., & Munro, M. J. (2006). English pronunciation and fluency development in Mandarin and Slavic speakers. System, 34, 183-193.
20. DeVelle, S. (2008). The revised IELTS pronunciation scale. Research Notes, 34, 36-38.
21. Douglas, D. (1994). Quantity and quality in speaking test performance. Language Testing, 11, 125-144.
22. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.). Cambridge,MA: MIT Press.
23. Flege, J. E., & Fletcher, K. L. (1992). Talker and listener effects on degree of perceived foreign accent. Journal of the Acoustical Society of America, 91, 370-389.
24. Fulcher, G. (1996). Does thick description lead to smart tests? A data-based approach to rating scale construction. Language Testing, 13, 208-238.
25. Garner, W. R. (1960). Rating scales, discriminability, and information transmission. Psychological Review, 67, 343-352.
26. Gass, S. M., & Mackey, A. (2000). Stimulated recall methodology in second language research. Mahwah, NJ: Erlbaum.
27. Green, A. (1998). Verbal protocol analysis in language testing research: A handbook. Cambridge, UK: Cambridge University Press.
28. Hamp-Lyons, L., & Henning, G. (1991). Communicative writing profiles: An investigation of the transferability of a multiple-trait scoring instrument across ESL writing assessment contexts. Language Learning, 41, 337-373.
29. Harding, L. (2008). Accent and academic listening assessment: A study of test-taker perceptions. Melbourne Papers in Language Testing, 13, 1-33.
30. Huck, S. W. (2004). Reading statistics and research (4th ed.). Boston, MA: Pearson.
31. Isaacs, T. (2014, in press). Assessing pronunciation. In A. J. Kunnan (Ed.), The companion to language assessment (pp. 140-155). Hoboken, NJ: Wiley-Blackwell.
32. Isaacs, T., & Trofimovich, P. (2011). Phonological memory, attention control, and musical ability: Effects of individual differences on rater judgments of second language speech. Applied Psycholinguistics, 32, 113-140.
33. Isaacs, T., & Trofimovich, P. (2012). "Deconstructing" comprehensibility: Identifying the linguistic influences on listeners' L2 comprehensibility ratings. Studies in Second Language Acquisition, 34, 475-505.
34. Kang, O. (2012). Impact of rater characteristics on ratings of international teaching assistants' oral performance. Language Assessment Quarterly, 9, 249-269.
35. Kennedy, S., & Trofimovich, P. (2008). Intelligibility, comprehensibility, and accentedness of L2 speech: The role of listener experience and semantic context. Canadian Modern Language Review, 64, 459-489.
36. Lennon, P. (1990). Investigating fluency in EFL: A quantitative approach. Language Learning, 40, 387-417.
37. Linacre, J. M. (2010a). Category misfit. In Winsteps help for Rasch analysis. Retrieved from http://www.winsteps.com/winman/index.htm?table3-2.htm
38. Linacre, J. M. (2010b). A user's guide to FACETS: Rasch-model computer programs [Software manual 3.67.0]. Chicago, IL: Winsteps.com.
39. Lumley, T. (2005). Assessing second language writing: The rater's perspective. Frankfurt, Germany: Peter Lang.
40. Matell, M. S., & Jacoby, J. (1971). Is there an optimal number of alternatives for Likert scale items? Educational and Psychological Measurement, 31, 657-674.
41. McKelvie, S. J. (1978). Graphic rating scales: How many categories? British Journal of Psychology, 69, 185-202.
42. Milanovic, M., Saville, N., Pollitt, A., & Cook, A. (1996). Developing rating scales for CASE: Theoretical concerns and analyses. In M. Milanovic, N. Saville, & A. Pollitt (Eds.), Validation in language testing (pp. 15-38). Clevedon, UK: Multilingual Matters.
43. Miller, G. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information Psychological Review, 63, 81-97.
44. Munro, M. J., & Derwing, T. M. (1994). Evaluations of foreign accent in extemporaneous and read material. Language Testing, 11, 254-266.
45. Munro, M. J., & Derwing, T. M. (1999). Foreign accent, comprehensibility, and intelligibility in the speech of second language learners. Language Learning, 49, 285-310.
46. Munro, M. J., Derwing, T. M., & Morton, S. L. (2006). The mutual intelligibility of L2 speech. Studies in Second Language Acquisition, 28, 113-131.
47. Myford, C. M., & Wolfe, E. W. (2000). Strengthening the ties that bind: Improving the linking network in sparsely connected rating designs (TOEFL Tech. Rep. No. 15). Princeton, NJ: Educational Testing Service.
48. Pollitt, A., & Murray, N. L. (1996).What raters really pay attention to. In M. Milanovic & N. Saville (Eds.), Performance testing, cognition and assessment (pp. 74-91). Cambridge, UK: Cambridge University Press.
49. Preston, C. C., & Colman, A. M. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104, 1-15.
50. Rossiter, M. J. (2009). Perceptions of L2 fluency by native and non-native speakers of English. Canadian Modern Language Review, 65, 395-412.
51. Schoonen, R., Vergeer, M., & Eiting, M. (1997). The assessment of writing ability: Expert readers versus lay readers. Language Testing, 14, 157-184.
52. Southwood, M. H., & Flege, J. E. (1999). Scaling foreign accent: Direct magnitude estimation versus interval scaling. Clinical Linguistics & Phonetics, 13, 335-349.
53. Stemler, S. E., & Tsai, J. (2008). Best practices in estimating interrater reliability. In J. Osborne (Ed.), Best practices in quantitative methods (pp. 29-49). Thousand Oaks, CA: Sage.
54. Thomson, R. I., & Isaacs, T. (2011, September). Perceptions of friendliness and intelligence in L2 accented speech: The contribution of race-independent variables. Paper presented at the Pronunciation in Second Language Learning and Teaching Conference, Ames, IA.
55. Thompson, I. (1991). Foreign accents revisited: The English pronunciation of Russian immigrants. Language Learning, 41, 177-204.
56. Underhill, N. (1987). Testing spoken language: A handbook of oral testing techniques. Cambridge, MA: Cambridge University Press.
57. Zielinski, B. W. (2008). The listener: No longer the silent partner in reduced intelligibility. System, 36, 69-84.