Effective rating scale development for speaking tests: Performance decision trees

Language Testing

Volumn 28, Issue 1, 2011, Pages 5-29

  Fulcher, Glenn ^a   Davidson, Fred ^b   Kemp, Jenny ^a  

^a UNIVERSITY OF LEICESTER   (United Kingdom)

^b UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

domain analysis; language testing; performance decision trees; rating scales; test design; validity

Indexed keywords

EID: 79751496059     PISSN: 02655322     EISSN: 14770946     Source Type: Journal    
DOI: 10.1177/0265532209359514     Document Type: Article

References (67)

1. Alderson JC (1991). Bands and scores. In Alderson JC, North B (Eds.), Language testing in the 1990s. London: Modern English Publications and the British Council.
2. Bachman LF (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.
3. Baker R. (1997). Classical test theory and item response theory in test analysis. Lancaster: Department of Linguistics and Modern English Language.
4. Bastos LC (1996). Power, solidarity and the construction of requests in service encounters. ESPecialist, 17(2), 151-174.
5. Biber D., Conrad S., Reppen R., Byrd P., and Helt M. (2002). Speaking and writing in the university. A multidimensional comparison. TESOL Quarterly, 36(1), 9-48.
6. Brooks L. (2009). Interacting in pairs in a test of oral proficiency: Co-constructing a better performance. Language Testing, 26(3), 341-366.
7. Canale M. (1983a). From communicative competence to communicative language pedagogy. In Richards C, Schmidt RW (Eds.) Language and communication (pp. 2-27). London: Longman.
8. Canale M. (1983b). On some dimensions of language proficiency. In Oller JW (Ed.), Issues in language testing research (pp. 333-342). Rowley, MA: Newbury House.
9. Canale M., Swain M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 1-47.
10. Chalhoub-Deville M. (2003). Second language interaction: Current perspectives and future trends. Language Testing, 20(4), 369-383.
11. Chandon JL, Piere-Yves L., and Philippe J. (1997). Service encounter dimensions - a dyadic perspective: Measuring the dimensions of service encounters as perceived by customers and personnel. International Journal of Service Industry Management, 8(1), 65-86.
12. Council of Europe (2001). Common European Framework of Reference for Languages: Learning, teaching and assessment. Cambridge: Cambridge University Press. Available online at: www.coe.int/t/dg4/linguistic/Source/Framework_EN.pdf.
13. Coupland N. (1983). Patterns of encounter management: Further arguments for discourse variables. Language in Society, 12, 459-476.
14. Coupland N., Ylänne-McEwen V. (2000). Talk about the weather: Small talk, leisure talk, and the travel industry. In Coupland J (Ed.), Small talk (pp. 163-182). London: Longman.
15. Davidson F., Fulcher G. (2007). The Common European Framework of Reference (CEFR) and the design of language tests: A matter of effect. Language Teaching, 40(3), 231-241.
16. Fairclough N. (1995). Critical discourse analysis: The critical study of language. London: Longman.
17. Fulcher G. (1987). Tests of oral performance: The need for data-based criteria. English Language Teaching Journal, 41(4), 287-291.
18. Fulcher G. (1993). The construction and validation of rating scales for oral tests in English as a Foreign Language. Unpublished PhD thesis, University of Lancaster, UK.
19. Fulcher G. (1996). Does thick description lead to smart tests? A data-based approach to rating scale construction. Language Testing, 13(2), 208-238.
20. Fulcher G. (2003). Testing second language speaking. London: Longman/Pearson.
21. Fulcher G. (2008). Criteria for evaluating language quality. In Shohamy E (Ed.), Language testing and assessment. Encyclopedia of Language and Education, Vol. 7 (pp. 157-176). Amsterdam: Springer.
22. Fulcher G., Davidson F. (2007). Language testing and assessment. London and New York: Routledge.
23. Fulcher G., Davidson F. (2009). Test architecture, test retrofit. Language Testing, 26(1), 123-144.
24. Gabbott M., Hogg G. (2000). An empirical investigation of the impact of non-verbal communication on service evaluation. European Journal of Marketing, 34(3/4), 384-398.
25. Gremler DD, Gwinner KP (2000). Customer-employee rapport in service relationships. Journal of Service Research, 3(1), 82-104.
26. Grönroos C. (1993). Toward a third phase in service quality research: Challenges and future directions. In Swartz AT, Bowen DE, and Brown SW (Eds.), Advances in service marketing management, Vol. II (pp. 49-64). Greenwich CT: JAI Press.
27. Halliday Mak (1985). Context of situation. In Halliday MAK, Hasan R, Language, context, and text: Aspects of language in a social-semiotic perspective (pp. 3-14). Victoria, Australia: Deakin University Press.
28. Hamp-Lyons L. (1991). Scoring procedures for ESL contexts. In Hamp-Lyons, L. (Ed.), Assessing second language writing in academic contexts (241-276). Norwood, NJ: Ablex.
29. Hasan R. (1985). The structure of a text. In Halliday MAK, Hasan R (1985). Language, context, and text: Aspects of language in a social-semiotic perspective (pp. 52-69). Victoria, Australia: Deakin University Press.
30. Hulstijn JA (2007). The shaky ground beneath the CEFR: Quantitative and qualitative dimensions of language proficiency. The Modern Language Journal, 91(4), 663-667.
31. Jacoby S., McNamara, T. (1999). Locating competence. English for Specific Purposes, 18(3), 213-241.
32. Jacoby S., Ochs E. (1995). Co-construction: An introduction. Research on Language and Social Interaction, 28(3), 171-183.
33. Keaveney SM (1995). Customer switching behavior in service industries: An exploratory study. Journal of Marketing, 59(2), 71-82.
34. Kramsch C. (1986). From language proficiency to interactional competence. The Modern Language Journal, 70(4), 366-372.
35. Lowe P. (1986). Proficiency: Panacea, framework, process? A reply to Kramsch, Schulz, and particularly Bachman and Savignon. Modern Language Journal, 70(4), 391-397.
36. Lumley T., Lynch B., and McNamara T. (1994). A new approach to standard-setting in language assessment. Melbourne Papers in Language Testing, 3(2), 19-40.
37. May LA (2007). Interaction in a paired speaking test: The rater's perspective. Unpublished PhD thesis, School of Languages and Linguistics, University of Melbourne, Australia.
38. McCarthy M. (2000). Captive audiences: The discourse of close contact service encounters. In Coupland J (Ed.), Small talk (pp. 84-109). London: Longman.
39. McCarthy M. (2003). Talking back: 'Small' interactional response tokens in everyday conversation. Research on Language and Social Interaction, 36(1), 33-63.
40. McCarthy M., Carter R. (1994). Language as discourse: Perspectives for language teaching. London: Longman.
41. Meisel JM (1980). Linguistic simplification. In Felix S (Ed.), Second language development: Trends and issues (pp. 13-40). Tübingen: Gunter Narr.
42. Mills S. (2009). An evaluation of a paired format oral test for Korean learners of English for Hotel and Tourism. University of Leicester: Unpublished MA dissertation.
43. Mitchell TF (1957/1978). The language of buying and selling in Cyrenaica: A situational statement. Hespėris XLIV: 31-71. Reprinted in TF Mitchell (1978). Principles of Firthian linguistics (pp. 167-200). London: Longman.
44. North B. (1993). Scales of language proficiency: A survey of some existing systems. Strasbourg: Council of Europe, Council for Cultural Co-operation, CC-LAND 94 24.
45. North B. (1995). The development of a common framework scale of descriptors of language proficiency based on a theory of measurement. System, 23(4), 445-465.
46. North B. (1996). The development of a common framework scale of descriptors of language proficiency based on a theory of measurement. In Huhta A, Kohonen V, Kurki-Suonio L, and Luoma S (Eds.), Current developments and alternatives in language assessment. Proceedings of the LTRC 1996 (pp. 423-447). Jyvaskyla: University of Jyvaskyla Press.
47. North B. (2000). The development of a common framework scale of language proficiency. New York, Peter Lang.
48. North B., Schneider G. (1998). Scaling descriptors for language proficiency scales. Language Testing, 15(2), 217-263.
49. Parasuraman A., Zeithaml AV, and Berry LL (1988). SERVQUAL: A multiple-item scale for measuring customer perceptions of service quality, Journal of Retailing, 64, 12-40.
50. Ryoo H.-K(2005). Achieving friendly interactions: a study of service encounters between Korean shopkeepers and African-American customers. Discourse and Society, 16(1), 79-105.
51. Shiel C., Bowen DE, and Pearson CM (1992). Service encounters as rites of integration: An information processing model. Organizational Science, 3(4), 537-555.
52. Spencer WJ (1997). 'We don't pay for bus tickets, but we can help you find work': The micropolitics of trouble in human service encounters. The Sociological Quarterly, 38(1), 185-203.
53. Stern BB, Thompson CJ, and Arnould EJ (1998). Narrative analysis of a marketing relationship: The consumer's perspective. Psychology and Marketing, 15(3), 195-214.
54. Storbacka K., Standvik T., and Grönroos, C. (1994). Managing customer relationships for profit: The dynamics of relationship quality. International Journal of Service Industry Management, 5(5), 21-38.
55. Suprenant CF, Solomon MR (1987). Predictability and personalization in the service encounter. Journal of Marketing, 51(2), 86-96.
56. Swain M. (2001). Examining dialogue: Another approach to content specification and to validating inferences drawn from test scores. Language Testing, 18(3), 275-302.
57. Turner CE (2000). Listening to the voices of rating scale developers: Identifying salient features for second language performance assessment. Canadian Modern Language Review, 56(4), 555-584.
58. Turner CE, Upshur J. (2002). Rating scales derived from student samples: Effects of the scale marker and the student sample on scale content and student scores. TESOL Quarterly, 36(1), 49-70.
59. Upshur J., Turner CE (1995). Constructing rating scales for second language tests. English Language Teaching Journal, 49(1), 3-12.
60. Upshur J., Turner CE (1999). Systematic effects in the rating of second-language speaking ability: Test method and learner discourse.' Language Testing, 16(1), 82-111.
61. van Lier L. (1989). Reeling, writhing, drawling, stretching, and fainting in coils: Oral Proficiency Interviews as conversation. TESOL Quarterly, 23(3), 489-508.
62. Ventola E. (1987). The structure of social interaction. London: Francis Pinter.
63. Ventola E. (2005). Revisiting service encounter genre: Some reflections. Folia Linguistica, 39(1-2), 19-43.
64. Westhoff G. (2007). Challenges and opportunities of the CEFR for reimagining foreign language pedagogy. The Modern Language Journal, 91(4), 676-679.
65. Wilds C. (1975). The oral interview test. In Jones RL, Spolsky B (Eds.), Testing language proficiency (pp. 29-44). Arlington, VA: Center for Applied Linguistics.
66. Ylänne-McEwen V. (2004). Shifting alignment and negotiating sociality in travel agency discourse. Discourse Studies, 6(4), 517-536.
67. Zeithaml VA (2000). Service quality, profitability, and the economic worth of customers: What we know and what we need to learn. Journal of the Academy of Marketing Science, 28(1), 67-85.