Raters, Functions, Item Types and the Dependability of L2 Pragmatics Tests

Investigating Pragmatics in Foreign Language Learning, Teaching and Testing

Volumn , Issue , 2008, Pages 224-248

  Brown, James Dean ^a  

^a UNIVERSITY OF HAWAII   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85071193817     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Chapter

References (52)

1. Ahn, R.C. (2005) Five measures of interlanguage pragmatics in KFL (Korean as a foreign language) learners. Unpublished PhD thesis, University of Hawaii at Manoa.
2. Bachman, L.F. (2004) Statistical Analyses for Language Assessment. Cambridge:Cambridge University Press.
3. Bachman, L.F., Lynch, B.K. and Mason, M. (1995) Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing 12 (2), 239-257.
4. Bolus, R.E., Hinofotis, F.B. and Bailey, K.M. (1982) An introduction to generalizability theory in second language research. Language Learning 32, 245-258.
5. Brown, J.D. (1982) Testing EFL reading comprehension in engineering English. Unpublished PhD thesis, University of California at Los Angeles.
6. Brown, J.D. (1984) A norm-referenced engineering reading test. In A.K. Pugh and J.M. Ulijn (eds) Reading for Professional Purposes: Studies and Practices in Native and Foreign Languages (pp. 213-222). London: Heinemann Educational Books.
7. Brown, J.D. (1988) 1987 Manoa Writing Placement Examination (Technical Report#1). Honolulu: University of Hawai’i at Manoa, Second Language Teaching and Curriculum Centre.
8. Brown, J.D. (1989) 1988 Manoa Writing Placement Examination (Technical Report#2). Honolulu: University of Hawai’i at Manoa, Second Language Teaching and Curriculum Centre.
9. Brown, J.D. (1990a) Short-cut estimates of criterion-referenced test consistency. Language Testing 7 (1), 77-97.
10. Brown, J.D. (1990b) 1989 Manoa Writing Placement Examination (Technical Report # 5). Honolulu: University of Hawai’i at Manoa, Second Language Teaching and Curriculum Centre.
11. Brown, J.D. (1991) 1990 Manoa Writing Placement Examination (Technical Report # 11). Honolulu: University of Hawai’i at Manoa, Second Language Teaching and Curriculum Centre.
12. Brown, J.D. (1993) A comprehensive criterion-referenced language testing project. In D. Douglas and C. Chapelle (eds) A New Decade of Language Testing Research (pp. 163-184). Washington, DC: TESOL.
13. Brown, J.D. (1999) Relative importance of persons, items, subtests and languages to TOEFL test variance. Language Testing 16 (2), 216-237.
14. Brown, J.D. (2000) Observing pragmatics: Testing and data gathering techniques. Pragmatics Matters 1 (2), 5-6.
15. Brown, J.D. (2001) Six types of pragmatics tests in two different contexts. In K.R. Rose and G. Kasper (eds) Pragmatics in Language Teaching (pp. 301-325). Cambridge: Cambridge University Press.
16. Brown, J.D. (2004) Observing pragmatics: Testing and data gathering techniques. The Language Teacher 20 (11) (reprinted from Brown, 2000).
17. Brown, J.D. (2005a) Testing in Language Programs: A Comprehensive Guide to English Language Assessment (new edn). New York: McGraw-Hill.
18. Brown, J.D. (2005b) Statistics corner, questions and answers about language testing statistics: Generalizability and decision studies. Shiken: JALT Testing & Evaluation SIG Newsletter 9 (1), 12-16. On WWW at http://jalt.org/test/bro_21.htm.
19. Brown, J.D. and Ahn, R.C. (unpublished) Effects of raters, functions, items, and item characteristics on L2 pragmatics tests. Unpublished manuscript, University of Hawai’i at Manoa.
20. Brown, J.D. and Bailey, K.M. (1984) A categorical instrument for scoring second language writing skills. Language Learning 34, 21-42.
21. Brown, J.D. and Hudson, T. (2002) Criterion-referenced Language Testing. Cambridge: Cambridge University Press.
22. Brown, J.D. and Ross, J.A. (1996) Decision dependability of item types, sections, tests, and the overall TOEFL test battery. In M. Milanovic and N. Saville (eds) Performance Testing, Cognition and Assessment (pp. 231-265). Cambridge:Cambridge University Press.
23. Brown, P. and Levinson, S. (1987) Politeness: Some Universals in Language Usage. Cambridge: Cambridge University Press.
24. Fraser, B. (1990) Perspectives on politeness. Journal of Pragmatics 14, 219-236.
25. Hudson, T. (2001a) Indicators for cross-cultural pragmatic instruction: Some quantitative tools. In K. Rose and G. Kasper (eds) Pragmatics in Language Teaching (pp. 283-300). Cambridge: Cambridge University Press.
26. Hudson, T. (2001b) Self-assessment methods in cross-cultural pragmatics. In T. Hudson and J.D. Brown (eds) A Focus on Language Test Development (pp. 57-74). Honolulu, HI: University of Hawai’i Press.
27. Hudson, T., Detmer, E. and Brown, J.D. (1992) A Framework for Testing Cross-Cultural Pragmatics. Honolulu, HI: University of Hawai’i Press.
28. Hudson, T., Detmer, E. and Brown, J.D. (1995) Developing Prototypic Measures of Cross-Cultural Pragmatics (Technical Report # 7). Honolulu: University of Hawai’i at Manoa, Second Language Teaching and Curriculum Centre.
29. Kozaki, Y. (2004). Using GENOVA and FACETS to set multiple standards on performance assessment for certification in medical translation of Japanese into English. Language Testing 21 (1), 1-27.
30. Kunnan, A.J. (1992) An investigation of a criterion-referenced test using G-theory, and factor and cluster analysis. Language Testing 9 (1), 30-49.
31. Lee, Y.-W. (2006) Dependability of scores for a new ESL speaking assessment consisting of integrated and independent tasks. Language Testing 23 (2), 131-166.
32. Lynch, B.K. and McNamara, T.F. (1998) Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing 15 (2), 158-180.
33. Molloy, H. and Shimura, M. (2005) An examination of situational sensitivity in medium-scale interlanguage pragmatics research. In T. Newfields, Y. Ishida, M. Chapman and M. Fujioka (eds) Proceedings of the May 22-23, 2004 JALT Pan-SIG Conference (pp. 16-32). Tokyo: JALT Pan SIG Committee. On WWW at www.jalt.org/pansig/2004/HTML/ShimMoll.htm.
34. Roever, C. (2001) Aweb-based test of interlanguage pragmalinguistic knowledge:Speech acts, routines, implicatures. Unpublished PhD thesis, University of Hawai’i at Manoa, Honolulu, HI.
35. Roever, C. (2005) Testing ESL Pragmatics. Frankfurt-am-Main, Germany: Peter Lang.
36. Rose, K.R. (2000) An exploratory cross-sectional study of interlanguage pragmatic development. Studies in Second Language Acquisition 22, 27-67.
37. Schoonen, R. (2005) Generalizability of writing scores: An application of structural equation modeling. Language Testing 22 (1), 1-3.
38. Shin, S. (2002) Effects of subskills and text types on Korean EFL reading scores. Second Language Studies 20 (2), 107-113. On WWW at http:// www.hawaii.edu/sls/uhwpesl/on-line_cat.html.
39. Solano-Flores, G. and Li, M. (2006) The use of generalizability (G) theory in testing of linguistic minorities. Educational Measurement: Issues and Practice 25 (1), 13-22.
40. Stansfield, C.W. and Kenyon, D.M. (1992) Research of the comparability of the oral proficiency interview and the simulated oral proficiency interview. System 20, 347-364.
41. Tada, M. (2005) Assessment of EFL pragmatic production and perception using video prompts. Unpublished PhD thesis, Temple University, Philadelphia.
42. Van Weeren, J. and Theunissen, T.J.J.M. (1987) Testing pronunciation: An application of generalizability theory. Language Learning 37 (1), 109-122.
43. Yamamori, K. (2003) Evaluation of students’ interest, willingness, and attitude toward English lessons: Multivariate generalizability theory. The Japanese Journal of Educational Psychology 51 (2), 195-204.
44. Yamanaka, H. (2005) Using generalizability theory in the evaluation of L2 writing. JALT Journal 27 (2), 169-185.
45. Yamashita, S.O. (1996a) Comparing six cross-cultural pragmatics measures. Unpublished PhD thesis, Temple University, Philadelphia.
46. Yamashita, S.O. (1996b) Six measures of JSL Pragmatics (Technical Report # 14). Honolulu: University of Hawai’i at Manoa, Second Language Teaching and Curriculum Centre.
47. Yoshida, H. (2004) An analytic instrument for assessing EFL pronunciation. Unpublished PhD thesis, Temple University, Philadelphia.
48. Yoshida, H. (2006) Using generalizability theory to evaluate reliability of a performance-based pronunciation measurement. Unpublished manuscript, Osaka Jogakuin College.
49. Yoshitake, S.S. (1997) Measuring interlanguage pragmatic competence of Japanese students of English as a foreign language: A multi-test framework evaluation. Unpublished PhD thesis, Columbia Pacific University, Novata, CA.
50. Yoshitake, S. and Enochs, K. (1996) Self assessment and role play for evaluating appropriateness in speech act realizations. ICU Language Research Bulletin 11, 57-76.
51. Zhang, S. (2006) Investigating the relative effects of persons, items, sections, and languages on TOEIC score dependability. Language Testing 23 (3), 351-369.
52. Zhang, Y. (2003) Effects of persons, items, and subtests on UH ELIPT reading test scores. Second Language Studies 21 (2), 107-128. On WWW at http://www.hawaii.edu/sls/uhwpesl/on-line_cat.html.