Validating a standards-based classroom assessment of English proficiency: A multitrait-multimethod approach

Language Testing

Volumn 24, Issue 4, 2007, Pages 489-515

  Llosa, Lorena ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 42649111743     PISSN: 02655322     EISSN: 14770946     Source Type: Journal    
DOI: 10.1177/0265532207080770     Document Type: Article

References (59)

1. Arkoudis, S. and O'Loughlin, K. 2004: Tensions between validity and outcomes: Teacher assessment of written work of recently arrived immigrant ESL students. Language Testing 21: 284-304.
2. Bachman, L.F., Davidson, F., Ryan, K. and Choi, I-C. 1995: An investigation into the comparability of two tests of English as a foreign language: The Cambridge-TOEFL comparability study. Studies in Language Testing, 1. Cambridge: Cambridge University Press.
3. Bachman, L.F. and Palmer, A. 1981: The construct validation of the FSI oral interview. Language Learning 31: 67-86.
4. Bachman, L.F. and Palmer, A. 1982: The construct validation of some components of communicative proficiency. TESOL Quarterly 16: 449-65.
5. Bae, J. and Bachman, L.F. 1998: A latent variable approach to listening and reading: Testing factorial invariance across two groups of children in the Korean/English two-way immersion program. Language Testing 15: 380-414.
6. Bentler, P.M. 1985-2005: EQS for Windows 6.1. Encino, CA: Multivariate Software.
7. Bentler, P.M. 1990: Comparative fit indexes in structural models. Psychological Bulletin 107: 238-46.
8. Bentler, P.M. and Bonnet, D.G. 1980: Significance tests and goodness-of-fit in the analysis of covariance structures. Psychological Bulletin 88: 588-606.
9. Bentler, P.M. and Dijkstra, T. 1985: Efficient estimation via linearization in structural models. In Krishnaiah, P.R., editor, Multivariate analysis VI. Amsterdam: North-Holland, 9-42.
10. Breen, M.P., Barrat-Pugh, C., Derewianka, B., House, H., Hudson, C., Lumley, T. and Rohl, M. 1997: Profiling ESL Children: How teachers interpret and use national and state assessment frameworks. Canberra: Department of Employment, Education, Training, and Youth Affairs, Commonwealth of Australia.
11. Brindley, G. 1998: Outcomes-based assessment and reporting in language learning programmes: A review of the issues. Language Testing 15: 45-85.
12. Brindley, G. 2001: Outcomes-based assessment in practice: Some examples and emerging insights. Language Testing 18: 393-407.
13. Brown, A. 1995: The effect of rater variables on the development of an occupation-specific language performance test. Language Testing 12: 1-15.
14. Byrne, B.M. 2006: Structural equation modeling with EQS: Basic concepts, applications, and programming. Mahwah, NJ: Lawrence Erlbaum.
15. Campbell, D.T. and Fiske, D.W. 1959: Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin 56: 81-105.
16. Chalhoub-Deville, M. 1995: Deriving oral assessment scales across different tests and rater groups. Language Testing 12: 16-33.
17. Conway, J.M., Scullen, S.E., Lievens, F. and Lance, C.E. 2004: Bias in the correlated uniqueness model for MTMM data. Structural Equation Modeling: A Multidisciplinary Journal 11: 535-59.
18. Cumming, A., Grant, L., Mulcahy-Ernt, P. and Powers, D.E. 2004: A teacher-verification study of speaking and writing prototype tasks for a new TOEFL. Language Testing 21: 107-45.
19. Davison, C. 2004: The contradictory culture of teacher-based assessment: ESL teacher assessment practices in Australian and Hong Kong secondary schools. Language Testing 21: 305-34.
20. Elder, C. 1993: How do subject specialists construe classroom language proficiency? Language Testing 10: 233-54.
21. Epp, L. and Stawychny, M. 2001: Using the Canadian Language Benchmarks (CLB) to benchmark college programs/courses and language proficiency tests. TESL Canada Journal 18: 32-47.
22. Gattullo, F. 2000: Formative assessment in ELT primary (elementary) classrooms: An Italian case study. Language Testing 17: 278-88.
23. Gipps, C. 1994: Beyond testing: Towards a theory of educational assessment. London: Falmer Press.
24. Grant, L. 1997: Testing the language proficiency of bilingual teachers: Arizona's Spanish proficiency test. Language Testing 14: 23-46.
25. Hoge, R. and Coldarci, T. 1989: Teacher-based judgments of academic achievement: A review of literature. Review of Educational Research 59: 297-313.
26. Hu, L.T. and Bentler, P.M. 1999. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 6: 1-55.
27. Kenny, D.A. 1976: Am empirical application of confirmatory factor analysis to the multitrait-multimethod matrix. Journal of Experimental Social Psychology 12: 247-52.
28. Kenny, D.A. 1979: Correlation and causality. New York: Wiley.
29. Kenny, D.A. and Kashy, D.A. 1992: Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin 112: 165-72.
30. Kunnan, A.J. 1995: Test taker characteristics and test performance: A structural modeling approach. Cambridge: Cambridge University Press.
31. Lance, C.E., Noble, C.L. and Scullen, S.E. 2002: A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods 7: 228-44.
32. Leung, C. and Mohan, B. 2004: Teacher formative assessment and talk in classroom contexts: Assessments as discourse and assessment of discourse. Language Testing 21: 335-59.
33. Linquanti, R. 2001: The redesignation dilemma: Challenges and choices in fostering meaningful accountability for English learners. Policy Report 2001-1. Santa Barbara, CA: University of California Linguistic Minority Research Institute, University of California, Santa Barbara.
34. Llosa, L. 2005a: Assessing English learners' language proficiency: A qualitative investigation of teachers' interpretations of the California ELD Standards. The CATESOL Journal 17: 7-18.
35. Llosa, L. 2005b: Building and supporting a validity argument for a standards-based classroom assessment of English proficiency. Unpublished PhD dissertation, University of California, Los Angeles.
36. Lumley, T.J.N. and McNamara, T.F. 1995: Rater characteristics and rater bias: Implications for training. Language Testing 12: 54-71.
37. Marsh, H.W. 1989: Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement 15: 47-70.
38. Marsh, H.W., Byrne, B.M. and Craven, R. 1992: Overcoming problems in confirmatory factor analyses of MTMM data: The correlated uniqueness model and factorial invariance. Multivariate Behavioral Research 27: 489-507.
39. Marsh, H.W. and Grayson, D. 1995: Latent variable models of multitrait-multimethod data. In Hoyle, R.H., editor, Structural equation modeling: Concepts, issues, and applications. Thousand Oaks, CA: Sage, 177-98.
40. McNamara, T. 2001: Language assessment as social practice: Challenges for research. Language Testing 18: 333-49.
41. Meisels, S.J., Bickel, D.D., Nicholson, J., Xue, Y. and Atkins-Burnett, S. 2001: Trusting teacher judgments: A validity study of a curriculum-embedded performance assessment in kindergarten to grade 3. American Educational Research Journal 38: 73-95.
42. Muthén, B.O. and Kaplan, D. 1985: A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology 38: 171-89.
43. North, B. 1993: The development of descriptors on scales of language proficiency. College Park, MD: The National Foreign Language Center.
44. North, B. 1995: The development of a common framework scale of descriptors of language proficiency based on a theory of measurement. System 23: 445-65.
45. North, B. 2000: The development of a common framework scale of language proficiency. New York: Peter Lang.
46. O'Sullivan, B., Weir, C. and Saville, N. 2002: Using observation checklists to validate speaking-test tasks. Language Testing 19: 33-56.
47. Popham, W.J. 2003: The trouble with testing: Why standards-based assessment doesn't measure up. American School Board Journal 190: 14-17.
48. Rea-Dickins, P. 2001: Mirror, mirror on the wall: Identifying processes of classroom assessment. Language Testing 18: 429-62.
49. Rea-Dickins, P. and Gardner, S. 2000: Snares and silver bullets: Disentangling the construct of formative assessment. Language Testing 17: 215-43.
50. Rindskopf, D. and Rose, T. 1988: Some theory and applications of confirmatory second-order factor analysis. Multivariate Behavioral Research 23: 51-67.
51. Satorra, A. and Bentler, P.M. 1988: Scaling corrections for chi-square statistics in covariance structure analysis. Proceedings of the American Statistical Association, 308-13.
52. Sawaki, Y. 2007: Construct validation of analytic rating scales in a speaking assessment: Reporting a score profile and a composite. Language Testing 24(3), 355-90.
53. Shepard, L.A. 2000: The role of classroom assessment in teaching and learning. CSE Technical Report 517. National Center for Research on Evaluation, Standards, and Student Testing (CRESST). University of California, Los Angeles. Retrieved on 10 April 2005 from: www.cse.ucla.edu/CRESST/Reports/TECH517.pdf
54. Shin, S.K. 2005: Did they take the same test? Examinee language proficiency and the structure of language tests. Language Testing 22: 31-57.
55. SPSS for Windows, Rel. 14.0.0. 2005. Chicago: SPSS.
56. Stansfield, C. and Kenyon, D. 1996: Comparing the scaling of speaking tasks by language teachers and by the ACTFL guidelines. In Cumming, A. and Berwick, R., editors, Validation in language testing. Clevedon, UK: Multilingual Matters, 124-53.
57. Stevens, J.J. and Clauser, P. 1995: Multitrait-multimethod comparisons of a writing portfolio and the ITBS. Paper presented at the annual meeting of the National Council for Measurement in Education, San Francisco, CA.
58. Taylor, C.S. 2002: Incorporating classroom-based assessments into large-scale assessment programs. In Tindal, G. and Haladyna, T.M., editors, Large-scale assessment: Programs for all students. Mahwah, NJ: Lawrence Erlbaum, 233-59.
59. Widaman, K.F. 1985: Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement 9: 1-26.