Generalizability of writing scores: An application of structural equation modeling

Language Testing

Volumn 22, Issue 1, 2005, Pages 1-30

  Schoonen, Rob ^a  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 13744250460     PISSN: 02655322     EISSN: 14770946     Source Type: Journal    
DOI: 10.1191/0265532205lt295oa     Document Type: Article

References (63)

1. Ackerman, T.A. and Smith, P.L. 1988: A comparison of the information provided by essay, multiple choice, and free-response writing tests. Applied Psychological Measurement 12, 117-28.
2. Bachman, L.F. 1990: Fundamental considerations in language testing. Oxford: Oxford University Press.
3. Bachman, L.F. 2002: Some reflections on task-based language performance assessment. Language Testing 19, 453-76.
4. Bachman, L.F., Lynch, B.K. and Mason, M. 1995: Investigating variability in tasks and rater judgements in a performance test of foreign language speaking. Language Testing 12, 238-57.
5. Bachman, L.F. and Palmer, A.S. 1996: Language testing in practice. Oxford: Oxford University Press.
6. Barrett, Th.J. 1994: Generalizability of writing tasks at fourth grade in the Riverside Unified School District. Paper presented at the Annual Meeting of California Educational Research Association, San Diego 1994. ERIC Document Reproduction Service No. ED 377 498.
7. Benton, S.L., Sharp, J.M., Corkill, A.J., Downey, R.G. and Khramtsova, I. 1995: Knowledge, interest, and narrative writing. Journal of Educational Psychology 87, 66-79.
8. Blok, H. 1986: Essay rating by the comparison method. Tijdschrift voor onderwijsresearch [Journal for educational research] 11, 169-76.
9. Bolus, R.E., Hinofotis, F.B. and Bailey, K.M. 1982: An introduction to generalizability theory in second language research. Language Learning 32, 245-58.
10. Boodoo, G.M. and Garlinghouse, P. 1983: Use of essay examination to investigate the writing skills of undergraduate education majors. Educational and Psychological Measurement 43, 1005 -10.
11. Brennan, R.L. 1992: Elements of generalizability theory. Iowa City, IA: The American College Testing Program. First edition: 1983.
12. Brennan, R.L., Gao, X. and Colton, D.A. 1995: Generalizability analyses of Work Keys listening and writing tests. Educational and Psychological Measurement 55, 157-76.
13. Brossell, G. 1983: Rhetorical specification in essay examination topics. College English 45, 165-73.
14. Brown, J.D. and Bailey, K.M. 1984: A categorical instrument for scoring second language writing skills. Language Learning 34, 21-42.
15. Brown, J.D., Hilgers, Th. and Marsella, J. 1991: Essay prompts and topics. Minimizing the effect of mean differences. Written Communication 8, 533-56.
16. Charney, D. 1984: The validity of using holistic scoring to evaluating writing: a critical overview. Research in the Teaching of English 18, 65-81.
17. Cooper, P.L. 1984: The assessment of writing ability: a review of research. Princeton, NJ: Educational Testing Service. GRE Board Research Report GREB No. 82-15R/ETS Research Report 84-12.
18. Cronbach, L.J., Gleser, G.C., Nanda, H. and Rajaratnam, N. 1972: The dependability of behavioral measurements: theory of generalizability scores and profiles. New York: Wiley.
19. Cumming, A. 1990: Expertise in evaluating second language compositions. Language Testing 7, 31-51.
20. Cumming, A. 1997: The testing of writing in a second language. In Clapham, C. and Corson, D., editors, Encyclopedia of language and education: language testing and assessment. Volume 7. Dordrecht: Kluwer Academic Publishers, 51-63.
21. Cumming, A., Kantor, R. and Powers, D.E. 2002: Decision making while rating ESL/EFL writing tasks: a descriptive framework. Modern Language Journal 86, 67-96.
22. De Jong, T. and Ferguson-Hessler, M.G.M. 1996: Types and qualities of knowledge. Educational Psychologist 31, 105-13.
23. Godshalk, F.I., Swineford, F. and Coffman, W.E. 1966: The measurement of writing ability. New York: College Entrance Examination Board.
24. Hamp-Lyons, L. 1990: Second language writing: assessment issues. In Kroll, B., editor, Second language writing: research insights for the classroom. Cambridge: Cambridge University Press, 69-87.
25. Hamp-Lyons, L, editor, 1991: Assessing second language writing in academic contexts. Norwood, NJ: Ablex.
26. Hogan, T.P. and Mishler, C. 1982: Relationships among measures of writing skill. ERIC Document Reproduction Service No. ED 224 812.
27. Hudson, S.A. and Veal, L.R. 1981: An empirical investigation of direct and indirect measures of writing. ERIC Document Reproduction Service No. ED 205 993.
28. Huot, B. 1990: The literature of direct writing assessment: major concerns and prevailing trends. Review of Educational Research 60, 237-63.
29. Jöreskog, K. and Sörbom, D. 1996: LISREL 8: user's reference guide. Chicago, IL: Scientific Software International, Inc.
30. Kellog, R.T. 1987: Effects of topic knowledge on the allocation of processing time and cognitive effort to writing processes. Memory and Cognition 15, 256-66.
31. Kroll, B. 1998: Assessing writing abilities. Annual Review of Applied Linguistics 18, 219-40.
32. Kunnan, A.J. 1998: An introduction to structural equation modelling for language assessment research. Language Testing 15, 295-332.
33. Langer, J.A. 1984: The effects of available information on responses to school writing tasks. Research in the Teaching of English 18, 27-44.
34. Lee, Y.-W., Kantor, R. and Mollaun, P. 2002: Score reliability as an essential prerequisite for validating new writing and speaking tasks for TOEFL. Paper presented at the annual meeting of Teachers of English to the Speakers of Other Languages (TESOL). Salt Lake City, UT, April 2002.
35. Lehmann, R.H. 1990: Reliability and generalizability of ratings of compositions. Studies in Educational Evaluation 16, 501-12.
36. Lumley, T. 2002: Assessment criteria in a large-scale writing test: what do they really mean to the raters? Language Testing 19, 246-76.
37. Lynch, B.K. and McNamara, T.F. 1998: Using G-theory and Many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing 15, 158-80.
38. Marcoulides, G.A. 1990: An alternative method for estimating variance components in generalizability theory. Psychological Reports 66, 379-86.
39. Marcoulides, G.A. 1996: Estimating variance components in generalizability theory: the covariance structure analysis approach. Structural Equation Modeling 3, 290-99.
40. Marcoulides, G.A. 1998: Applied generalizability theory models. In Marcoulides, G.A., Editor, Modern methods of business research. Mahwah, NJ: Lawrence Erlbaum Associates.
41. Marcoulides, G.A. 2000: Advances and new developments in G-theory: applications of structural equation modeling. Workshop presented at the 22nd annual Language Testing Research Colloquium. Vancouver, BC, Canada.
42. McCutchen, D. 1986: Domain knowledge and linguistic knowledge in the development of writing ability. Journal of Memory and Language 25, 431-44.
43. Moon, T.R., Loyd, B.H. and Hughes, K.R. 1996: Generalizability analyses of a large-scale writing assessment. Paper presented at the Annual Meeting of AERA, New York, 1996.
44. Moss, P.A., Cole, N.S. and Khampalikit, C. 1982: A comparison of procedures to assess written language skills at grade 4, 7, and 10. Journal of Educational Measurement 19, 37-47.
45. Norris, J.M., Brown, J.D., Hudson, T.D. and Bonk, W. 2002: Examinee abilities and task difficulty in task-based second language performance assessment. Language Testing 19, 395-418.
46. Novak, J.R., Herman, J.L. and Gearhart, M. 1996: Establishing validity for performance-based assessments: an illustration for collections of student writing. The Journal of Educational Research 89, 220-33.
47. Parkes, J. 2000: The Relationship between the reliability and cost of performance and cost performance assessments. Education Policy Analysis Archives 8.
48. Purves, A.C. 1992: Reflections on research and assessment in written composition. Research in the Teaching of English 26, 108-22.
49. Quellmalz, E.S., Capell, F.J. and Chou, Ch.-P. 1982: Effects of discourse and response mode on the measurement of writing competence. Journal of Educational Measurement 19, 241-58.
50. Raykov, T. and Marcoulides, G.A. 2000: A first course in structural equation modeling. Mahwah, NJ: Lawrence Erlbaum Associates.
51. Read, J. 1990: The validity of writing test tasks. Plenary paper presented at the RELC Regional Seminar on Language Testing and Language Programme Evaluation. ERIC Document Reproduction Service No. ED 328 072.
52. Schoonen, R. 1991: De evaluatie van schrijfvaardigheidsmetingen. Een empirische studie naar betrouwbaarheid, validiteit en bruikbaarheid van schrijfvaardigheidsmetingen in de achtste groep van het basisonderwijs. [The evaluation of measures of writing ability. An empirical study into the reliability, validity and usefulness of measures of writing ability at grade 6.] Amsterdam: University of Amsterdam. Doctoral dissertation with English summary.
53. Schoonen, R. 1991b: Facetten van schrijfvaardigheidsmetingen: taken, beoordelingsaspecten en beoordelingsmethoden. [Facets of writing assessments: tasks, aspects to be rated and scoring procedures.] In Bax, M.M.H. and Vuijk, W., editors, Thema's in de taalbeheersing. [Themes in language performance.] Dordrecht: ICG Publications, 194-203.
54. Schoonen, R. 1993: The validity of writing assignments. Paper presented at the Annual Meeting of AERA, Atlanta, GA, 12-16 April 1993.
55. Schoonen, R., Van Gelderen, A., De Glopper, K., Hulstijn, J., Snellings, P., Simis, A. and Stevenson, M. 2002: Linguistic knowledge, metacognitive knowledge and retrieval speed in L1, L2 and EFL writing. a structural equation approach. In Ransdell, S. and Barbier, M.-L., editors, New directions for research in L2 writing. Amsterdam: Kluwer Academic Publishers, 101-122.
56. Schoonen, R., Vergeer, M. and Eiting, M. 1997: The assessment of writing ability: expert readers versus lay readers. Language Testing 14, 157-84.
57. Shavelson, R.J., Baxter, G.P. and Gao, X. 1993: Sampling variability of performance assessments. Journal of Educational Measurement 30, 215-32.
58. Shavelson, R.J. and Webb, N.M. 1991: Generalizability theory: a primer. Newbury: Sage.
59. Stevens, J.J. and Clauser, P. 1996: Longitudinal examination of writing portfolio and the ITBS. Paper presented at the Annual Meeting of AERA, New York, 1996. ERIC Document Reproduction Service No. ED 397 116.
60. Swartz, R., Patience, W. and Whitney, D.R. 1985: Adding an essay to the GED writing skills test: reliability and validity issues. GED Testing Service Research Studies, No.7. ERIC Document Reproduction Service No. ED 266 288.
61. Swartz, C.-W., Hooper, S.-R., Mongomery, J.-W., Wakely, M.-B., De-Kruif, R.-E.L., Reed, M., Brown, T.-T., Levine, M.-D. and White, K.-P. 1999: Using generalizability theory to estimate the reliability of writing scores derived from holistic and analytical scoring methods. Educational and Psychological Measurement 59, 492-506.
62. Van den Bergh, H., De Glopper, K. and Schoonen, R. 1988: Directe metingen van schrijfvaardigheid: validiteit en taakeffecten. [Direct measures of writing ability: validity and task effects.] In van Eemeren, F.H. and Grootendorst, R., editors, Taalbeheersing. [Language Performance.] Dordrecht: Foris Publications, 370-78.
63. Weigle, S.C. 1994: Effects of training on raters of ESL compositions. Language Testing 11, 197-223.