Rater types in writing performance assessments: A classification approach to rater variability

Language Testing

Volumn 25, Issue 2, 2008, Pages 155-185

  Eckes, Thomas ^a  

^a UNIVERSITY OF HAGEN   (Germany)

Author keywords

Classification; Large scale performance assessment; Rater cognition; Rater monitoring; Rater variability

Indexed keywords

EID: 55249090887     PISSN: 02655322     EISSN: 14770946     Source Type: Journal    
DOI: 10.1177/0265532207086780     Document Type: Article

References (70)

1. Bachman, L.F., Lynch, B.K. & Mason, M. (1995). Investigating variability in tasks and rater judgements in a performance test of foreign language speaking. Language Testing, 12, 238-57.
2. Bachman, L.F. & Palmer, A.S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.
3. Barrett, S. (2001). The impact of training on rater variability. International Education Journal, 2, 49-58.
4. Brown, J.D. (1991). Do English and ESL faculties rate writing samples differently? TESOL Quarterly, 25, 587-603.
5. Castillo, W. & Trejos, J. (2000). Recurrence properties in two-mode hierarchical clustering. In R. Decker & W. Gaul, editors, Classification and information processing at the turn of the millennium (pp. 68-73). Berlin: Springer-Verlag.
6. Congdon, P.J. & McQueen, J. (2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37, 163-178.
7. Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. Cambridge: Cambridge University Press.
8. Cumming, A. (1990). Expertise in evaluating second language compositions. Language Testing, 7, 31-51.
9. Cumming, A., Kantor, R. & Powers, D.E. (2001). Scoring TOEFL essays and TOEFL 2000 prototype writing tasks: An investigation into raters' decision making and development of a preliminary analytic framework (TOEFL Monograph Series, MS-22). Princeton, NJ: Educational Testing Service.
10. Cumming, A., Kantor, R. & Powers, D.E. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal, 86, 67-96.
11. DeCarlo, L.T. (2005). A model of rater behavior in essay grading based on signal detection theory. Journal of Educational Measurement, 42, 53-76.
12. DeRemer, M.L. (1998). Writing assessment: Raters' elaboration of the rating task. Assessing Writing, 5, 7-29.
13. Eckes, T. (1996). Recent developments in multimode clustering. In W. Gaul & D. Pfeifer, editors, From data to knowledge: Theoretical and practical aspects of classification, data analysis, and knowledge organization (pp. 151-158). Berlin: Springer-Verlag.
14. Eckes, T. (2003). Qualitätssicherung beim TestDaF: Konzepte, Methoden, Ergebnisse [Assuring the quality of the TestDaF: Concepts, methods, results]. Fremdsprachen und Hochschule, 69, 43-68.
15. Eckes, T. (2004). Beurteilerübereinstimmung und Beurteilerstrenge: Eine Multifacetten-Rasch-Analyse von Leistungsbeurteilungen im "Test Deutsch als Fremdsprache" (TestDaF) [Rater agreement and rater severity: A many-facet Rasch analysis of performance assessments in the "Test Deutsch als Fremdsprache " (TestDaF)]. Diagnostica, 50, 65-77.
16. Eckes, T. (2005a). Assuring the quality of TestDaF examinations. Paper presented at the 2nd International Conference of the Association of Language Testers in Europe (ALTE), Berlin, Germany.
17. Eckes, T. (2005b). Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly, 2, 197-221.
18. Eckes, T. (2006). Raters' perceptions of scoring criteria in writing and speaking performance assessments. Paper presented at the 28th Language Testing Research Colloquium (LTRC), Melbourne, Australia.
19. Eckes, T., Ellis, M., Kalnberzina, V., Pižorn, K., Springer, C., Szollás, K., & Tsagari, C. (2005). Progress and problems in reforming public language examinations in Europe: Cameos from the Baltic States, Greece, Hungary, Poland, Slovenia, France, and Germany. Language Testing, 22, 355-377.
20. Eckes, T. & Orlik, P. (1991). An agglomerative method for two-mode hierarchical clustering. In H.-H. Bock & P. Ihm, editors, Classification, data analysis, and knowledge organization: Models and methods with applications (pp. 3-8). Berlin: Springer-Verlag.
21. Eckes, T. & Orlik, P. (1993). An error variance approach to two-mode hierarchical clustering. Journal of Classification, 10, 51-74.
22. Edgeworth, F.Y. (1890). The element of chance in competitive examinations. Journal of the Royal Statistical Society, 53, 460-475, 644-663.
23. Elder, C., Knoch, U., Barkhuizen, G. & von Randow, J. (2005). Individual feedback to enhance rater training: Does it work? Language Assessment Quarterly, 2, 175-196.
24. Engelhard, G., Jr. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93-112.
25. Engelhard, G., Jr. (2002). Monitoring raters in performance assessments. In G. Tindal & T. M. Haladyna, editors, Large-scale assessment programs for all students: Validity, technical adequacy, and implementation (pp. 261-287). Mahwah, NJ: Lawrence Erlbaum.
26. Engelhard, G., Jr. & Myford, C.M. (2003). Monitoring faculty consultant performance in the Advanced Placement English Literature and Composition Program with a many-faceted Rasch model (College Board Research Report No. 2003-1). New York: College Entrance Examination Board.
27. Everitt, B.S., Landau, S. & Leese, M. (2001). Cluster analysis (4th ed.). London: Arnold.
28. Freedman, S.W. (1979). How characteristics of student essays influence teachers' evaluation. Journal of Educational Psychology, 71, 328-338.
29. Green, A. (1998). Verbal protocol analysis in language testing research: A handbook. Cambridge: Cambridge University Press.
30. Grotjahn, R. (2004). TestDaF: Theoretical basis and empirical research. In M. Milanovic & C. J. Weir, editors, European language testing in a global context: Proceedings of the ALTE Barcelona Conference July 2001 (pp. 189-203). Cambridge: Cambridge University Press.
31. Guttman, L. (1959). A structural theory for intergroup beliefs and action. American Sociological Review, 24, 318-328.
32. Hamp-Lyons, L. (2003). Writing teachers as assessors of writing. In B. Kroll, editor, Exploring the dynamics of second language writing (pp. 162-189). Cambridge: Cambridge University Press.
33. Hinkel, E. (1994). Native and nonnative speakers' pragmatic interpretations of English texts. TESOL Quarterly, 28, 353-376.
34. Hoyt, W.T. & Kerns, M.-D. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403-424.
35. Huot, B.A. (1993). The influence of holistic scoring procedures on reading and rating student essays. In M. M. Williamson & B. A. Huot, editors, Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 206-236). Cresskill, NJ: Hampton Press.
36. Lievens, F. (2001). Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity. Journal of Applied Psychology, 86, 255-264.
37. Linacre, J.M. (2002). What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions, 16(2), 878.
38. Linacre, J.M. (2004). Optimizing rating scale category effectiveness. In E. V. Smith & R. M. Smith, editors, Introduction to Rasch measurement (pp. 258-278). Maple Grove, MN: JAM Press.
39. Linacre, J.M. (2005). A user's guide to FACETS: Rasch-model computer programs [Software manual]. Chicago, IL: Winsteps.com.
40. Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19, 246-276.
41. Lumley, T. (2005). Assessing second language writing: The rater's perspective. Frankfurt: Lang.
42. Lumley, T. & Brown, A. (2005). Research methods in language testing. In E. Hinkel, editor, Handbook of research in second language teaching and learning (pp. 833-855). Mahwah, NJ: Lawrence Erlbaum.
43. Lumley, T. & McNamara, T.F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12, 54-71.
44. McNamara, T.F. (1990). Item response theory and the validation of an ESP test for health professionals. Language Testing, 7, 52-75.
45. McNamara, T.F. (1996). Measuring second language performance. London: Longman.
46. Milanovic, M., Saville, N. & Shuhong, S. (1996). A study of the decision-making behaviour of composition markers. In M. Milanovic & N. Saville, editors, Performance testing, cognition and assessment: Selected papers from the 15th Language Testing Research Colloquium (Studies in language testing, Vol. 3, pp. 92-114). Cambridge: Cambridge University Press.
47. Mirkin, B., Arabie, P. & Hubert, L.J. (1995). Additive two-mode clustering: The error-variance approach revisited. Journal of Classification, 12, 243-263.
48. Myford, C.M., Marr, D.B. & Linacre, J.M. (1996). Reader calibration and its potential role in equating for the Test of Written English (TOEFL Research Report No. 95-40). Princeton, NJ: Educational Testing Service.
49. Myford, C.M. & Wolfe, E.W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4, 386-422.
50. Myford, C.M. & Wolfe, E.W. (2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of Applied Measurement, 5, 189-227.
51. Pollitt, A. & Murray, N.L. (1996). What raters really pay attention to. In M. Milanovic & N. Saville, editors, Performance testing, cognition and assessment: Selected papers from the 15th Language Testing Research Colloquium (Studies in language testing, Vol. 3, pp. 74-91). Cambridge: Cambridge University Press.
52. Pula, J.J. & Huot, B.A. (1993). A model of background influences on holistic raters. In M. M. Williamson and B. A. Huot, editors, Validating holistic scoring for writing assessment: Theoretical and empirical foundations. (pp. 237-265). Cresskill, NJ: Hampton Press.
53. Pulakos, E.D. (1986). The development of training programs to increase accuracy with different rating tasks. Organizational Behavior and Human Decision Processes 38, 76-91.
54. Reed, D.J. and Cohen, A.D. (2001). Revisiting raters and ratings in oral language assessment. In C. Elder et al., editors, Experimenting with uncertainty: Essays in honour of Alan Davie (pp. 82-96). Cambridge: Cambridge University Press.
55. Sakyi, A.A. (2000). Validation of holistic scoring for ESL writing assessment: How raters evaluate compositions. In J. J. Kunnan, editor, Fairness and validation in language assessment (pp. 129-152). Cambridge: Cambridge University Press.
56. Schoonen, R. (2005). Generalizability of writing scores: An application of structural equation modeling. Language Testing, 22, 1-30.
57. Schoonen, R., Vergeer, M. & Eiting, M. (1997). The assessment of writing ability: Expert readers versus lay readers. Language Testing, 14, 157-184.
58. Shohamy, E. (1995). Performance assessment in language testing. Annual Review of Applied Linguistics, 15, 188-211.
59. Smith, D. (2000). Rater judgments in the direct assessment of competency-based second language writing ability. In G. Brindley, editor, Studies in immigrant English language assessment, Vol. 1, (pp. 159-189). Sydney: National Centre for English Language Teaching and Research, Macquarie University.
60. Smith, R.M. (2004). Fit analysis in latent trait measurement models. In E. V. Smith & R. M. Smith, editors, Introduction to Rasch measurement (pp. 73-92). Maple Grove, MN: JAM Press.
61. Tucker, L.R. (1964). The extension of factor analysis to three-dimensional matrices. In N. Frederiksen & H. Gulliksen, editors, Contributions to mathematical psychology (pp. 109-127). New York: Holt, Rinehart and Winston.
62. Van Mechelen, I., Bock, H.-H. & De Boeck, P. (2004). Two-mode clustering methods: A structured overview. Statistical Methods in Medical Research, 13, 363-394.
63. Vaughan, C. (1991). Holistic assessment: What goes on in the rater's mind? In L. Hamp-Lyons, editor, Assessing second language writing in academic contexts (pp. 111-125). Norwood, NJ: Ablex.
64. Weigle, S.C. (1994). Effects of training on raters of ESL compositions. Language Testing, 11, 197-223.
65. Weigle, S.C. (1998). Using FACETS to model rater training effects. Language Testing, 15, 263-287.
66. Weigle, S.C. (1999). Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing, 6, 145-178.
67. Weigle, S.C. (2002). Assessing writing. Cambridge: Cambridge University Press.
68. Weir, C.J. (2005). Language testing and validation: An evidence-based approach. Basingstoke: Palgrave Macmillan.
69. Wolfe, E.W. (1997). The relationship between essay reading style and scoring proficiency in a psychometric scoring system. Assessing Writing, 4, 83-106.
70. Wolfe, E.W., Kao, C.-W. & Ranney, M. (1998). Cognitive differences in proficient and nonproficient essay scorers. Written Communication, 15, 465-492.