Rater effects: Ego engagement in rater decision-making

Assessing Writing

Volumn 17, Issue 3, 2012, Pages 150-173

  Wiseman, Cynthia S ^a  

^a CITY UNIVERSITY OF NEW YORK   (United States)

Author keywords

L2 writing assessment; Methods research design; Mixed; Rasch analysis; Rater decision making; Think aloud protocols

Indexed keywords

EID: 84859418826     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.asw.2011.12.001     Document Type: Article

References (51)

1. Anderson J.R. The architecture of cognition 1983, Harvard University Press, Cambridge.
2. Bachman L.F., Lynch B.K., Mason M. Investigating variability in asks and rater judgments in a performance test of foreign language speaking. Language Testing 1995, 12(2):238-257.
3. Bachman L.E., Palmer A.S. Language testing in practice: Designing and developing useful language tests 1996, Oxford University Press, Oxford.
4. Barkaoui K. Variability in ESL essay rating processes: The role of the rating scale and rater experience. Language Assessment Quarterly 2010, 7(1):54-74.
5. Breland, H., Bridgeman, B., & Fowles, M. E. (1999). Writing assessment in admission to higher education: Review and framework (ETS RR-99-3). Princeton, NJ: Educational Testing Service.
6. Brennan R.L., Gao X., Colton D.A. Generalizability analyses of work key listening and writing tests. Educational and Psychological Measurement 1995, 55:157-176.
7. Brown J.D Do English and ESL faculties rate writing samples differently?. TESOL Quarterly 1991, 25:587-603.
8. Caracelli V.J., Greene J.C. Data analysis strategies for mixed-method evaluation designs. Educational Evaluation and Policy Analysis 1993, 15(2):195-207.
9. Connor U., Carrell P. The interpretation of tasks by writers and readers in holistically rated direct assessment of writing. Reading in the composition classroom 1993, 141-160. Heinle and Heinle, Boston, MA. J.G. Carson, I. Leki (Eds.).
10. Creswell J.W., Clark V.L.P. Designing and conducting mixed methods research 2007, Sage Publications, Inc, Thousand Oaks, CA.
11. Cronbach L.I. Essentials of psychological testing 1990, Harper and Row, New York. 5th ed.
12. Cumming A. Expertise in evaluating second language compositions. Language Testing 1990, 7(1):31-51.
13. Cumming A. Assessing L2 writing: Alternative constructs and ethical dilemmas. Assessing Writing 2001, 8:73-83.
14. Cumming, A., Kantor, R., Powers, D., Santos, T., & Taylor, C. (2000). TOEFL 2000 writing framework: A working paper (TOEFL Monograph Series Repot No. 18). Princeton, NJ: Educational Testing Service.
15. DeRemer M.L. Writing assessment: Raters' elaboration of the rating task. Assessing Writing 1998, 5(1):7-29.
16. Engelhard G. Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement 1994, 31(2):93-112.
17. Ericsson K., Simon H. Protocol analysis: Verbal reports as data 1984, MIT Press, Cambridge, MA.
18. Fulcher G. Testing second language speaking 2003, Pearson Professional Education, Essex, England.
19. Guilford J.P. Psychometric methods 1954, McGraw Hill, New York. 2nd ed.
20. Hamp-Lyons L., Matthias S.P. Examining expert judgments of task difficulty on essay tests. Journal of Second Language Writing 1994, 3(1):49-68.
21. Henning G.H. Accounting for nonsystematic error in performance ratings. Language Testing 1996, 13:53-61.
22. Huot B. The influence of holistic scoring procedures on reading and rating student essays. Validating holistic scoring for writing assessment 1993, Hampton Press, Cresskill, NJ, 307-236. M.A. Williamson, B.A. Huot (Eds.).
23. Johnson J.S. The influence of rater language background on writing performance assessment. Language Testing 2009, 26(4):485-505.
24. Kim Y.H. An investigation into native and non-native teachers' judgments of oral English performance: A mixed methods approach. Language Testing 2009, 26(2):187-217.
25. Kobayashi T. Native and nonnative reactions to ESL compositions. TESOL Quarterly 1992, 26:81-112.
26. Kondo-Brown K. A FACETS analysis of rater bias in measuring Japanese second language writing performance. Language Testing 2002, 19(1):3-31.
27. Linacre, J. M. (2005). A user's guide to FACETS: Rasch measurement computer program. Version 3.57. Chicago, IL.
28. Lumley T. Assessing second language writing: The rater's perspective 2005, Peter Lang, Frankfurt am Main.
29. Lynch B.K., McNamara T.F. Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills in immigrants. Language Testing 1998, 15(2):158-180.
30. McNamara T.F. Measuring second language performance 1996, Longman, London and New York.
31. Mendelsohn D., Cumming A. Professors' ratings of language use and rhetorical organizations in ESL compositions. TESL Canada Journal 1987, 5(1):9-26.
32. Milanovic M., Saville N., Shuhong S. A study of the decision-making behavior of composition markers. Studies in language testing, Vol. 3: Performance testing, cognition and assessment 1996, 92-111. Cambridge University Press. M. Milanovic, N. Saville (Eds.).
33. Myford, C., & Wolfe, E. (2000). Monitoring sources of variability within the Test of Spoken English assessment system (Research Project 65). Princeton, NJ: Educational Testing Service.
34. Myford C., Wolfe E. Detecting and measuring Edward effects using many-facet Rasch measurement: Part 1. Introduction to Rasch measurement 2004, 460-517. JAM Press, Maple Grove, MN. E.V. Smith, R.M. Smith (Eds.).
35. Orr M. The FCE speaking test: Using rater reports to help interpret test scores. System 2002, 30:143-154.
36. Saal F.E., Downey R.G., Lahey M.A. Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin 1980, 88(2):413-428.
37. Sakyi A. Validation of holistic scoring for ESL writing assessment: How raters evaluate ESL compositions. Fairness and validation in language assessment 2000, 129-152. Cambridge University Press, Cambridge. A. Kunnan (Ed.).
38. Santos T. Professors' reactions to the academic writing of non-native speaking students. TESOL Quarterly 1988, 20:38-95.
39. Shohamy E., Gordon C.M., Kraemer R. The effect of raters' background and training on the reliability of direct writing tests. The Modern Language Journal 1992, 76(1):27-33.
40. Smith D. Rater judgments in the direct assessment of competency-based second language writing ability. Studies in immigrant English language assessment 2000, 159-189. National Centre for English Language Teaching and Research, Macquarie University, Sydney, Australia. G. Brindley (Ed.).
41. Spool M.D. Training programs for observers of behaviors: A review. Personnel Psychology 1978, 31:853-888.
42. Stansfield C., Ross J. A long-term research agenda for the Test of Written English. Language Testing 1988, 5:160-186.
43. Stock P.L., Robinson J.L. Taking on testing: Teachers as testers researchers. English Education 1987, 19:93-121.
44. Sweedler-Brown C.O. ESL essay evaluation: The influences of sentence-level and rhetorical features. Journal of Second Language Writing 1993, 2:3-17.
45. Van Weeren J., Theunissen T.J.J. Testing pronunciation: An application of generablizability theory. Language Learning 1987, 37:109-122.
46. Vann R.J., Lorenz F.L., Meyer D.M. Error gravity: Faculty response to errors in the written discourse of nonnative speakers of English. Assessing second language writing in academic contexts 1990, 181-195. Ablex, Norwood, NJ. L. Hamp-Lyons (Ed.).
47. Vaughan C. Holistic assessment: What goes on in the raters' minds?. Assessing second language writing in academic contexts 1991, 111-126. Ablex, Norwood, NJ. L. Hamp-Lyons (Ed.).
48. Weigle S. Effects of training on raters of ESL compositions. Language Testing 1994, 11(2):197-223.
49. Weigle S.C. Using FACETS to model rater training effects. Language Testing 1998, 15(2):263-287.
50. Weir C.J. Language testing and validation: An evidence-based approach 2005, Palgrave MacMillan, New York.
51. Wiseman, C. (2005). A validation study comparing an analytic scoring rubric and a holistic scoring rubric in the assessment of L2 writing samples. Unpublished paper, Teachers College, Columbia University, NY.