Severity differences among self-assessors, peer-assessors, and teacher assessors rating EFL essays

Assessing Writing

Volumn 18, Issue 2, 2013, Pages 111-131

  Esfandiari, Rajab ^a   Myford, Carol M ^b  

^a IMAM KHOMEINI INTERNATIONAL UNIVERSITY   (Iran)

^b UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

Author keywords

Peer assessment; Rater effects; Rating scales; Self assessment; Writing assessment

Indexed keywords

EID: 84873463679     PISSN: 10752935     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.asw.2012.12.002     Document Type: Article

References (82)

1. Andrade H., Du Y., Mycek K. Rubric-referenced self-assessment and middle school students' writing. Assessment in Education 2010, 17(2):199-214.
2. Basturk R. Applying the many-facet Rasch model to evaluate PowerPoint presentation performance in higher education. Assessment & Evaluation in Higher Education 2008, 33(4):431-444.
3. Blanche P., Merino M. Self-assessment of foreign-language skills: Implications for teachers and researchers. Language Learning 1989, 39(3):313-340.
4. Blue G. Self-assessment of foreign language skills: Does it work?. CLE Working Papers 1994, 3:18-35. (ERIC Document Reproduction Service, No. ED396569).
5. Brown A. Self-assessment of writing in independent language learning programs: The value of annotated samples. Assessing Writing 2005, 10(3):174-191.
6. Brown H. Language assessment: Principles and classroom practices 2004, Longman, New York, NY.
7. Butler Y.G., Lee J. The effects of self-assessment among young learners of English. Language Testing 2010, 27(1):5-31.
8. Chen Y. Learning to self-assess oral performance in English: A longitudinal case study. Language Teaching Research 2008, 12(2):235-262.
9. Cheng W., Warren M. Peer and teacher assessment of the oral and written tasks of a group project. Assessment & Evaluation in Higher Education 1999, 24(3):301-314.
10. Cheng W., Warren M. Peer assessment of language proficiency. Language Testing 2005, 22(1):93-121.
11. Eckes T. Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments 2011, Peter Lang, Frankfurt, Germany.
12. Elder C., Knoch U., Barkhuizen G., von Randow J. Individual feedback to enhance rater training: Does it work?. Language Assessment Quarterly 2005, 2(3):175-196.
13. Engelhard G. Constructing rater and task banks for performance assessments. Journal of Outcome Measurement 1997, 1:19-33.
14. Engelhard G., Myford C.M. Comparison of single- and double-assessor scoring designs for the assessment of accomplished teaching. Journal of Applied Measurement 2009, 10(1):52-69.
15. Evans A.W., McKenna C., Oliver M. Trainees' perspectives on the assessment and self-assessment of surgical skills. Assessment & Evaluation in Higher Education 2005, 30(2):163-174.
16. Falchikov N. Peer feedback marking: Developing peer-assessment. Innovations in Education & Training International 1995, 32(2):175-187.
17. Falchikov N., Boud D. Student self-assessment in higher education: A meta-analysis. Review of Educational Research 1989, 59(4):330-395.
18. Falchikov N., Goldfinch J. Student peer assessment: A meta-analysis comparing peer and teacher remarks. Review of Educational Research 2000, 70(3):287-322.
19. Farhady H., Hedayati H. Language assessment policy in Iran. Annual Review of Applied Linguistics 2009, 29:132-141.
20. Freeman M. Peer assessment by groups of group work. Assessment & Evaluation in Higher Education 1995, 20(3):289-300.
21. Gielen S., Peeters E., Dochy F., Onghena P., Struyven K. Improving the effectiveness of peer feedback for learning. Learning and Instruction 2010, 20(4):304-315.
22. Goldfinch J.M., Raeside R. Development of a peer assessment technique for obtaining individual marks on a group project. Assessment & Evaluation in Higher Education 1990, 15(3):210-225.
23. Hamayan E.V. Approaches to alternative assessment. Annual Review of Applied Linguistics 1995, 15:212-226.
24. Hargreaves L., Earl L., Schmidt M. Perspectives on alternative assessment reform. American Educational Research Journal 2002, 39(1):69-96.
25. Herman J.L., Aschbacher P.R., Winters L. A practical guide to alternative assessment 1992, Association for Supervision and Curriculum Development, Alexandria, VA.
26. Jacobs G.M., Curtis A., Braine G., Huang S.-Y. Feedback on student writing: Taking the middle path. Journal of Second Language Writing 1998, 7(3):307-317.
27. Jacobs H.L., Zinkgraf S.A., Wormuth D.R., Hartfiel V.F., Hughey J.B. Testing ESL composition: A practical approach 1981, Newbury House, Rowley, MA.
28. Jiliang C., Kun W. An investigation of scorer reliability of self and peer assessment of EFL writing among Chinese college students. CELEA Journal 2007, 30(1):3-11.
29. Kirby N.F., Downs C. Self-assessment and the disadvantaged student: Potential for encouraging self-regulated learning?. Assessment & Evaluation in Higher Education 2008, 32(4):194-475.
30. Knoch U. Investigating the effectiveness of individualized feedback to rating behavior-A longitudinal study. Language Testing 2011, 28(2):179-200.
31. Knoch U., Read J., von Randow J. Re-training writing raters online: How does it compare with face-to-face training?. Assessing Writing 2007, 12(1):26-43.
32. Kwan K., Leung R. Tutor versus peer group assessment of student performance in a simulation training exercise. Assessment & Evaluation in Higher Education 1996, 21(3):205-215.
33. Leach, L. (2000). Self-directed learning: Theory and practice (Unpublished doctoral thesis). University of Technology, Sydney, Australia.
34. Leach L. Optional self-assessment: Some tensions and dilemmas. Assessment & Evaluation in Higher Education 2012, 37(2):137-147.
35. Linacre J.M. Many-facet Rasch measurement 1994, MESA Press, Chicago, IL.
36. Linacre J.M. Optimizing rating scale effectiveness. Introduction to Rasch measurement 2004, 257-578. JAM Press, Maple Grove, MN. E.V. Smith, R.M. Smith (Eds.).
37. Linacre J.M. FACETS (Version 3.68.1) [Computer software] 2011, MESA Press, Chicago, IL.
38. Linacre J.M., Wright B.D. Construction of measures from many-facet data. Journal of Applied Measurement 2002, 3(4):484-509.
39. Lindblom-Ylänne S., Pihlajamaki H., Kotkas T. Self-, peer- and teacher-assessment of student essays. Active Learning in Higher Education 2006, 7(1):51-62.
40. Liu N.F., Carless D. Peer feedback: The learning element of peer assessment. Teaching in Higher Education 2006, 11(3):279-290.
41. López-Pastor V.M., Fernández-Balboa J., Pastor M.L.S., Aranda A.F. Students' self-grading, professor's grading and negotiated final grading at three university programmes: Analysis of reliability and grade difference ranges and tendencies. Assessment & Evaluation in Higher Education 2012, 37(4):453-464. 10.1080/02602938.2010.545868.
42. Lumley T., McNamara T.F. Rater characteristics and rater bias: Implications for training. Language Testing 1995, 12(1):16-33.
43. Lunt, H., Morton, J., & Wigglesworth, G. (1994 July). Rater behaviour in performance testing: Evaluating the effect of bias feedback. Paper presented at 19th annual congress of applied linguistics association of Australia, University of Melbourne.
44. Maclellan E. How convincing is alternative assessment?. Assessment & Evaluation in Higher Education 2004, 29(3):311-321.
45. Matsuno S. Self-, peer-, and teacher-assessments in Japanese university EFL writing classrooms. Language Testing 2009, 26(1):75-100.
46. McGarrell H.M. Native and non-native English speaking student teachers engage in peer feedback. Canadian Journal of Applied Linguistics 2010, 13(1):71-90.
47. McIntyre, P. N. (1993). The importance and effectiveness of moderation training on the reliability of teacher assessments of ESL writing samples (Unpublished master's thesis). Melbourne, Australia: The University of Melbourne.
48. McNamara T.F. Measuring second language performance 1996, Longman, New York, NY.
49. Mok J. A case study of students' perceptions of peer assessment in Hong Kong. ELT Journal 2011, 65(3):240-250.
50. Myford C.M., Wolfe E.W. Detecting and measuring rater effects using manyfacet Rasch measurement: Part II. Introduction to Rasch measurement 2004, 460-517. JAM Press, Maple Grove, MN. E.V. Smith, R.M. Smith (Eds.).
51. Nakamura Y. Teacher assessment and peer assessment in practice (Educational Studies 44) 2002, International Christian University, Tokyo, Japan, (ERIC Document Reproduction Service No. ED464483).
52. O'Sullivan B., Rignall M. Assessing the value of bias analysis feedback to raters for the IELTS writing module. IELTS collected papers 2007, Cambridge University Press, Cambridge, UK. L. Taylor, P. Falvey (Eds.).
53. Orsmond P., Merry S., Reiling K. The importance of marking criteria in the use of peer assessment. Assessment & Evaluation in Higher Education 1996, 21(3):239-250.
54. Oscarson M. Self-assessment of language proficiency: Rationale and applications. Language Testing 1989, 6(1):1-13.
55. Percival F., Ellington H. A handbook of educational technology 1984, Kogan Page, London, UK.
56. Pope N. An examination of the use of peer rating for formative assessment in the context of the theory of consumption values. Assessment & Evaluation in Higher Education 2001, 26(3):235-246.
57. Ross A. The reliability, validity, and utility of self-assessment. Practical Assessment, Research & Evaluation 2006, 11(10):1-13. Available online:. http://pareonline.net/getvn.asp%3Fv=11%26n=10.
58. Ross S. Self-assessment in second language testing: A meta-analysis and analysis of experiential factors. Language Testing 1998, 15(1):1-20.
59. Ross S. The impact of assessment method on foreign language proficiency growth. Applied Linguistics 2005, 26(3):317-342.
60. Ross A., Starling M. Self-assessment in a technology-supported environment: The case of grade 9 geography. Assessment in Education: Principles, Policy, & Practice 2008, 15(2):183-199.
61. Sadler P.M., Good E. The impact of self-and peer-rating on student learning. Educational Assessment 2006, 11(1):1-31.
62. Saito H. EFL classroom peer assessment: Training effects on rating and commenting. Language Testing 2008, 25(4):553-581.
63. Saito H., Fujita T. Characteristics and user acceptance of peer rating in EFL writing classrooms. Language Teaching Research 2004, 8(1):31-54.
64. Saito H., Fujita T. Peer-assessing peers' contribution to EFL group presentations. RELC Journal 2009, 40(2):149-171.
65. Schaefer E. Rater bias patterns in an EFL writing assessment. Language Testing 2008, 25(4):465-493.
66. Schelfhout W., Dochy F., Janssens S. The use of self, peer and teacher assessment as a feedback system in a learning environment aimed at fostering skills of cooperation in an entrepreneurial context. Assessment & Evaluation in Higher Education 2004, 29(2):177-201.
67. Segers M., Dochy F. New assessment forms in problem-based learning: The value added of the students' perspective. Studies in Higher Education 2001, 26(3):327-343.
68. Shohamy E. Democratic assessment as an alternative. Language Testing 2001, 18(4):373-391.
69. Shohamy E., Gordon C.M., Kraemer R. The effect of raters' background and training on the reliability of direct writing tests. The Modern Language Journal 1992, 76(1):7-33.
70. Sullivan K., Hall C. Introducing students to self-assessment. Assessment & Evaluation in Higher Education 1997, 22(3):289-306.
71. Topping K.J. Peer assessment between students in colleges and universities. Review of Educational Research 1998, 68(3):249-276.
72. Topping K.J. Self- and peer-assessment in school and university: Reliability, validity and utility. Optimizing new methods of assessment: In search of qualities and standards 2003, 55-87. Kluwer Academic Publishers, Dordrecht, Netherlands. M. Segers, E. Cascallar (Eds.).
73. Topping K.J. Peer assessment. Theory into Practice 2009, 48(1):20-27.
74. Topping K.J. Peers as a source of formative assessment. Handbook of formative assessment 2010, 69-75. Routledge, New York, NY. H.L. Andrade, G.J. Cizek (Eds.).
75. van Gennip N.A.E., Segers M., Tillema H.H. Peer assessment for learning from a social perspective: The influence of interpersonal and structural features. Learning and Instruction 2009, 4(1):41-54.
76. Vu T., Dall'Alba G. Students' experience of peer assessment in a professional course. Assessment & Evaluation in Higher Education 2007, 32(5):541-556.
77. Weaver D., Esposto A. Peer assessment as a method of improving student engagement. Assessment & Evaluation in Higher Education 2012, 37(7):805-816.
78. Weigle S.C. Effects of training on raters of ESL compositions. Language Testing 1994, 11(2):197-223.
79. Weigle S.C. Using FACETS to model rater training effects. Language Testing 1998, 15(2):263-287.
80. Weigle S.C. Assessing writing 2002, Cambridge University Press, Cambridge, UK.
81. Wigglesworth G. Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction. Language Testing 1993, 10(3):305-323.
82. Winke P., Gass S., Myford C.M. The relationship between raters' prior language study and the evaluation of foreign language speech samples (TOEFL iBT™ Research Report No. 16) 2011, The Educational Testing Service, Princeton, NJ.