Is teaching experience necessary for reliable scoring of extended english questions?

Educational Measurement: Issues and Practice

Volumn 28, Issue 2, 2009, Pages 2-8

  Royal Dawson, Lucy ^a   Baird, Jo Anne ^a  

^a UNIVERSITY OF BRISTOL   (United Kingdom)

Author keywords

Accuracy; English test; Extended response; Scoring; Selection criteria

Indexed keywords

EID: 66749159684     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/j.1745-3992.2009.00142.x     Document Type: Article

References (31)

1. Baird, J., Mac, Q. (1999). How should examiner adjustments be calculated? A discussion paper. Research Committee Paper No. RC/13. Guildford, UK : Assessment and Qualifications Alliance.
2. Brooks, V. (2004) Double scoring revisited. British Journal of Educational Studies, 52 (1 29 46.
3. Brown, A. (1995). The effect of rater variables in the development of an occupation specific language performance test. Language Testing, 12 (1 1 15.
4. Coffman, W. E. (1971). Essay examinations. In R. L. Thorndike (Ed. Educational Measurement (2nd ed., pp. 271 302). Washington, DC : American Council on Education.
5. Cronbach, L. J. (1990) Essentials of Psychological Testing. Fifth Edition. New York : Harper & Row.
6. Curtis, P. (2005). Exam board under fire over scoring claims. The Guardian (http://education.guardian.co.uk/gcses/story/01557161,00.html accessed April 21, 2006).
7. Education Quality and Accountability Office (2006). Opportunities for Ontario educators: Scorer job description (http://www.eqao.com/Employment/ Employment.aspx?Lang=E#scoring accessed March 23, 2006).
8. Educational Testing Service (2006). Paper and pencil scoring opportunities (http://www.ets.org/portal/site/ets/menuitem. 22f30af61d34e9c39a77b13bc3921509/?vgnextoid= 88067f95494f4010VgnVCM10000022f95190RCRD accessed March 23, 2006).
9. Fowles, D. (2002). Evaluation of an e-scoring pilot in GCE Chemistry: Effects on scoring and examiners' views (Research Committee Paper No. RC/190 Guildford, UK : Assessment and Qualifications Alliance.
10. Fowles, D. E., Adams, C. (2005). How does assessment differ when e-scoring replaces paper-based scoring Paper presented at the 31st Annual Meeting of the International Association for Educational Assessment, Abuja, Nigeria.
11. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed. Educational Measurement (4th ed.). New York : Praeger.
12. Hambleton, R. K., Pitoniak, M. J. (2006). Setting performance standards. In R. L. Brennan (Ed. Educational Measurement (4th ed.). New York : Praeger.
13. International Baccalaureate Organization (2006). General terms of contract for IB diploma programme: Assistant examiners (http://www.ibo.org/ examiners/assistant-posts/assistantexaminers/documents/AEcontract1Aug02.doc accessed March 23, 2006).
14. Journal Officiel de la République Française (1993). Décret portant règlement général du baccalauréat general, Décret no. 93-1091 du 15 septembre 1993 (http://www.acnancy-metz.fr/enseign/lettres/inspection/Divers/baccalaureat-regl. htm accessed March 23, 2006).
15. Lumley, T. L., Lynch, B. K., McNamara, T. F. (1994). A new approach to standard setting in language assessment. Melbourne Papers in Language Testing, 3 (2 19 40.
16. Lunz, M. E., Stahl, J. A., Wright, B. D. (1994). Interjudge reliability and decision reproducibility. Educational and Psychological Measurement, 54, 913 925.
17. Meadows, M. L., Billington, L. (2005). A review of the literature on scoring reliability. Research Committee Paper No. RC/304. Guildford, UK : Assessment and Qualifications Alliance.
18. Murphy, R. J. L. (1978). Reliability of scoring in eight GCE examinations. British Journal of Educational Psychology, 48, 196 200.
19. Murphy, R. J. L. (1982). A further report of investigations into the reliability of scoring of GCE examinations. British Journal of Educational Psychology, 52, 58 63.
20. National Assessment Agency (2006). Behind every paper there's a student: Who can do it http://www.examinerrecruitment.org/whocandoit/default.htm accessed March 23, 2006).
21. New Zealand Qualifications Authority (2006). Role definition: Rater 2005 (http://www.nzqa.govt.nz/about/jobs/contracts/docs/rater.doc accessed March 28, 2006).
22. Pinot de Moira, A. (2003). Examiner backgrounds and the effect on scoring reliability. Research Committee Paper No. RC/218. Guildford, UK : Assessment and Qualifications Alliance.
23. Powers, D., Kubota, M. (1998). Qualifying essay readers for an Online Scoring Network (OSN). Princeton, NJ : Educational Testing Service.
24. Queensland Studies Authority (2006). Queensland Core Skills Test: New applicant information package (http://www.qsa.qld.edu.au/testing/raters/docs/ NewRaterpackage2006.pdf accessed March 23, 2006).
25. Rasbash, J., Goldstein, H. (1994) Efficient analysis of mixed hierarchical and cross-classified random structures using a multilevel model. Journal of Educational and Behavioral Statistics, 19 (4 337 350.
26. Scottish Qualifications Authority (2006). You've taken them this far Brochure http://www.sqa.org.uk/files-ccc/RatersRecruitmentForm.PDF accessed March 23, 2006).
27. Senior Secondary Assessment Board of South Australia (2006). Information on SSABSA Stage 2 Assessment Panels (http://www.ssabsa.sa.edu.au/panels/scoring. htm accessed March 23, 2006).
28. Shaw, M. (2005, August 26). Raters had 20 minutes' training. Times Educational Supplement (http://www.tes.co.uk/search/story/?story-id=2126566 accessed April 21, 2006).
29. Shohamy, E., Gordon, C., Kramer, R. (1992). The effects of raters' background and training on the reliability of direct writing tests. Modern Language Journal, 7 (1 27 33.
30. Shurnik, L. S., Nuttall, D. L. (1968). Describing the reliability of examinations. The Statistician, 18, 119 128.
31. Whetton, C., Newton, P. (2002, September). An evaluation of on-line scoring. Paper presented at the 28th International Association for Educational Assessment Conference, Hong Kong.