The interrelations of features of questions, mark schemes and examinee responses and their impact upon marker agreement

Assessment in Education: Principles, Policy and Practice

Volumn 18, Issue 3, 2011, Pages 295-318

  Black, Beth ^a   Suto, Irenka ^a   Bramley, Tom ^a  

^a University of Cambridge Local Examinations Syndicate   (United Kingdom)

Author keywords

Examinations; Marker accuracy; Marker reliability

Indexed keywords

EID: 79961042705     PISSN: 0969594X     EISSN: 1465329X     Source Type: Journal    
DOI: 10.1080/0969594X.2011.555328     Document Type: Article

References (49)

1. AERA (American Educational Research Association). 1999. Standards for educational and psychological testing. Washington, DC: American Psychological Association.
2. AQA (Assessment and Qualifications Alliance). 2004. GCE Business Studies Unit BUS2 Mark Scheme January 2004. http://www.thetimes100.co.uk/downloads/general_files/AQA/AQA-BUS2-W-MS-Jan04.pdf (accessed January 11, 2010).
3. Bejar, I.I., D.M. Williamson, and R.J. Mislevy. 2006. Human scoring. In Automated scoring of complex tasks in computer-based testing, ed. D.M. Williamson, R.J. Mislevy, and I.I. Bejar, 49-82. Mahwah, NJ: Lawrence Erlbaum.
4. Bell, J.F., T. Bramley, M.J.A. Claessen, and N. Raikes. 2006. Quality control of marking: Some models and simulations. Paper presented at the annual conference of the British Educational Research Association, September 6-9, in Warwick, UK.
5. Bennett, R.E., and D.H. Gitomer. 2009. Tranforming K-12 assessment: Integrating accountability testing, formative assessment and professional support. In Educational assessment in the 21st century, ed. C. Wyatt-Smith and J. Cumming, 43-61. New York: Springer.
6. Black, B. 2010. Investigating seeding items used for monitoring on-line marking: Factors affecting marker agreement with the gold standard marks. Paper presented at the annual conference of the International Association for Educational Assessment, August 22-27, in Bangkok, Thailand.
7. Black, B., and M. Curcin. 2010. Group dynamics in determining 'gold standard' marks for seeding items and subsequent marker agreement. Paper presented at the annual conference of the British Educational Research Association, September 1-4, at the University of Warwick, UK.
8. Bramley, T. 2001. The question tariff problem in GCSE Mathematics. Evaluation & Research in Education 15, no. 2: 95-107.
9. Bramley, T. 2007. Quantifying marker agreement: Terminology, statistics and issues. Research Matters: A Cambridge Assessment Publication 4: 22-28.
10. Bramley, T. 2008. Mark scheme features associated with different levels of marker agreement. Paper presented at the annual conference of the British Educational Research Association, September 3-6, in Edinburgh, UK.
11. Briggs, D. 1980. A study of the influence of handwriting upon grades using examination scripts. Educational Review 32, no. 2: 185-193.
12. Cambridge Assessment. 2009. The Cambridge Approach. Principles for designing, administering and evaluating assessment. http://www.cambridgeassessment.org.uk/ca/digitalAssets/181348_cambridge_approach.pdf (accessed January 12, 2010).
13. Clark, L. 2008. Pupils should not expect to receive 'accurate' results from exams, warns new watchdog. Mail Online. http://www.dailymail.co.uk/news/article-019938/Pupilsexpectreceive-accurate-results-exams-warms-watchdog (accessed December 14, 2009).
14. Crisp, V. 2008. Improving students' capacity to show their knowledge, understanding and skills in exams by using combined question and answer papers. Research Papers in Education 23, no. 1: 69-84.
15. Darling-Hammond, L. 2010. Performance counts: Assessment systems that support highquality learning. Washington, DC: Council of Chief State School Officers.
16. ETS (Educational Testing Service). 2002. ETS standards for quality and fairness. Educational Testing Service. http://www.ets.org/Media/About_ETS/pdf/standards.pdf (accessed September 28, 2010).
17. Frean, A. 2008. GCSE and A level exam results inaccurate, Ofqual warns pupils. Times Online. http://www.timesonline.co.uk/tol/life_and_style/education/article3941601.ece (accessed December 14, 2009).
18. Greatorex, J., and J.F. Bell. 2008. What makes AS marking reliable? An experiment with some stages from the standardisation process Research Papers in Education 23, no. 3: 233-255.
19. Hudson, G., B.H. Donahue, S. Rutt, and I. Schagen. 2007. Is electronic marking just about efficiency? Further analysis of electronic marking data to investigate factors related to marking reliability. Milton Keynes, UK: DRS Data Services Limited.
20. Johnson, M., and R. Nádas. 2009. An investigation into marker reliability and some qualitative aspects of on-screen essay marking. Research Matters: A Cambridge Assessment Publication 8: 2-9.
21. Johnson, M., R. Nádas, and J.F. Bell. 2009. Marking essays on screen: An investigation into the reliability of marking extended subjective texts. British Journal of Educational Technology 41, no. 5: 812-826.
22. Kahneman, D., and S. Frederick. 2002. Representativeness revisited: Attribute substitution in intuitive judgment. In Heuristics and biases: The psychology of intuitive judgment, ed. T. Gilovich, D. Griffin, and D. Kahneman, 49-81. Cambridge: Cambridge University Press.
23. Lebus, S. 2009. Issues of control and innovation: The role of the state in assessment systems. Opening remarks made at the 4th Cambridge Assessment Conference, October 19, 2009, at Robinson College, Cambridge. http://www.cambridgeassessment.org.uk/ca/digitalAssets/182126_annual_conference2009_opening_remarks.pdf (accessed December 14, 2009).
24. Massey, A. 1983. The effects of handwriting and other incidental variables on GCE 'A' Level marks in English Literature. Educational Review 35, no. 1: 45-50.
25. Massey, A., and T. Dexter. 2002. An evaluation of spelling, punctuation and grammar assessments in GCSE. Paper presented at the annual conference of the British Educational Research Association, September 11-14, in Exeter, UK.
26. Massey, A.J., and N. Raikes. 2006. Item level examiner agreement. Paper presented at the annual conference of the British Educational Research Association, September 6-9, in Warwick, UK.
27. McClellan, C.A. 2010. Constructed Response Scoring. R&D Connections. Princeton, NJ: Educational Testing Service.
28. Messick, S. 1989. Validity. In Educational measurement, 3rd ed., ed. R.L. Linn, 13-103. New York, NY: American Council on Education and Macmillan.
29. Myford, C.M., and E.W. Wolfe. 2003. Detecting and measuring rater effects using manyfacet Rasch measurement: Part I. Journal of Applied Measurement 4, no. 4: 386-422.
30. Nádas, R., and I. Suto. 2010. Speed isn't everything: A study of examination marking. Educational Studies 36, no. 1: 115-118.
31. Neyland, D. 2007. Achieving transparency: The visible, invisible and divisible in academic accountability networks. Organization 14, no. 4: 499-516.
32. Ofqual. 2009a. Reliability programme. http://www.ofqual.gov.uk/1999.aspx (accessed December 14, 2009).
33. Ofqual. 2009b. GCSE, GCE and AEA code of practice, April 2009. http://www.ofqual.gov.uk/files/2009-04-14-code-of-practice.pdf (accessed January 8, 2010).
34. Power, M. 1997. The audit society: Rituals of verification. Oxford: Oxford University Press.
35. Raikes, N., and A. Massey. 2007. Item-level examiner agreement. Research Matters: A Cambridge Assessment Publication 4: 34-38.
36. Royal-Dawson, L., and J.-A. Baird. 2009. Is teaching experience necessary for reliable scoring of extended English questions? Educational Measurement: Issues and Practice 28, no. 2: 2-8.
37. Shaw, M. 2005. Pupils will pay the price. Times Educational Supplement (TES), September 2. http://www.tes.co.uk/article.aspx?storycode=2128711 (accessed January 11, 2010).
38. Stanovich, K., and R. West. 2002. Individual differences in reasoning. In Heuristics and biases: The psychology of intuitive judgment, ed. T. Gilovich, D. Griffin, and D. Kahneman, 421-40. Cambridge: Cambridge University Press.
39. Stewart, W. 2009. On-screen marking gets A for accuracy. Times Educational Supplement (TES), September 18.
40. Stobart, G. 2010. Assessment fit-for-the-future. Paper presented at the annual conference of the International Association for Educational Assessment, August 22-27, in Bangkok, Thailand.
41. Suto, W.M.I., and J. Greatorex. 2008a. What goes through an examiner's mind? Using verbal protocols to gain insights into the GCSE marking process. British Educational Research Journal 34, no. 2: 213-233.
42. Suto, W.M.I., and J. Greatorex. 2008b. A quantitative analysis of cognitive strategy usage in the marking of two GCSE examinations. Assessment in Education: Principles, Policy & Practice 15, no. 1: 73-90.
43. Suto, W.M.I., and R. Nádas. 2008. What determines GCSE marking accuracy? An exploration of expertise among maths and physics markers. Research Papers in Education 23, no. 4: 477-479.
44. th International Conference on Thinking, June 22-26, in Kuala Lumpur, Malaysia.
45. Suto, W.M.I., and R. Nádas. 2009b. Why are some GCSE examination questions harder to mark accurately than others? Using Kelly's Repertory Grid technique to identify relevant question features. Research Papers in Education 24, no. 3: 335-377.
46. Suto, I., R. Nádas, and J.F. Bell. 2009. Who should mark what? A study of factors affecting marking accuracy in a biology examination. Research Papers in Education 26, no. 1: 21-51.
47. Sykes, R.C., K. Ito, and Z. Wang. 2008. Effects of assigning raters to items. Educational Measurement: Issues and Practice 27, no. 1: 47-55.
48. Wolf, A. 2009. The role of the state in educational assessment. Keynote presentation given at the 4th Cambridge Assessment Conference, October 19, in Cambridge, UK.
49. Woolcock, N. 2008. 60,000 pupils get wrong exam grades in marking fiasco. Times Online. http://www.timesonline.co.uk/tol/life_and_style/education/article5934150.ece (accessed December 14, 2009).