Why are some gcse examination questions harder to mark accurately than others? Using kelly’s repertory grid technique to identify relevant question features

Research Papers in Education

Volumn 24, Issue 3, 2009, Pages 335-377

  Suto, W M Irenka ^a   Nádas, Rita ^a  

^a University of Cambridge Local Examinations Syndicate   (United Kingdom)

Author keywords

GCSE; Kelly s Repertory Grid technique; Marking accuracy; Maths; Physics; Question features

Indexed keywords

EID: 85015534711     PISSN: 02671522     EISSN: 14701146     Source Type: Journal    
DOI: 10.1080/02671520801945925     Document Type: Article

References (52)

1. Ahmed, A., and A. Pollitt. 2007. Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education 14, no. 2: 201-232.
2. Baird, J.-A., J. Greatorex, and J.F. Bell. 2004. What makes marking reliable? Experiments with UK examinations. Assessment in Education 11, no. 3: 333-347.
3. Bramley, T. 2007. Quantifying marker agreement: Terminology, statistics and issues. Research Matters: A Cambridge Assessment Publication 4: 22-28.
4. Branthwaite, A., M. Trueman, and T. Berrisford. 1981. Unreliability of marking: Further evidence and a possible explanation. Education Review 33, no. 1: 41-46.
5. Butterworth, J., and G. Thwaites. 2005. Preparing for the BMAT: The official guide to the BioMedical admissions test. Oxford: Heinemann.
6. Crisp, V. 2007. Researching the judgement processes involved in A-level marking. Research Matters: A Cambridge Assessment Publication 4: 13-18.
7. Crisp, V., and M. Johnson. 2007. The use of annotations in examination marking: Opening a window into markers’ minds. British Educational Research Journal 33, no 6: 943-961.
8. Crisp, V., and E. Sweiry. 2006. Can a picture ruin a thousand words? The effects of visual resources and layout in exam questions. Educational Research 48, no 2: 139-154.
9. Cumming, A. 1990. Expertise in evaluating second language compositions. Language Testing 7: 31-51.
10. Fisher-Hoch, H., S. Hughes, and T. Bramley. 1997. What makes GCSE examination questions difficult? Outcomes of manipulating difficulty of GCSE questions. Paper presented at the annual conference of the British Educational Research Association in York, UK.
11. Greatorex, J. 2001. Making the grade - how question choice and type affect the development of grade descriptors. Educational Studies 27, no. 4: 451-464.
12. Greatorex, J., and J.F. Bell. 2004. Does the gender of examiners influence their marking? Research in Education 7: 25-36.
13. Greatorex, J., R. Nádas, W.M.I. Suto, and J.F. Bell. 2007. Exploring how the cognitive strategies used to mark examination questions relate to the efficacy of examiner training. Paper presented at the annual conference of the European Conference on Educational Research, September 1922, in Ghent, Belgium.
14. Greatorex, J., and W.M.I. Suto. 2005. What goes through a marker’s mind? Gaining theoretical insights into the A-level and GCSE marking process. Paper presented at the annual conference of Association for Educational Assessment (AEA)-Europe, November 3-5, in Dublin, Ireland.
15. Greatorex, J., and W.M.I. Suto. 2006. An empirical exploration of human judgement in the marking of school examinations. Paper presented at the 32nd annual conference of the International Association for Educational Research, May 21-26, Singapore.
16. Hamilton, F. 2007. New “dumbed-down” GCSE will be as easy as A, B, C. The Times, July 14. http://www.timesonline.co.uk/tol/life.
17. Herman, R.L. 1996. The repertory grid technique: Path to teacher description or teacher potential? Instructional Science 24: 439-459.
18. Hoskonen, K. 1999. A good pupil’s beliefs about mathematics learning assessed by repertory grid methodology. Psychology of Mathematics Education Conference 3: 97-104.
19. Hughes, S., A. Pollitt, and A. Ahmed. 1998. The development of a tool for gauging the demands of GCSE and A level exam questions. Paper presented at the annual conference of the British Educational Research Association, August 27-30, in Belfast, Ireland.
20. Jankowicz, D. 2004. The easy guide to repertory grids. Chichester: John Wiley and Sons.
21. Kahneman, D., and S. Frederick. 2002. Representativeness revisited: Attribute substitution in intuitive judgement. In Heuristics and biases: The psychology of intuitive judgement, ed. Thomas Gilovich, Dale Griffin, and Daniel Kahneman, 49-81. Cambridge: Cambridge University Press.
22. Kelly, G.A. 1955. The psychology of personal constructs. New York: Norton.
23. Laming, D. 1990. The reliability of a certain university examination compared with the precision of absolute judgements. Quarterly Journal of Experimental Psychology 42A: 239-254.
24. Laming, D. 2004. Human judgement: The eye of the beholder. London: Thomson.
25. Lengnink, K., and Prediger, S. 2003. Development of personal constructs about mathematical tasks - a qualitative study using repertory grid methodology. Psychology of Mathematics Education Conference 4: 39-46.
26. McCloughlin, T. 2002. Repertory grid analysis as a form of concept mapping in science education research. Irish Educational Studies 21, no. 2: 25-32.
27. Meadows, M. 2007. The effect of marker background and training on the quality of marking in GCSE English. Paper presented at the annual conference of the International Association for Educational Assessment, September 17-21, in Baku, Azerbaijan.
28. Meadows, M., and L. Billington. 2007. Can we predict who will be a reliable marker? Paper presented at the Institute of Assessors National Conference, June, in London.
29. Mikulas, C., and K. Kern. 2006. A comparison of accuracy of automated essay scoring using prompt-specific and prompt-independent training. Paper presented at the annual conference of the American Educational Research Association, April 7-11, in San Francisco, USA.
30. Milanovic, M., N. Saville, and S. Shuhong. 1996. A study of the decision-making behaviour of composition markers. In Studies in language testing 3: Performance testing, cognition and assessment, ed. Michael Milanovic and Nick Saville, 92-114. Cambridge: Cambridge University Press.
31. Murphy, R.J.L. 1978. Reliability of marking in eight GCE examinations. British Journal of Educational Psychology 48: 196-200.
32. Murphy, R.J.L. 1982. A further report of investigations into the reliability of marking GCE examinations. British Journal of Educational Psychology 52: 58-63.
33. Newton, P. 1996. The reliability of marking General Certificate of Secondary Education scripts: Mathematics and English. British Educational Research Journal 22: 404-420.
34. Pinot de Moira, A., C. Massey, J. Baird, and M. Morrissey. 2002. Marking consistency over time. Research in Education 67: 79-87.
35. Powers, D., and M. Kubota. 1998. Qualifying essay readers for an online scoring network. Educational Testing Service Report. Princeton, NJ: Educational Testing Service.
36. Qualifications and Curriculum Authority (QCA). 2007. GCSE, GCE, VCE, GNVQ and AEA Code of Practice 2006-2007. London: Qualifications and Curriculum Authority.
37. Raikes N. and A.J. Massey. 2007. Item-level examiner agreement. Research Matters: A Cambridge Assessment Publication 4: 34-37.
38. Rayment, T. 2000. Art teachers’ views of National Curriculum art: A repertory grid analysis. Educational Studies 26, no. 2: 165-176.
39. Royal-Dawson, L. 2005. Is teaching experience a necessary condition for markers of Key Stage 3 English? Assessment and Qualifications Alliance report, commissioned by the Qualification and Curriculum Authority.
40. Shohamy, E., C.M. Gordon, and R. Kraemer. 1992. The effects of raters’ background and training on the reliability of direct writing tests. The Modern Language Journal 76: 27-33.
41. Siraj-Blatchford, J. 1995. Kelly’s repertory grid: a technique for developing evaluation in design and technology. Paper presented at the Design and Technology Educational Research and Curriculum Development conference, Loughborough University, in Loughborough, UK.
42. Stahl, J.A., and M.E. Lunz. 1996. Judge performance reports: Media and message. In Objective measurement. Theory into practice, ed J.R. Engelhard and M. Wilson, 113-126. Norwood, NJ: Ablex Publishing.
43. Stanovich, K.E. and R.F. West. 2002. Individual differences in reasoning. In Heuristics and biases: The psychology of intuitive judgement, ed. Thomas Gilovich, Dale Griffin, and Daniel Kahne-man, 421-440. Cambridge: Cambridge University Press.
44. Suto, W.M.I., and J. Greatorex. 2006. A cognitive psychological exploration of the GCSE marking process. Research Matters: A Cambridge Assessment Publication 2: 7-11.
45. Suto, W.M.I., and J. Greatorex. 2007. What goes through an examiner’s mind? Using verbal protocols to gain insights into the GCSE marking process. British Educational Research Journal 34, no. 2: 213-233.
46. Suto, W.M.I., and J. Greatorex. 2008. A quantitative analysis of cognitive strategy usage in the marking of two GCSE examinations. Assessment in Education. Principles, policy and practice 15, no. 1: 73-89.
47. Suto, W.M.I., and R. Nádas. 2007. Research matters. A Cambridge Assessment Publication 4: 2-5.
48. Suto, W.M.I., and R. Nádas. In press. What determines GCSE marking accuracy? An exploration of expertise among maths and physics markers. Research Papers in Education.
49. Valentine, C.M. 1932. The reliability of examinations: An enquiry with special reference to the entrance examinations to secondary schools, the school certificate examinations, and the award of scholarships at universities. London: University of London Press.
50. Vaughn, C. 1992. Holistic assessment: What goes on in the rater’s mind? In Assessing second language writing in academic contexts, ed. L. Hamp-Lyons. Norwood, NJ: Ablex Publishing Corporation.
51. Wigglesworth, G. 1993. Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction. Language Testing 10, no. 3: 305-335.
52. Wood, A., and A. Pollitt. In submission. The development of the Preferred Alternative Construct Elicitation (PACE) approach to eliciting the construct of demand from student reviewers.