A quantitative analysis of cognitive strategy usage in the marking of two GCSE examinations

Assessment in Education: Principles, Policy and Practice

Volumn 15, Issue 1, 2008, Pages 73-89

  Suto, W M Irenka ^a   Greatorex, Jackie ^a  

^a University of Cambridge Local Examinations Syndicate   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77955152518     PISSN: 0969594X     EISSN: 1465329X     Source Type: Journal    
DOI: 10.1080/09695940701876177     Document Type: Article

References (57)

1. Alavi, S.M., 2005. On the adequacy of verbal protocols in examining an underlying construct of a test. Studies in Educational Evaluation, 31 (1): 1–26.
2. Baddeley, A., 1999. Essentials of human memory, Hove: Psychology Press.
3. Baird, J-A., Greatorex, J., and Bell, J.F., 2004. What makes marking reliable? Experiments with UK examinations. Assessment in Education, 11 (3): 333–347.
4. BioMedical Admissions Test. 2006. University of Cambridge Local Examinations Syndicate http://www.bmat.org.uk/index.html
5. Branthwaite, A., Trueman, M., and Berrisford, T., 1981. Unreliability of marking: further evidence and a possible explanation. Education Review, 33 (1): 41–46.
6. Cambridge International Examinations. 2006. University of Cambridge Local Examinations Syndicate http://www.cie.org.uk/CIE/WebSite/home.jsp
7. Crisp, V., and Johnson, M., 2007. The use of annotations in examination marking: opening a window into markers' minds. British Educational Research Journal, 53 (6): 943–961.
8. Cumming, A., 1990. Expertise in evaluating second language compositions. Language Testing, 7 (1): 31–51.
9. Delany, C.M., The evaluation of university students' written work. Paper presented at the annual conference of the British Educational Research Association. September14–17, Warwick, UK.
10. Department for Education and Skills. 2005. 14–19 Education and skills, London: The Stationary Office.
11. Dreyfus, H.L., and Dreyfus, S.E., 1986. Mind over machine: The power of human intuition and expertise in the era of the computer, New York: The Free Press.
12. Dunning, D., Heath, C., and Suls, J.M., 2004. Flawed self-assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5 (3): 69–106.
13. Ericsson, K., and Simon, H., 1993. Protocol analysis: verbal reports as data, London: MIT Press.
14. Gelman, R., Meck, E., and Merkin, S., 1986. Young children's numerical competence. Cognitive Development, 1 (1): 1–29.
15. Gilhooly, K.J., and Fioratou, E., Verbalisation and working memory aid effects on insight and non-insight problem solving. Poster presented at the British Psychological Society Conference. March21–23, York, UK.
16. Gilovich, T., Griffin, D., and Kahneman, D., 2002. Heuristics and biases. The psychology of intuitive judgment, Cambridge: Cambridge University Press.
17. Goldstein, H., and Cresswell, M., 1996. The comparability of different subjects in public examinations: a theoretical and practical critique. Oxford Review of Education, 22 (4): 435–442.
18. Greatorex, J., and Bell, J.F., 2004. Does the gender of examiners influence their marking?. Research in Education, 71: 25–36.
19. Greatorex, J., and Suto, W.M.I., An empirical exploration of human judgement in the marking of school examinations. Paper presented at the conference of the International Association for Educational Assessment. May21–26, Singapore.
20. Green, A., 1998. Studies in language testing 5: Verbal protocol analysis in language testing research–a handbook, Cambridge: Cambridge University Press/University of Cambridge Local Examinations Syndicate.
21. Hamilton, L.S., Nussbaum, E.M., and Snow, E.M., 1997. Interview procedures for validating science assessments. Applied Measurement in Education, 10 (2): 181–200.
22. Jankowicz, D., 2004. The easy guide to repertory grids, Chichester: John Wiley and Sons.
23. Kahneman, D., and Frederick, S., 2002. “ Representativeness revisited: attribute substitution in intuitive judgement ”. In Heuristics and biases: the psychology of intuitive judgement, Edited by: Gilovich, T., Griffin, D., and Kahneman, D., 49–81. Cambridge: Cambridge University Press.
24. Kahneman, D., Slovic, P., and Tversky, A., 1982. Judgement under uncertainty: Heuristics and biases, Cambridge: Cambridge University Press.
25. Kahneman, D,., and Tversky, A., 2000. Choices, values, and frames, Cambridge: Cambridge University Press.
26. Katz, I.R., Bennett, E., and Berger, A.E., 2000. Effects of response format on difficulty of SAT-mathematics items: it's not a strategy. Journal of Educational Measurement, 37 (1): 39–57.
27. Kramer, A., Coles, M., and Logan, G., 1996. Converging operations in the study of visual selective attention, Washington, DC: American Psychological Association.
28. Kuusela, H., and Paul, P., 2000. A comparison of concurrent and retrospective verbal protocol analysis. American Journal of Psychology, 113 (3): 387–404.
29. Laming, D., 1997. The measurement of sensation, Oxford: Oxford University Press.
30. Laming, D., 2004. Human judgement: The eye of the beholder, London: Thomson.
31. Leighton, J., Winter 2004. “ Avoiding misconception, misuse, and missed opportunities: the collection of verbal reports in educational achievement testing ”. In Educational Measurement: Issues and Practice Winter, 6–15.
32. Maughan, S., Monteath, P., Morley, J., and Sharp, S., Extending examination board activities: An international review of innovative responses to educational need. Paper presented at the International Association for Educational Assessment Conference. May21–26, Singapore.
33. Meadows, M., Can we predict who will be a reliable marker?. Paper presented at the Conference of International Association for Educational Assessment. May21–26, Singapore.
34. Milanovic, M., Saville, N., and Shuhong, S., 1996. “ A study of the decision-making behaviour of composition markers ”. In Studies in language testing 3: performance testing, cognition and assessment–selected papers from the 15th Language Testing Research Colloquium, Cambridge and Arnhem, Edited by: Milanovic, M., and Saville, N., 92–114. Cambridge: Cambridge University Press/University of Cambridge Local Examinations Syndicate.
35. National Assessment Agency. 2006. National Assessment Agency http://www.examinerrecruitment.org/faqs/default.htm#q10
36. Newton, P., 1996. The reliability of marking General Certificate of Secondary Education scripts: mathematics and English. British Educational Research Journal, 22 (4): 404–420.
37. OCR. 2007. University of Cambridge Local Examinations Syndicate http://www.ocr.org.uk/
38. Qualifications and Curriculum Authority. 2005. GCSE, GCE, VCE, GNVQ and AEA Code of Practice 2005/6, London: Qualifications and Curriculum Authority.
39. Qualifications and Curriculum Authority. 2006. GCSE, GCE, VCE, GNVQ and AEA Code of Practice 2006/7, London: Qualifications and Curriculum Authority.
40. Rabbitt, R., and Dornic, S., 1975. Attention and performance V, London: Academic Press.
41. Robinson, K.M., 2001. The validity of verbal reports in children's subtraction. Journal of Educational Psychology, 93 (1): 211–222.
42. Royal-Dawson, L., 2005. Is teaching experience a necessary condition for markers of Key Stage 3 English? Assessment and Qualifications Alliance report, London: Qualifications and Curriculum Authority. commissioned by the Qualification and Curriculum Authority
43. Russo, J.E., Johnson, E.J., and Stephens, D.L., 1989. The validity of verbal protocols. Memory and Cognition, 17 (6): 759–769.
44. Sanderson, J.P., 2001. Language and Differentiation in Examining at A Level, Leeds, UK: University of Leeds. Unpublished doctoral dissertation
45. Shohamy, E., Gordon, C.M., and Kraemer, R., 1992. The effects of raters' background and training on the reliability of direct writing tests. The Modern Language Journal, 76 (1): 27–33.
46. Stanovich, K.E., and West, R.F., 2002. “ Individual differences in reasoning ”. In Heuristics and biases: the psychology of intuitive judgement, Edited by: Gilovich, T., Griffin, D., and Kahneman, D., 421–440. Cambridge: Cambridge University Press.
47. Suto, W.M.I., Clare, I.C.H., Holland, A.J., and Watson, P.C., 2005a. Capacity to make financial decisions among people with mild intellectual disabilities. Journal of Intellectual Disability Research, 14 (3): 199–209.
48. Suto, W.M.I., Clare, I.C.H., Holland, A.J., and Watson, P.C., 2005b. The relationships among three factors affecting the financial decision-making abilities of adults with mild intellectual disabilities. Journal of Intellectual Disability Research, 14 (3): 210–217.
49. Suto, W.M.I., Clare, I.C.H., Holland, A.J., and Watson, P.C., 2006. Understanding of basic financial concepts among adults with mild learning disabilities. British Journal of Clinical Psychology, 45 (2): 261–266.
50. Suto, W.M.I., Greatorex, J., and J. What do GCSE examiners think of ‘thinking aloud’? Interesting findings from a preliminary study. Paper presented at the annual conference of the British Educational Research Association. September6–9, Warwick, UK. pp. s
51. Suto, W.M.I., Greatorex, J., and J. 2008. What goes through an examiner's mind? Using verbal protocols to gain insights into the GCSE marking process. British Educational Research Journal, 34 (2): 167–187.
52. Suto, W.M.I., and Nadas, R., in press. What determines GCSE marking accuracy? An exploration of expertise among maths and physics markers. Research Papers in Education.
53. Taylor, K.L., and Dionne, J.-P., 2000. Accessing problem-solving strategy knowledge: the complementary use of concurrent verbal protocols and retrospective debriefing. Journal of Educational Psychology, 92 (3): 413–425.
54. Van Someren, M., Barnard, Y., and Sandberg, J., 1994. The think aloud method: a practical guide to modelling cognitive processes, London: Academic Press.
55. Vaughan, C., 1992. “ Holistic assessment: what goes on in the rater's mind? ”. In Assessing second language writing in academic contexts, Edited by: Hamp-Lyons, L., 111–126. Norwood, NJ: Ablex.
56. Weigle, S.C., 1994. Effects of training on raters of ESL compositions. Language Testing, 11 (2): 197–223.
57. Wilson, T., 1994. The proper protocol: validity and completeness of verbal reports. Psychological Science, 5 (5): 249–252.