What do GCSE examiners think of 'thinking aloud'? Findings from an exploratory study

Educational Research

Volumn 50, Issue 4, 2008, Pages 319-331

  Greatorex, Jackie ^a   Irenka Suto, W M ^a  

^a University of Cambridge Local Examinations Syndicate   (United Kingdom)

Author keywords

Assessment; Examiners; GCSE; Thinking aloud; Verbal protocols

Indexed keywords

EID: 56449123114     PISSN: 00131881     EISSN: 14695847     Source Type: Journal    
DOI: 10.1080/00131880802499720     Document Type: Article

References (30)

1. Anderson, M.J. 1985. Some evidence on the effect of verbalisation on process: A methodological note. Journal of Accounting Research 23: 843-52.
2. Clark, H.H., and S.E. Haviland. 1977. Comprehension and the given-new contract. In Discourse production and comprehension, ed. R.O. Freidle, 1-40. Norwood, NJ: Ablex.
3. Crisp, V. 2008. The validity of using verbal protocol analysis to investigate the processes involved in examination marking. Research in Education 79: 1-12.
4. Cumming, A. 1990. Expertise in evaluating second language compositions. Language Testing 7: 31-51.
5. Dreyfus, H.L., and S.E. Dreyfus. 1986. Mind over machine: The power of human intuition and expertise in the era of the computer. Oxford: Blackwell.
6. Ericsson, K., and H. Simon. 1980. Verbal reports as data. Psychological Review 87, no. 3: 215-51.
7. _. 1993. Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
8. Green, A. 1998. Studies in language testing, 5: Verbal protocol analysis in language testing research. Cambridge: Cambridge University Press.
9. Hamilton, L.S., E.M. Nussbaum, and E.M. Snow. 1997. Interview procedures for validating science assessments. Applied Measurement in Education 10: 181-200.
10. Hayes, J.R., and L. Flower. 1980. Identifying organisation of writing process. In Cognitive processes in writing, ed, L. Gregg, and E. Steinberg, 3-30. Mahwah, NJ: Lawrence Erlbaum.
11. Jankowicz, D. 2004. The easy guide to repertory grids. Chichester: John Wiley.
12. Johnson, M.J., and S. Green. 2006. Online mathematics assessment: The impact of mode on performance and question answering strategies. Journal of Technology, Learning and Assessment 4. http://www.jtla.org (last accessed 7 October 2008).
13. Kahneman, D., and S. Frederick. 2002. Representativeness revisited: Attribute substitution in intuitive judgment. In Heuristics and biases: The psychology of intuitive judgement, ed. T. Gilovich, D. Griffin, and D. Kahneman. Cambridge: Cambridge University Press.
14. Katz, I.R., E. Bennett, and A.E. Berger. 2000. Effects of response format on difficulty of SAT Mathematics items: It's not a strategy. Journal of Educational Measurement 37: 39-57.
15. Kuusela, H., and P. Paul. 2000. A comparison of concurrent and retrospective verbal protocol analysis. American Journal of Psychology 113: 387-404.
16. Leighton, J. 2004. Avoiding misconception, misuse, and missed opportunities: The collection of verbal reports in educational achievement testing. Educational Measurement: Issues and Practice no. Winter: 6-15.
17. Milanovic, M., N. Saville, and S. Shuhong. 1996. A study of decision-making behaviour of composition markers. In Studies in language testing: 3, ed. M. Milanovic, and N. Saville. Cambridge: Cambridge University Press.
18. Nisbett, R.E., and T.D. Wilson. 1977. Telling more than we can know: Verbal reports on mental processes. Psychological Review 84: 231-59.
19. Robinson, K.M. 2001. The validity of verbal reports in children's subtraction. Journal of Educational Psychology 93: 211-22.
20. Sanderson, P.J. 2001. Language and differentiation in examining at 'A'-level. Unpublished doctoral dissertation, University of Leeds.
21. Stanovich, K., and R. West. 2002. Individual differences in reasoning. In Heuristics and biases: The psychology of intuitive judgment, ed. T. Gilovich, D. Griffin, and D. Kahneman. Cambridge: Cambridge University Press.
22. Sütö, W.M.I., and J. Greatorex. 2006. A cognitive psychological exploration of the GCSE marking process. Research Matters: A Cambridge Assessment Publication 2: 7-11.
23. _. 2008. What goes through an examiner's mind? Using verbal protocols to gain insights into the GCSE marking process. British Educational Research Journal 34: 213-33.
24. Taylor, K.L., and J.-P. Dionne. 2000. Accessing problem-solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing. Journal of Educational Psychology 92: 413-25.
25. Van Someren, M., Y. Barnard, and J. Sandberg. 1994. The think aloud method: A practical guide to modelling cognitive processes. London: Academic Press.
26. Vaughan, C. 1992. Holistic assessment: What goes on in the rater's mind? In Assessing second language writing in academic contexts, ed. L. Hamp-Lyons. Norwood, NJ: Ablex.
27. Weigle, S.C. 1994. Effects of training on raters of ESL composition. Language Testing 11: 197-223.
28. Wilson, T. 1994. The proper protocol: Validity and completeness of verbal reports. Psychological Science 5: 249-52.
29. Wood, A., and A. Pollitt, forthcoming. The development of the Preferred Alternative Construct Ellicitation (PACE) approach to eliciting the construct of demand from student reviewers. QCA report.
30. Zucker, S., C. Sassman, and B.J. Case. 2004. Cognitive labs: Harcourt technical report. San Antonio, TX: Harcourt Assessment Inc.