Summative assessment: Dealing with the 'measurement fallacy'

Studies in Higher Education

Volumn 36, Issue 3, 2011, Pages 251-273

  Yorke, Mantz ^a  

^a LANCASTER UNIVERSITY   (United Kingdom)

Author keywords

Academic achievement; Academic success; Assessment; Grading; Measurement

Indexed keywords

EID: 79957839881     PISSN: 03075079     EISSN: 1470174X     Source Type: Journal    
DOI: 10.1080/03075070903545082     Document Type: Article

References (73)

1. Attwood, R. 2009. Why is it so difficult for law students to get a first? The Times, October 15. http://business.timesonline.co.uk/tol/business/law/student/article6872381.ece (accessed November 24, 2009).
2. Australian Vice-Chancellors' Committee. 2002. Grades for honours programs (concurrent with pass degree), 2002. http://www.avcc.edu.au/documents/universities/key_survey_ summaries/Grades_for_Degree_Subjects_Jun02.xls (accessed November 24, 2009).
3. Baume, D., and M. Yorke, with M. Coffey. 2004. What is happening when we assess, and how can we use our understanding of this to improve assessment? Assessment and Evaluation in Higher Education 29, no. 4: 451-77.
4. Bloxham, S. 2009. Marking and moderation in the UK: False assumptions and wasted resources. Assessment and Evaluation in Higher Education 34, no. 2: 209-20.
5. Brandon, J., and M. Davies. 1979. The limits of competence in social work: The assessment of marginal work in social work education. British Journal of Social Work 9, no. 3: 295-347.
6. Bridges, P., B. Bourdillon, D. Collymore, A. Cooper, W. Fox, C. Haines, D. Turner, H. Woolf, and M. Yorke. 1999. Discipline-related marking behaviour using percentages: A potential cause of inequity in assessment. Assessment and Evaluation in Higher Education 24, no. 3: 285-300.
7. Brooks, V. 2004. Double marking revisited. British Journal of Educational Studies 52, no. 1: 29-46.
8. Brumfield, C. 2004. Current trends in grades and grading practices in higher education: Results of the 2004 AACRAO survey. Washington, DC: American Association of Collegiate Registrars and Admissions Officers.
9. Carless, D. 2009. Trust, distrust and their impact on assessment reform. Assessment and Evaluation in Higher Education 34, no. 1: 79-89.
10. Claxton, G. 1998. Hare brain, tortoise mind. London: Fourth Estate.
11. Cliff, N. 1988. The eigenvalues-greater-than-one rule and the reliability of components. Psychological Bulletin 103, no. 2: 276-79.
12. Cowan, N. 2000. The magical number 4 in short-term memory: A reconsideration of mental storage capacity [plus subsequent commentaries]. Behavioral and Brain Science 24, no. 1: 87-185.
13. Dalziel, J. 1998. Using marks to assess student performance: Some problems and alternatives. Assessment and Evaluation in Higher Education 23, no. 4: 351-66.
14. Dennis, I., S.E. Newstead, and D.E. Wright. 1996. A new approach to exploring biases in educational assessment. British Journal of Psychology 87, no. 4: 515-34.
15. Dreyfus, H.L., and S.E. Dreyfus. 2005. Expertise in real world contexts. Organization Studies 26, no. 5: 779-92.
16. Ecclestone, K. 2001. 'I know a 2:1 when I see it': Understanding criteria for degree classifications in franchised university programmes. Journal of Further and Higher Education 25, no. 3: 301-313.
17. Einhorn, H.J. 1972/2000. Expert judgement: Some necessary conditions and an example. In Judgement and decision making: An interdisciplinary reader, ed. T. Connolly, H.R. Arkes, and K.R. Hammond, 2nd ed., 324-35. Cambridge: Cambridge University Press. [Abridged version of 'Expert judgement and mechanical combination', Organizational Behavior and Human Performance 7, no. 1: 86-106.]
18. Eisner, E.W. 1985. The art of educational evaluation: A personal view. London: Falmer Press.
19. Ekstrom, R.B., and A.M. Villegas. 1994. College grades: An exploratory study of policies and practices. New York: College Entrance Examination Board.
20. Elander, J., and D. Hardman. 2002. An application of judgment analysis to examination marking in psychology. British Journal of Psychology 93, no. 3: 303-28.
21. Elton, L., and B. Johnston. 2002. Assessment in universities: A critical review of research. http://www.heacademy.ac.uk/resources/detail/resource_database/ id13_assessment_in_universities (accessed November 24, 2009).
22. European Commission. 2009. ECTS users' guide. Brussels: European Commission. http:// ec.europa.eu/education/lifelong-learning-policy/doc/ects/guide_en.pdf (accessed September 13, 2009).
23. Gigerenzer, G., P. Todd, and the ABC Research Group. 1999. Simple heuristics that make us smart. Oxford: Oxford University Press.
24. Govaerts, M.J.B., C.P.M. van der Vleuten, L.W.T. Schuwirth, and A.M.M. Muitjens. 2007. Broadening perspectives on clinical performance assessment: Rethinking the nature of intraining assessment. Advances in Health Sciences Education 12, no. 2: 239-60.
25. Grainger, P., K. Purnell, and R. Zipf. 2008. Judging quality through substantive conversations between markers. Assessment and Evaluation in Higher Education 33, no. 2: 133-42.
26. Guba, E.G., and Y.S. Lincoln. 1989. Fourth generation evaluation. London: Sage.
27. Hager, P., and J. Butler. 1996. Two models of educational assessment. Assessment and Evaluation in Higher Education 21, no. 4: 367-78.
28. Hand, L., and D. Clewes. 2000. Marking the difference: An investigation of the criteria used for assessing undergraduate dissertations in a business school. Assessment and Evaluation in Higher Education 25, no. 1: 5-21.
29. Hawe, E. 2003. 'It's pretty difficult to fail': The reluctance of lecturers to award a failing grade. Assessment and Evaluation in Higher Education 28, no. 4: 371-82.
30. Hornby, W. 2003. Assessing using grade-related criteria: A single currency for universities? Assessment and Evaluation in Higher Education 28, no. 4: 435-54.
31. Innovation, Universities, Science and Skills Committee of the House of Commons. 2009. Students and universities. Eleventh Report of Session 2008-09, 2 volumes. London: The Stationery Office.
32. Jawitz, J. 2009. Learning in the academic workplace: The harmonisation of the collective and the individual habitus. Studies in Higher Education 34, no. 6: 601-14.
33. Johnson, V.E. 2003. Grade inflation: A crisis in college education. New York: Springer.
34. Johnston, B. 2004. Summative assessment of portfolios: An examination of different approaches to agreement over outcomes. Studies in Higher Education 29, no. 3: 395-412.
35. Kahneman, D., and S. Frederick. 2002. Representativeness revisited: Attribute substitution in intuitive judgment. In Heuristics and biases: The psychology of intuitive judgment, ed. T. Gilovich, D. Griffin, and D. Kahneman, 49-81. Cambridge: Cambridge University Press.
36. Kline, P. 1994. An easy guide to factor analysis. London: Routledge.
37. Knight, P.T. 2002. Summative assessment in higher education: Practices in disarray. Studies in Higher Education 27, no. 3: 275-86.
38. Knight, P.T. 2006. The local practices of assessment. Assessment and Evaluation in Higher Education 31, no. 4: 435-52.
39. Knight, P.T. 2007. Grading, classifying and future learning. In Rethinking assessment in higher education: Learning for the longer term, ed. D. Boud and N. Falchikov, 72-86. Abingdon: Routledge.
40. Knight, P.T., and A. Page. 2007. The assessment of 'wicked' competences. Report to the Practice- Based Professional Learning Centre. http://www.open.ac.uk/cetl-workspace/cetlcontent/ documents/460d21bd645f8.pdf (accessed November 24, 2009).
41. Knight, P.T., and M. Yorke. 2003. Assessment, learning and employability. Maidenhead: Open University Press.
42. Knight, P.T., and M. Yorke. 2006. Employability: Judging and communicating achievements. York: The Higher Education Academy. http://www.heacademy.ac.uk/resources/detail/ ourwork/employability/employability338 (accessed November 24, 2009).
43. Law, J. 2004. After method: Mess in social science research. London: Routledge.
44. Lievens, F. 2001. Assessor training strategies and their effects on accuracy, interrater reliability and discriminant validity. Journal of Applied Psychology 86, no. 2: 225-64.
45. Middendorf, C.H., and T.H. Macan. 2002. Note-taking in the employment interview: Effects on recall and judgments. Journal of Applied Psychology 87, no. 2: 293-303.
46. Miller, G.A. 1956. The magic number seven, plus or minus two. Psychological Review 63, no. 2: 81-97.
47. Price, M. 2005. Assessment standards: The role of communities of practice and the scholarship of assessment. Assessment and Evaluation in Higher Education 30, no. 3: 215-30.
48. Price, M., and C. Rust. 1999. The experience of introducing a common criteria assessment grid across an academic department. Quality in Higher Education 5, no. 2: 133-44.
49. Quality Assurance Agency for Higher Education (QAA). 2006. Code of practice for the assurance of academic quality and standards in higher education. Section 6: Assessment of students. Gloucester: QAA.
50. Quality Assurance Agency for Higher Education (QAA). 2008. Outcomes from institutional audit: Assessment of students. Second series. Gloucester: QAA.
51. Rust, C. 2007. Towards a scholarship of assessment. Assessment and Evaluation in Higher Education 32, no. 2: 229-37.
52. Rust, C., B. O'Donovan, and M. Price. 2005. A social constructivist assessment process model: How the research literature shows us this could be best practice. Assessment and Evaluation in Higher Education 30, no. 3: 231-40.
53. Sadler, D.R. 1987. Specifying and promulgating achievement standards. Oxford Review of Education 13, no. 2: 191-209.
54. Sadler, D.R. 2005. Interpretations of criteria-based assessment and grading in higher education. Assessment and Evaluation in Higher Education 30, no. 2: 176-94.
55. Sadler, D.R. 2009a. Indeterminacy in the use of preset criteria for assessment and grading. Assessment and Evaluation in Higher Education 34, no. 2: 159-79.
56. Sadler, D.R. 2009b. Transforming holistic assessment and grading into a vehicle for learning. In Assessment, learning and judgement in higher education, ed. G. Joughin, 45-63. Dordrecht: Springer.
57. Sadler, D.R. 2010. Fidelity as a precondition for integrity in grading academic achievement. Assessment and Evaluation in Higher Education 35, no. 6: 727-43.
58. Sanchez, J.I., and P. de la Torre. 1996. A second look at the relationship between rating and behavioral accuracy in performance appraisal. Journal of Applied Psychology 81, no. 1: 3-10.
59. Saunders, M.N.K., and S.M. Davis. 1998. The use of assessment criteria to ensure consistency of marking: Some implications for good practice. Quality Assurance in Education 6, no. 3: 162-71.
60. Stanovich, K., and R. West. 2002. Individual differences in reasoning. In Heuristics and biases: The psychology of intuitive judgement, ed. T. Gilovich, D. Griffin, and D. Kahneman, 421-40. Cambridge: Cambridge University Press.
61. Walvoord, B.E., and V.J. Anderson. 1998. Effective grading: A tool for learning and assessment. San Francisco: Jossey-Bass.
62. Webster, F., D. Pepper, and A. Jenkins. 2000. Assessing the undergraduate dissertation. Assessment and Evaluation in Higher Education 25, no. 1: 71-80.
63. Woolf, H. 2004. Assessment criteria: Reflections on current practices. Assessment and Evaluation in Higher Education 29, no. 4: 479-93.
64. Yorke, M. 2008a. Grading student achievement in higher education: Signals and shortcomings. Abingdon: Routledge.
65. Yorke, M. 2008b. Faulty signals? Inadequacies of grading systems and a possible response. In Assessment, learning and judgement, ed. G. Joughin, 65-84. Dordrecht: Springer.
66. Yorke, M. 2009a. Honours degree classifications: What we can and cannot tell from the statistics. Gloucester: Quality Assurance Agency for Higher Education.
67. Yorke, M. 2009b. Trends in honours degree classifications, 1994-95 to 2006-07, for England, Wales and Northern Ireland. http://www.heacademy.ac.uk/resources/detail/publications/ trends_in_honours_degree_classifications (accessed November 24, 2009).
68. Yorke, M. 2010. How finely grained does summative assessment need to be? Studies in Higher Education 35, no. 6: 677-89.
69. Yorke, M., G. Barnett, P. Bridges, P. Evanson, C. Haines, D. Jenkins, P. Knight, D. Scurry, M. Stowell, and H. Woolf. 2002. Does grading method influence honours degree classification? Assessment and Evaluation in Higher Education 27, no. 3: 269-79.
70. Yorke, M., P. Bridges, and H. Woolf. 2000. Mark distributions and marking practices in UK higher education. Active Learning in Higher Education 1, no. 1: 7-27.
71. Yorke, M., A. Cooper, W. Fox, C. Haines, P. McHugh, D. Turner, and H. Woolf. 1996. Module mark distributions in eight subject areas and some issues they raise. In Modular higher education in the UK, ed. N. Jackson, 105-7. London: Higher Education Quality Council.
72. Yorke, M., H. Woolf, M. Stowell, R. Allen, C. Haines, M. Redding. D. Scurry, G. Taylor- Russell, W. Turnbull, and L. Walker. 2008. Enigmatic variations: Honours degree assessment regulations in the UK. Higher Education Quarterly 62, no. 3: 157-80.
73. Zwick, W.R., and W.F. Velicer. 1986. Comparison of five rules for determining the number of components to retain. Psychological Bulletin 99, no. 3: 432-42.