Ofqual's Reliability Programme: A case study exploring the potential to improve public understanding and confidence

Oxford Review of Education

Volumn 39, Issue 1, 2013, Pages 93-113

  Newton, Paul E ^a  

^a University of Cambridge Local Examinations Syndicate   (United Kingdom)

Author keywords

assessment; error; examinations; Ofqual; reliability

Indexed keywords

EID: 84874494845     PISSN: 03054985     EISSN: 14653915     Source Type: Journal    
DOI: 10.1080/03054985.2012.760285     Document Type: Article

References (57)

1. Baird, J., Beguin, A., Black, P., Pollitt, A. & Stanley, G. (2011). The Reliability Programme: final report of the technical advisory group. Ofqual/11/4825. Coventry, Ofqual.
2. Black, P., Gardner, J. & Wiliam, D. (2007). Joint memorandum, in: House of Commons Children, Schools and Families Committee Testing and Assessment. Third Report of session 2007-08. Volume II. Oral and written evidence. HC 169-II (Ev.202-205). London, The Stationery Office.
3. Black, B., Suto, I. and Bramley, T. 2011. The interrelations of features of questions, mark schemes and examinee responses and their impact upon marker agreement. Assessment in Education: Principles, Policy & Practice, 18(3): 295-318.
4. Boyle, A., Opposs, D. & Kinsella, A. (2009). No news is good news? Talking to the public about the reliability of assessment. Ofqual/09/4362. Paper presented at the 35th International Association for Educational Assessment (IAEA) Annual Conference, Brisbane, 13-18 September. Coventry, Ofqual.
5. Burslem, S. (2011). The Reliability Programme. Final report of the Policy Advisory Group. Ofqual/11/4829. Coventry, Ofqual.
6. Chamberlain, S. (2010). Public perceptions of reliability. Ofqual/10/4708. Coventry, Ofqual.
7. Charlton, J. (2009, 18 September). No room for error in exams. Sun.
8. Clark, L. (2009, 18 March). One in three children is given wrong grade for SATs every year. MailOnline.
9. Clark, L. (2010a, 6 February). Strictly come marking: As 1 in 3 pupils get wrong grade, SATs are branded as unreliable as TV show judging. MailOnline.
10. Clark, L. (2010b, 27 July). One in five SATs results are incorrect. Daily Mail.
11. Collins, N. (2010, 26 July). One in five pupils receive wrong Sats grade. Telegraph.
12. Crisp, V. 2008. Exploring the nature of examiner thinking during the process of examination marking. Cambridge Journal of Education, 38(2): 247-264.
13. Crisp, V. and Johnson, M. 2007. The use of annotations in examination marking: opening a window into markers' minds. British Educational Research Journal, 33(6): 943-961.
14. Edgeworth, F.Y. (1888). The statistics of examinations. Journal of the Royal Statistical Society, LI, 599-635.
15. Greatorex, J. and Bell, J.F. 2004. Does the gender of examiners influence their marking?. Research in Education, 71: 25-36.
16. Hartog, P. and Rhodes, E.C. 1935. An examination of examinations, London: Macmillan.
17. He, Q., Boyle, A. and Opposs, D. 2011. Public perceptions of reliability in examination results in England. Evaluation & Research in Education, 24(4): 255-283.
18. He, Q. and Opposs, D. 2011. The reliability of results from national tests, public examinations, and vocational qualifications in England. Educational Research and Evaluation, 18(8): 779-799.
19. House of Commons Children, Schools and Families Committee (2008). Testing and assessment. Third report of session 2007-08. Volume II. Oral and written evidence. HC 169-II. London, TSO Ltd.
20. Hurst, G. (2011, 8 December). Thousands of A-level results upgraded as complaints soar. The Times.
21. Ipsos MORI (2009). Public perceptions of reliability in examinations: A research study conducted for Ofqual. Coventry, Ofqual.
22. Lawn, M. (Ed.) (2008). An Atlantic crossing? The work of the International Examinations Inquiry, its researchers, methods and influence. Oxford, Symposium Books.
23. Mansell, W. & Paton, G. (2009, 12 May). Exam grades 'should be scrapped'. Telegraph.co.uk.
24. Mansell, W. (2003, 17 October). Test result bombshell kept under wraps. The Times Educational Supplement.
25. Mansell, W. (2006, 18 August). Persistent professor returns. The Times Educational Supplement.
26. Murphy, R.J.L. 1978. Reliability of marking in eight GCE examinations. British Journal of Educational Psychology, 48: 196-200.
27. Newton, P.E. 1996. The reliability of marking of General Certificate of Secondary Education Scripts: mathematics and English. British Educational Research Journal, 22(4): 405-420.
28. Newton, P.E. 2005a. The public understanding of measurement inaccuracy. British Educational Research Journal, 31(4): 419-442.
29. Newton, P.E. 2005b. Threats to the professional understanding of assessment error. Journal of Education Policy, 20(4): 457-483.
30. Newton, P.E. 2007. "Contextualising the comparability of examination standards". In Techniques for monitoring the comparability of examination standards, Edited by: Newton, P.E., Baird, J., Goldstein, H., Patrick, H. and Tymms, P. London: Qualifications and Curriculum Authority.
31. Newton, P. E. (2008a). Recognising the error of our ways. Paper presented at the Cambridge Assessment Forum for New Developments in Educational Assessment. Downing College, Cambridge, 10 December.
32. Newton, P. E. (2008b). Ofqual's reliability of results programme. Paper presented at the Association of Colleges Annual Conference (Changing Landscapes). International Convention Centre, Birmingham, 18-20 November.
33. Newton, P.E. 2011. A-level pass rates and the enduring myth of norm-referencing. Research Matters, Special Issue, 2: 20-26.
34. Ofqual (2011a). Inquiry into the failure of part of AQA's GCSE, AS and A-level script-marking process in the summer 2010 examination series. Coventry, Ofqual.
35. Ofqual (2011b, 4 August). Action taken on exam errors. Available online at:http://www.ofqual.gov.uk/news-and-announcements/130-news-and-announcements-press-releases/720-action-taken-on-exam-errors.
36. Ofqual (2012). GCSE English 2012. Coventry, Ofqual.
37. Opposs, D. & He, Q. (2011). The Reliability Programme. Final report. Ofqual/11/4828. Coventry, Ofqual.
38. Opposs, D. & He, Q. (2011). Ofqual's reliability compendium. Coventry, Ofqual.
39. Paton, G. (2009a, 30 January). Exam marking errors'inevitable', says regulator. Telegraph.co.uk.
40. Paton, G. (2009b). Exam watchdog bans 'errors'. Telegraph.co.uk, 18 September.
41. Paton, G. (2009c, 13 November). Sats results 'wrong for thousands of pupils'. Telegraph.co.uk.
42. Paton, G. (2011, 8 December). Record number of A-levels marked up amid scramble for university places. The Daily Telegraph.
43. Petch, J.A. 1953. Fifty years of examining, London: Harrap.
44. Roach, J. 1971. Public examinations in England 1850-1900, London: Cambridge University Press.
45. Ross, T. (2009, 31 January). New Sats fiasco as one in three pupils'will get wrong exam results'. MailOnline.
46. Shepard, J. (2009, 22 December). Exam results could carry inaccuracy warning. Guardian.co.uk.
47. Stewart, W. (2009a, 27 March). Ofqual chief wrong-footed in marking row. The Times Educational Supplement.
48. Stewart, W. (2009b, 18 September). Exams: Ofqual admits to 'shying away' from talking about test result reliability. The Times Educational Supplement.
49. Stewart, W. (2010, 19 March). Primary teachers have 'almost blind faith' in grades. The Times Educational Supplement.
50. Sutherland, S. (2008). The Sutherland Inquiry: an independent inquiry into the delivery of National Curriculum tests in 2008. A report to Ofqual and the Secretary of State for Children, Schools and Families. London, The Stationery Office.
51. Suto, W.M.I. and Greatorex, J. 2008. A quantitative analysis of cognitive strategy usage in the marking of two GCSE examinations. Assessment in Education: Principles, Policy & Practice, 15(1): 73-89.
52. Suto, W.M.I. and Greatorex, J. 2008. What goes through an examiner's mind? Using verbal protocols to gain insights into the GCSE marking process. British Educational Research Journal, 34(2): 213-233.
53. Tattersall, K. 2007. "A brief history of policies, practices and issues relating to comparability". In Techniques for monitoring the comparability of examination standards, Edited by: Newton, P.E., Baird, J., Goldstein, H., Patrick, H. and Tymms, P. London: Qualifications and Curriculum Authority.
54. Tattersall, K. (2008). Speech at Ofqual Launch Event, National Motorcycle Museum. Solihull, 16 May.
55. Warmington, P. and Murphy, R. 2004. Could do better? Media depictions of UK educational assessment results. Journal of Education Policy, 19(3): 285-299.
56. Wiliam, D. (2001). Reliability, validity, and all that jazz. Education, 3-13, 29(3), 17-21.
57. Wiseman, S. 1961. "The efficiency of examinations". In Examinations in education, Edited by: Wiseman, S. Manchester: Manchester University Press.