Using calibrated exemplars in the teacher-assessment of writing: An empirical study

Educational Research

Volumn 55, Issue 3, 2013, Pages 219-235

  Heldsinger, Sandra A ^a   Humphry, Stephen M ^a  

^a UNIVERSITY OF WESTERN AUSTRALIA   (Australia)

Author keywords

accountability; calibrated exemplars; pairwise comparison; reliability; scoring methods; teacher assessment

Indexed keywords

EID: 84883437773     PISSN: 00131881     EISSN: 14695847     Source Type: Journal    
DOI: 10.1080/00131881.2013.825159     Document Type: Article

References (38)

1. Andrich, D. 1978. Relationships between the Thurstone and Rasch Approaches to Item Scaling. Applied Psychological Measurement, 2 (3): 449 - 460.
2. Andrich, D. 1988. Rasch Models for Measurement, Beverly Hills, CA: Sage Publications.
3. Andrich, D. 2006. A Report to the Curriculum Council Regarding Assessment for Tertiary Selection. Perth: Curriculum Council of Western Australia. www.curriculum.wa.edu.au/internet/_Documents/Publications/Andrich+Report.pdf (accessed 6 March 2012).
4. Assessment Reform Group. 2002. Assessment for Learning: 10 Principles. Research-based principles to guide classroom practice. http://www.aaia.org.uk/content/uploads/2010/06/Assessment-for-Learning-10-principles.pdf (accessed 24 February 2012).
5. Bond, T. G. and Fox, C. M. 2001. Applying the Rasch Model: Fundamental Measurement in the Human Sciences, Mahwah, NJ: Lawrence Erlbaum.
6. Bradley, R. A. and Terry, M. E. 1952. Rank Analysis of Incomplete Block Designs, I. the Method of Paired Comparisons. Biometrika, 39: 324 - 345.
7. Bramley, T., Bell, J. F. and Pollitt, A. 1998. Assessing Changes in Standards over Time Using Thurstone's Paired Comparisons. Education Research and Perspectives, 2: 1 - 23.
8. Bresciani, M.J., Oakleaf, M., Kolkhorst, F., Nebeker, C., Barlow, J., Duncan, K. and Hickmott, J. 2009. Examining design and inter-rater reliability of a rubric measuring research quality across multiple disciplines. Practical Assessment, Research and Evaluation., 14 (12)
9. Clarke, S. and Gipps, C. 2000. Assessment in England 1996-1998. Evaluation and Research in Education, 14 (1): 38 - 52.
10. Delandshere, G. and Petrosky, A. R. 1998. Assessment of Complex Performances: Limitations of Key Measurement Assumptions. Educational Researcher, 27 (14): 14 - 24.
11. Green, S. and Oates, T. 2009. Considering Alternatives to National Assessment Arrangements in England: Possibilities and Opportunities. Education Researcher, 51 (2): 229 - 245.
12. Gyagenda, I. S. and Engelhard, G. 2009. Using Classical and Modern Measurement Theories to Explore Rater, Domain, and Gender Influences on Student Writing Ability. Journal of Applied Measurement, 10 (3): 225 - 246.
13. Harlen, W. 2005. Trusting teachers' Judgement: Research Evidence of the Reliability and Validity of teachers' Assessment Used for Summative Purposes. Research Papers in Education, 20 (3): 245 - 270.
14. Heldsinger, S. and Humphry, S. M. 2010. Using the Method of Pairwise Comparison to Obtain Reliable Teacher Assessments. The Australian Educational Researcher, 37 (2): 1 - 19.
15. Herman, J., E. Osmundson, Y. Dai, C. Ringstaff, and M. Timms. 2011 Relationships between Teacher Knowledge, Assessment Practice, and Learning - Chicken, Egg, or Omelet? CRESST Report 809. http://www.cse.ucla.edu/products/reports/R809.pdf.
16. Jonsson, A. and Svingby, G. 2007. The Use of Scoring Rubrics: Reliability, Validity and Educational Consequences. Educational Research Review, 2: 130 - 144.
17. Klenowski, V. 2011. Assessment for Learning in the Accountability Era: Queensland, Australia. Studies in Educational Evaluation, 37 (1): 78 - 83.
18. Klenowski, V. and Adie, L. 2009. Moderation as Judgement Practice: Reconciling System Level Accountability and Local Level Practice. Curriculum Perspectives, 29 (1): 10 - 28.
19. Louden, B., E. Chapman, S. Clarke, M. Cullity, and H. House. 2006. Evaluation of the Curriculum Improvement Program Phase 2. Report for the Department of Education and Training prepared in the Graduate School of Education, University of Western Australia. http://www.det.wa.edu.au/education/accountability/docs/curriculumreport.pdf (accessed 10 January 2012).
20. Luce, R. D. 1959. Individual Choice Behaviours: A Theoretical Analysis, New York: J. Wiley.
21. McGaw, B. 2008. The Role of the OECD in International Comparative Studies of Achievement. Assessment in Education: Principles Policy and Practice, 15: 223 - 243.
22. Moskal, B. M. and J. A. Leydens (2000). "Scoring Rubric Development: Validity and Reliability". Practical Assessment, Research & Evaluation 7 (10).
23. Newton, P. 2009. The Reliability of Results from National Curriculum Testing in England. Educational Research, 51 (2): 181 - 212.
24. Reddy, Y. M. and Andrade, H. 2010. A Review of Rubric Use in Higher Education. Assessment and Evaluation in Higher Education, 35 (4): 435 - 448.
25. Rezaei, A. R. and Lovorn, M. 2010. Reliability and Validity of Rubrics for Assessment through Writing. Assessing writing, 15 (1): 18 - 39.
26. Sloane, F. C. and Kelly, A. E. 2003. Issues in High-Stakes Testing Programs. Theory into Practice, 42 (1): 12 - 17.
27. Stobart, G. 2009. Determining Validity in National Curriculum Assessments. Educational Research, 51 (2): 161 - 179.
28. Thurstone, L. L. 1927. A Law of Comparative Judgement. Psychological Review, 34: 278 - 286.
29. Thurstone, L. L. 1928. Attitudes Can Be Measured. American Journal of Sociology, 33: 529 - 554.
30. Tierney, R. and Simon, M. 2004. What's Still Wrong with Rubrics: Focusing on the Consistency of Performance Criteria Across Scale Levels. Practical Assessment, Research and Evaluation, 9 (2): 2004
31. Wald, H. S., Borkan, J. M., Scott Taylor, J., Anthony, D. and Shmuel, P. R. 2012. Fostering and Evaluating Reflective Capacity in Medical Education: Developing the REFLECT Rubric for Assessing Reflective Writing. Academic Medicine, 87 (1): 41 - 50.
32. Whetton, C. 2009. A Brief History of a Testing Time: National Curriculum Assessment in England 1989-2008. Educational Research, 51 (2): 131 - 135.
33. Wiliam, D. 2010. Standardized Testing and School Accountability. Educational Psychologist, 45 (2): 107 - 122.
34. Wilkinson, A., Barnsley, G., Hanna, P. and Swan, M. 1980. Assessing Language Development, Oxford: Oxford University Press.
35. Wilmut, J. 2005. Experiences of Summative Teacher Assessment in the UK: A Review Conducted for the Qualifications and Curriculum Authority. Qualifications and Curriculum Authority. http://www.ofqual.gov.uk (accessed 6 March 2012).
36. Wilson, M. 2006. Rethinking Rubrics in Writing Assessment, Portsmouth: Heinemann.
37. Wyatt-Smith, C., Klenowski, V. and Gunn, S. 2010. The Centrality of teachers' Judgement Practice in Assessment: a Study of Standards in Moderation. Assessment in Education: Principles, Policy & Practice, 17 (1): 59 - 75.
38. Wu, M. 2010. Measurement, Sampling, and Equating Errors in Large-Scale Assessments. Educational Measurement: Issues and Practice, 29: 15 - 27.