Common Structural Design Features of Rubrics May Represent a Threat to Validity

Educational Researcher

Volumn 43, Issue 5, 2014, Pages 253-263

  Humphry, Stephen Mark ^a   Heldsinger, Sandra Allison ^a  

^a UNIVERSITY OF WESTERN AUSTRALIA   (Australia)

Author keywords

assessment; item response theory; mixed methods; performance assessment; teacher assessment; test theory development; validity reliability

Indexed keywords

EID: 84904313682     PISSN: 0013189X     EISSN: 1935102X     Source Type: Journal    
DOI: 10.3102/0013189X14542154     Document Type: Article

References (43)

1. Andrade H.Using rubrics to promote thinking and learning.Educational Leadership. 2000;57 (5): 13-18.
2. Andrade H. G.Teaching with rubrics: The good, the bad, and the ugly.College Teaching. 2005;53 (1): 27-30.
3. Andrich D.A rating formulation for ordered response categories.Psychometrika. 1978;43:357-374.
4. Andrich D.A framework relating outcomes based education and the taxonomy of educational objectives.Studies in Educational Evaluation. 2002;28:35-59.
5. AndrichD. (2006). A report to the Curriculum Council of Western Australia regarding assessment for tertiary selection. Curriculum Council, Western Australia, Australia. Retrieved June19, 2012, from http://platowa.com/documents/Andrich/report.pdf.
6. Andrich D.,Humphry S.,Marais I.Quantifying local, response dependence between two polytomous items using the Rasch model.Applied Psychological Measurement. 2012;36 (4): 309-324.
7. Australian Curriculum, Assessment and Reporting Authority. (2010). NAPLAN 2010 writing narrative marking guide. Retrieved March5, 2014, from http://www.nap.edu.au/verve/_resources/2010_Marking_Guide.pdf.
8. Bechger T. M.,Maris G.,Hsiao Y. P.Detecting halo effects in performance-based examinations.Applied Psychological Measurement. 2010;34:607-619.
9. Borsboom D.,Mellenbergh G. J.,van Heerden J.The concept of validity.Psychological Review. 2004;111 (4): 1061-1071.
10. BrescianiM. J.OakleafM.KolkhorstF.NebekerC.BarlowJ.DuncanK.HickmottJ. (2009). Examining design and inter-rater reliability of a rubric measuring research quality across multiple disciplines. Practical Assessment, Research and Evaluation, 14(12). Available at http://pareonline.net/.
11. Outcomes and standards framework: Overview, student outcomes statements. Osborne Park, Australia: Curriculum Council of Western Australia; 1998:.
12. Darling-Hammond L.Performance-based assessment and educational equity.Harvard Educational Review. 1994;64 (1): 5-30.
13. Delandshere G.,Petrosky A. R.Assessment of complex performances: Limitations of key measurement assumptions.Educational Researcher. 1998;27 (2): 14-24.
14. Ercikan K.,Roth W.- M.What good is polarizing research into qualitative and quantitative?.Educational Researcher. 2006;35 (5): 14-23.
15. Fisicaro S. A.,Lance C. E.Implications of three causal models for the measurement of halo error.Applied Psychological Measurement. 1990;14:419-429.
16. Fisicaro S. A.,Vance R. J.Comments on the measurement of halo.Educational and Psychological Measurement. 1994;2:366-371.
17. Heldsinger S.,Humphry S. M.Using the method of pairwise comparison to obtain reliable teacher assessments.The Australian Educational Researcher. 2010;37 (2): 1-20.
18. Jonsson A.,Svingby G.The use of scoring rubrics: Reliability, validity and educational consequences.Educational Research Review. 2007;2:130-144.
19. King L. M.,Hunter J. E.,Schmidt F. L.Halo in a multidimensional forced-choice performance evaluation scale.Journal of Applied Psychology. 1980;65:507-516.
20. Kohn A.The trouble with rubrics.English Journal. 2006;95 (4): 12-15.
21. Marais I.,Andrich D.Formalising dimension and response violations of local independence in the unidimensional Rasch model.Journal of Applied Measurement. 2008;9:200-215.
22. Marais I.,Andrich D.Diagnosing a common rater halo effect using the polytomous Rasch model.Journal of Applied Measurement. 2011;12 (3): 194-211.
23. Masters G. N.A Rasch model for partial credit scoring.Psychometrika. 1982;47:149-174.
24. Meier S. L.,Rich B. S.,Cady J.Teachers' use of rubrics to score non-traditional tasks: Factors related to discrepancies in scoring.Assessment in Education: Principles, Policy and Practice. 2006;13 (1): 69-95.
25. Messick S.Meaning and values in test validation: The science and ethics of assessment.Educational Researcher. 1989;18 (2): 5-11.
26. Messick S.The interplay of evidence and consequences in validation of performance assessment.Educational Researcher. 1994;23 (2): 13-23.
27. Moskal B. M.,Leydens J. A.Scoring rubric development: Validity and reliability.Practical Assessment, Research & Evaluation. 2000;7 (10):.
28. Myford C. M.,Wolfe E. W.Introduction to Rasch measurement. Smith E. V.Smith R. M., ed. Maple Grove, MN: JAM Press; 2004:460-515.
29. Papay J. P.Refocusing the debate: Assessing the purposes and tools of teacher evaluation.Harvard Educational Review. 2012;82 (1): 123-143.
30. Perlman C. C.Performance assessment: Designing appropriate performance tasks and scoring rubrics.Measuring Up: Assessment Issues for Teachers, Counselors, and Administrators. 2003 2012;:.
31. Popham W. J.What's wrong-and what's right-with rubrics.Educational Leadership. 1997;55 (2): 72-75.
32. Rasch G.On general laws and the meaning of measurement in psychology. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability; 1961Berkeley: University of California Press; 1961. 321.
33. Reddy Y. M.,Andrade H.A review of rubric use in higher education.Assessment and Evaluation in Higher Education. 2010;35 (4): 435-448.
34. Rezaei A. R.,Lovorn M.Reliability and validity of rubrics for assessment through writing.Assessing Writing. 2010;15 (1): 18-39.
35. Sadler D. R.Indeterminacy in the use of preset criteria for assessment and grading in higher education.Assessment and Evaluation in Higher Education. 2009;34:159-179.
36. Silvestri L.,Oescher J.Using rubrics to increase the reliability of assessment in health classes.International Electronic Journal of Health Education. 2006;9:25-30.
37. Thorndike E. L.A constant error in psychological ratings.Journal of Applied Psychology. 1920;4:25-29.
38. Tierney R.,Simon M.What's still wrong with rubrics: Focusing on the consistency of performance criteria across scale levels.Practical Assessment, Research and Evaluation. 2004 2004;9 (2):.
39. Wald H. S.,Borkan J. M.,Scott Taylor J.,Anthony D.,Shmuel P. R.Fostering and evaluating reflective capacity in medical education: Developing the REFLECT rubric for assessing reflective writing.Academic Medicine. 2012;87 (1): 41-50.
40. Wiliam D.Embedded formative assessment. Bloomington. IN: Solution Tree; 2011:.
41. Wilson M.Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum Associates; 2005:.
42. Wilson M.Rethinking rubrics in writing assessment. Portsmouth, NH: Heinemann; 2006:.
43. Wright B. D.,Masters G. N.Rating scale analysis. Chicago: MESA Press; 1982:.