The use of scoring rubrics: Reliability, validity and educational consequences

Educational Research Review

Volumn 2, Issue 2, 2007, Pages 130-144

  Jonsson, Anders ^a   Svingby, Gunilla ^a  

^a MALMÖ UNIVERSITY   (Sweden)

Author keywords

Alternative assessment; Performance assessment; Reliability; Scoring rubrics; Validity

Indexed keywords

EID: 36049027816     PISSN: 1747938X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.edurev.2007.05.002     Document Type: Review

References (76)

1. Arter J., and McTighe J. Scoring rubrics in the classroom (2001), Corwin Press Inc., Thousand Oaks
2. Baartman, L. K. J., Bastiaens, T. J., Kirschner, P. A., & Van der Vleuten, C. P. M. (submitted for publication). Assessment in competence-based education: How can assessment quality be evaluated? Educational Research Review.
3. Baartman L.K.J., Bastiaens T.J., Kirschner P.A., and Van der Vleuten C.P.M. The wheel of competency assessment: Presenting quality criteria for competency assessment programmes. Studies in Educational Evaluation 32 (2006) 153-170
4. Birenbaum M., Breuer K., Cascallar E., Dochy F., Dori Y., Ridgway J., et al. A learning integrated assessment system. Educational Research Review 1 (2006) 61-67
5. Black P. Testing: Friend or foe? (1998), Falmer Press, London
6. Borsboom D., Mellenbergh G.J., and van Heerden J. The concept of validity. Psychological Review 111 (2004) 1061-1071
7. Brennan R. Generalizability of performance assessments. In: Phillips G. (Ed). Technical issues in large-scale performance assessment (1996), National Center for Education Statistics, Washington, DC 19-58
8. Brown G., Bull J., and Pendlebury M. Assessing student learning in higher education (1997), Routledge, London
9. Busching B. Grading inquiry projects. New Directions for Teaching and Learning (1998) 89-96
10. Darling-Hammond L., and Snyder J. Authentic assessment of teaching in context. Teaching and Teacher Education 16 (2000) 523-545
11. Davidson M., Howell K.W., and Hoekema P. Effects of ethnicity and violent content on rubric scores in writing samples. Journal of Educational Research 93 (2000) 367-373
12. Dochy F., Gijbels D., and Segers M. Learning and the emerging new assessment culture. In: Verschaffel L., Dochy F., Boekaerts M., and Vosniadou S. (Eds). Instructional psychology: Past, present and future trends (2006), Elsevier, Oxford, Amsterdam
13. Dochy F., Segers M., and Sluijsmans D. The use of self-, peer and co-assessment in higher education: A review. Studies in Higher Education 24 (1999) 331-350
14. Falchikov N., and Boud D. Student self-assessment in higher education: A meta-analysis. Review of Educational Research 59 (1989) 395-430
15. Falchikov N., and Goldfinch J. Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research 70 (2000) 287-322
16. Frederiksen J.R., and Collins A. A systems approach to educational testing. Educational Researcher 18 (1989) 27-32
17. Gielen S., Dochy F., and Dierick S. Evaluating the consequential validity of new modes of assessment: The influence of assessment on learning, including pre-, post-, and true assessment effects. In: Segers M., Dochy F., and Cascallar E. (Eds). Optimizing new modes of assessment: In search of qualities and standards (2003), Kluwer Academic Publishers, Dordrecht
18. Mabry L. Writing to the rubric: Lingering effects of traditional standardized testing on direct writing assessment. Phi Delta Kappan 80 (1999) 673-679
19. McMillan J.H. Educational research: Fundamentals for the consumer (2004), Pearson Education Inc., Boston
20. Messick S. Validity of performance assessments. In: Phillips G. (Ed). Technical issues in large-scale performance assessment (1996), National Center for Education Statistics, Washington, DC 1-18
21. Morrison G.R., and Ross S.M. Evaluating technology-based processes and products. New Directions for Teaching and Learning 74 (1998) 69-77
22. Moskal B.M., and Leydens J.A. Scoring rubric development: Validity and reliability. Practical Assessment, Research & Evaluation 7 (2000) 71-81
23. Orsmond P., and Merry S. The importance of marking criteria in the use of peer assessment. Assessment & Evaluation in Higher Education 21 (1996) 239-250
24. Perlman, C.C. (2003). Performance assessment: Designing appropriate performance tasks and scoring rubrics. North Carolina, USA.
25. Shavelson R.J., Gao X., and Baxter G. On the content validity of performance assessments: Centrality of domain-specifications. In: Birenbaum M., and Dochy F. (Eds). Alternatives in assessment of achievements, learning processes and prior knowledge (1996), Kluwer Academic Publishers, Boston
26. Stemler S.E. A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation 9 (2004)
27. Topping K. Self and peer assessment in school and university: Reliability, validity and utility. In: Segers M., Dochy F., and Cascallar E. (Eds). Optimizing new modes of assessment: In search of qualities and standards (2003), Kluwer Academic Publishers, Dordrecht
28. Van de Watering G., and van der Rijt J. Teachers' and students' perceptions of assessments: A review and a study into the ability and accuracy of estimating the difficulty levels of assessment items. Educational Research Review 1 (2006) 133-147
29. Wiggins G. Educative assessment (1998), Jossey-Bass, San Francisco
30. Andrade H.G. The role of instructional rubrics and self-assessment in learning to write: A smorgasbord of findings. Paper Presented at the Annual Meeting of the American Educational Research Association. Montreal, Canada (1999)
31. Andrade H.G. Student self-assessment: At the intersection of metacognition and authentic assessment. Paper Presented at the Annual Meeting of the American Educational Research Association. Montreal, Canada (1999)
32. Baker E.L. Learning-based assessments of history understanding. Educational Psychologist 29 (1994) 97-106
33. Baker E.L., Abedi J., Linn R.L., and Niemi D. Dimensionality and generalizability of domain-independent performance assessments. Journal of Educational Research 89 (1995) 197-205
34. Beeth M.E., Cross L., Pearl C., Pirro J., Yagnesak K., and Kennedy J. A continuum for assessing science process knowledge in grades K-6. Electronic Journal of Science Education (2001) 5
35. Bissell A.N., and Lemons P.R. A new method for assessing critical thinking in the classroom. BioScience 56 (2006) 66-72
36. Brown G.T.L., Glasswell K., and Harland D. Accuracy in the scoring of writing: Studies of reliability and validity using a new zealand writing assessment system. Assessing Writing 9 (2004) 105-121
37. Chi E. Comparing holistic and analytic scoring for performance assessment with many-facet rasch model. Journal of Applied Measurement 2 (2001) 379-388
38. Cho K., Schunn C.D., and Wilson R.W. Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives. Journal of Educational Psychology 98 (2006) 891-901
39. Denner P.R., Salzman S.A., and Harris L.B. Teacher work sample assessment: An accountability method that moves beyond teacher testing to the impact of teacher performance on student learning. Paper presented at the annual meeting of the American Association of Colleges for Teacher Education. New York, USA (2002)
40. DeRemer M.L. Writing assessment: Raters' elaboration of the rating task. Assessing Writing 5 (1998) 7-29
41. Duke, B. L. (2003). The influence of using cognitive strategy instruction through writing rubrics on high school students' writing self-efficacy, achievement goal orientation, perceptions of classroom goal structures, self-regulation, and writing achievement. Unpublished doctoral dissertation. USA: University of Oklahoma.
42. Flowers C.P., and Hancock D.R. An interview protocol and scoring rubric for evaluating teacher performance. Assessment in Education: Principles, Policy and Practice 10 (2003) 161-168
43. Gearhart M., Herman J.L., Novak J.R., and Wolf S.A. Toward the instructional utility of large-scale writing assessment: Validation of a new narrative rubric. Assessing Writing 2 (1995) 207-242
44. Green R., and Bowser M. Observations from the field: Sharing a literature review rubric. Journal of Library Administration 45 (2006) 185-202
45. Hafner J.C., and Hafner P.M. Quantitative analysis of the rubric as an assessment tool: An empirical study of student peer-group rating. International journal of science education 25 (2003) 1509-1528
46. Johnson R.L., Penny J., and Gordon B. The relation between score resolution methods and interrater reliability: An empirical study of an analytic scoring rubric. Applied Measurement in Education 13 (2000) 121-138
47. Johnson R.L., Penny J., and Gordon B. Score resolution and the interrater reliabilityof holistic scores in rating essays. Written Communication 18 (2001) 229-249
48. Lindblom-Ylänne S., Pihlajamäki H., and Kotkas T. Self-, peer- and teacher-assessment of student essays. Active Learning in Higher Education 7 (2006) 51-62
49. Luft J. Rubrics: Design and use in science teacher education. Paper Presented at the Annual Meeting of the Association for the Education of Teachers in Science. Minneapolis, USA (1998)
50. Lunsford, B. E. (2002). Inquiry and inscription as keys to authentic science instruction and assessment for preservice secondary science teachers. Unpublished doctoral dissertation. USA: University of Tennessee.
51. Marzano R.J. A comparison of selected methods of scoring classroom assessments. Applied Measurement in Education 15 (2002) 249-267
52. McMartin F., McKenna A., and Youssefi K. Scenario assignments as assessment tools for undergraduate engineering education. IEEE Transactions on Education 43 (2000) 111-120
53. Morrell P.D., and Ackley B.C. Practicing what we teach: Assessing pre-service teachers' performance using scoring guides. Paper presented at the annual meeting of the American Educational Research Association. Montreal, Canada (1999)
54. Mullen, Y. K. (2003). Student improvement in middle school science. Unpublished master thesis. USA: University of Wisconsin.
55. Myford C.M., Johnson E., Wilkins R., Persky H., and Michaels M. Constructing scoring rubrics: Using "facets" to study design features of descriptive rating scales. Paper presented at the annual meeting of the American Educational Research Association. New York, USA (1996)
56. Osana H.P., and Seymour J.R. Critical thinking in preservice teachers: A rubric for evaluating argumentation and statistical reasoning. Educational Research and Evaluation 10 (2004) 473-498
57. Paratore J.R. Assessing literacy: Establishing common standards in portfolio assessment. Topics in Language Disorders 16 (1995) 67-83
58. Penny J., Johnson R.L., and Gordon B. The effect of rating augmentation on inter-rater reliability: An empirical study of a holistic rubric. Assessing Writing 7 (2000) 143-164
59. Penny J., Johnson R.L., and Gordon B. Using rating augmentation to expand the scale of an analytic rubric. Journal of Experimental Education 68 (2000) 269-287
60. Piscitello, M. E. (2001). Using rubrics for assessment and evaluation in art. Unpublished master thesis. USA: Saint Xavier University.
61. Popp S.E.O., Ryan J.M., Thompson M.S., and Behrens J.T. Operationalizing the rubric: The effect of benchmark selection on the assessed quality of writing. Paper Presented at Annual Meeting of the American Educational Research Association. Chicago, USA (2003)
62. Ramos K.D., Schafer S., and Tracz S.M. Validation of the fresno test of competence in evidence based medicine. British Medical Journal 326 (2003) 319-321
63. Roblyer M.D., and Wiencke W.R. Design and use of a rubric to assess and encourage interactive qualities in distance courses. American Journal of Distance Education 17 (2003) 77-99
64. Sadler P.M., and Good E. The impact of self- and peer-grading on student learning. Educational Assessment 11 (2006) 1-31
65. Schafer W.D., Swanson G., Bené N., and Newberry G. Effects of teacher knowledge of rubrics on student achievement in four content areas. Applied Measurement in Education 14 (2001) 151-170
66. Schamber J.F., and Mahoney S.L. Assessing and improving the quality of group critical thinking exhibited in the final projects of collaborative learning groups. Journal of General Education 55 (2006) 103-137
67. Schirmer B.R., Bailey J., and Fitzgerald S.M. Using a writing assessment rubric for writing development of children who are deaf. Exceptional Children 65 (1999) 383-397
68. Shaw J. Demystifying the evaluation process for parents: Rubrics for marking student research projects. Teacher Librarian 32 (2004) 16-19
69. Smith J., and Hanna M.A. Using rubrics for documentation of clinical work supervision. Counselor Education and Supervision 37 (1998) 269-278
70. Stoddart T., Abrams R., Gasper E., and Canaday D. Concept maps as assessment in science inquiry learning-A report of methodology. International Journal of Science Education 22 (2000) 1221-1246
71. Stoering J.M., and Lu L. Combining the national survey of student engagement with student portfolio assessment. Paper Presented at Annual Meeting of the Association for Institutional Research. Toronto, Canada (2002)
72. Stuhlmann J., Daniel C., Dellinger A., Denny R.K., and Powers T. A generalizability study of the effects of training on teachers' abilities to rate children's writing using a rubric. Journal of Reading Psychology 20 (1999) 107-127
73. Toth E.E., Suthers D.D., and Lesgold A.M. Mapping to know: The effects of representational guidance and reflective assessment on scientific inquiry. Science Education 86 (2002) 264-286
74. Waltman, K., Kahn, A., & Koency, G. (1998). Alternative approaches to scoring: The effects of using different scoring methods on the validity of scores from a performance assessment. CSE Technical Report 488. Los Angeles.
75. Weigle S.C. Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing 6 (1999) 145-178
76. Williams L., and Rink J. Teacher competency using observational scoring rubrics. Journal of Teaching in Physical Education 22 (2003) 552-572