Modeling Holistic Marks With Analytic Rubrics

Frontiers in Education

Volumn 4, Issue , 2019, Pages

  Tomas, Carmen ^a   Whitt, Emma ^a   Lavelle Hill, Rosa ^a   Severn, Katie ^a  

^a UNIVERSITY OF NOTTINGHAM   (United Kingdom)

Author keywords

analytic; criteria; holistic; judgement; marks; rubrics; validity; weightings

Indexed keywords

EID: 85078282260     PISSN: None     EISSN: 2504284X     Source Type: Journal    
DOI: 10.3389/feduc.2019.00089     Document Type: Article

References (59)

1. Allison P., (2012). When Can You Safely Ignore Multicollinearity? Statistical Horizons. Available online at: https://statisticalhorizons.com/multicollinearity.
2. Andrade H., (2005). Teaching with rubrics: the good, the bad, and the ugly. Coll. Teach. 53, 27–31. 10.3200/CTCH.53.1.27-31
3. Andrade H., Du Y., (2005). Student perspectives on rubric-referenced assessment. Pract. Assess. Res. Eval. 10, 1–11. Available online at: http://PAREonline.net/getvn.asp?v=10&n=3
4. Biggs J., Tang C., (2011). Teaching For Quality Learning At University, 4th Edn. Buckingham: The Society for Research into Higher Education and Open University Press.
5. Biggs J. B., Collis K. F., (1982). Evaluating The Quality Of Learning: The SOLO Taxonomy (Structure of the observed learning outcome). London: Academic Press.
6. Björklund L., Stolpe K., Lundström M., Åström M., (2015). To measure the unmeasurable: using the repertory grid technique to elicit tacit criteria used by examiners, in 5th International Assessment in Higher Education Conference (Birmingham).
7. Bloxham S., Boyd P., Orr S., (2011) Mark my words: the role of assessment criteria in UK higher education grading practices. Stud. Higher Educ. 36, 655–670. 10.1080/03075071003777716
8. Bloxham S., den-Outer B., Hudson J., Price M., (2016a). Let's stop the pretence of consistent marking: exploring the multiple limitations of assessment criteria. Assess. Eval. High. Educ. 41, 466–481. 10.1080/02602938.2015.1024607
9. Bloxham S., Hughes C., Adie L., (2016b). What's the point of moderation? A discussion of the purposes achieved through contemporary moderation practices. Assess. Eval. High. Educ. 41, 638–653. 10.1080/02602938.2015.1039932
10. Bloxham S., Price M., (2013). External examining: fit for purpose?. Stud. High. Educ. 40, 195–211. 10.1080/03075079.2013.823931
11. Boston C., (2002). Understanding Scoring Rubrics: A Guide for Teachers. College Park, MD: ERIC Clearinghouse on Assessment and Evaluation.
12. Boud D., (2018). Assessment could demonstrate learning gains, but what is required to do so? High. Educ. Pedagogies 3, 4–6. 10.1080/23752696.2017.1413671
13. Boud D., Ajjawi R., Dawson P., Tai J., (2018). Developing Evaluative Judgement in Higher Education. Abingdon: Routledge. 10.4324/9781315109251
14. Breiman L., Friedman J. H., Olshen R. A., Stone C. J., (1984). Classification and Regression Trees. Boca Raton, FL: Chapman and Hall/CRC. Available online at: https://www.crcpress.com/Classification-and-Regression-Trees/Breiman-Friedman-Stone-Olshen/p/book/9780412048418#googlePreviewContainer
15. Brookhart S. M., (2018). Appropriate criteria: key to effective rubrics. Front. Educ. 3:22. 10.3389/feduc.2018.00022
16. Brookhart S. M., Chen F., (2015). The quality and effectiveness of descriptive rubrics. Educ. Rev. 67, 343–368. 10.1080/00131911.2014.929565
17. Brookhart S. M., Nitko A. J., (2019). Educational Assessment Of Students, 8th Edn. Boston, MA: Pearson Education
18. Brown G., Bull J., Pendlebury M., (1997). Assessing Student Learning in Higher Education. London: Routledge.
19. Brown George., (2001). Assessment: A Guide For Lecturers. York: LTSN Generic Centre: Assessment Series 3.
20. Dawson P., (2015). Assessment rubrics: towards clearer and more replicable design, research and practice. Assess. Eval. High. Educ. 42, 347–360. 10.1080/02602938.2015.1111294
21. Dochy F., (2009). The edumetric quality of new modes of assessment: some issues and prospects, in Assessment, Learning and Judgement in Higher Education, ed Joughin G., (Dordrecht: Springer Netherlands), 1–30. 10.1007/978-1-4020-8905-3_6
22. Elton L., (1998). Are UK degree standards going up, down or sideways?. Stud. High. Educ. 23, 35–42. 10.1080/03075079812331380472
23. Elton L., Johnston B., (2002). Assessment in Universities: a critical review of research. Learning and Teaching Support Network. Available online at: https://eprints.soton.ac.uk/59244/1/59244.pdf.
24. Friedman J., (2001). Greedy function approximation: a gradient boosting machine. Ann. Statist. 29, 1189–1232. 10.1214/aos/1013203451
25. Grainger P., Purnell K., Zipf R., (2008) Judging quality through substantive conversations between markers. Assess. Eval. High. Educ. 33, 133–142. 10.1080/02602930601125681
26. Grömping U., (2009). Variable importance assessment in regression: linear regression versus random forest. Am. Statist. 63, 308–319. 10.1198/tast.2009.08199
27. Harsch C., Martin G., (2013). Comparing holistic and analytic scoring methods: issues of validity and reliability. Assess. Educ. 20, 281–307. 10.1080/0969594X.2012.742422
28. Hay P., Macdonald D., (2008). (Mis)appropriations of criteria and standards-referenced assessment in a performance-based subject. Assess. Educ. 15, 153–168. 10.1080/09695940802164184
29. Hunter D. M., Jones R. M., Randhawa B. S., (1996). The use of holistic versus analytic scoring for large-scale assessment of writing. Can. J. Prog. Eval. 11, 61–85.
30. Huot B., (1990). Reliability, validity and holistic scoring: what we know and what we need to know. Coll. Composit. Commun. 41, 201–213. 10.2307/358160
31. Jones I., Alcock L., (2014). Peer assessment without assessment criteria. Stud. High. Educ. 39, 1774–1787. 10.1080/03075079.2013.821974
32. Jönsson A., Prins F., (2019). Editorial: transparency in assessment—Exploring the influence of explicit assessment criteria. Front. Education. 3:119. 10.3389/feduc.2018.00119
33. Jönsson A., Svingby G., (2007). The use of scoring rubrics: reliability, validity and educational consequences. Educ. Res. Rev. 22, 130–144. 10.1016/j.edurev.2007.05.002
34. Kuo S., (2007). Which rubric is more suitable for NSS liberal studies? Analytic or holistic? Educ. Res. J. 22, 179–199. Available online at: http://www.fed.cuhk.edu.hk/ceric/erj/200700220002/0179.htm
35. Macdonald R., Joughin G., (2009). Changing assessment in Higher Education: a model in support of institution-wide improvement, in Assessment, Learning and Judgment in Higher Education, ed Joughin G. R., (Dordrecht: Springer), 193–213. 10.1007/978-1-4020-8905-3_11
36. Messick S., (1994). The interplay of evidence and consequences in the validation of performance assessments. Educ. Res. 23, 13–23. 10.3102/0013189X023002013
37. Messick S., (1995). Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. Am. Psychol. 50, 741–749. 10.1037/0003-066X.50.9.741
38. Messick S., (1996). Validity in performance assessments, in Technical Issues in Large-scale Performance Assessment, ed Phillips G. W., (Washington DC: National Centre for Educational Statistics, 1–18.
39. Mini Tab (2013). Enough Is Enough! Handling Multicollinearity in Regression Analysis. The Minitab Blog. Available online at: https://blog.minitab.com/blog/understanding-statistics/handlingmulticollinearity-in-regression-analysis.
40. Panadero E., Broadbent J., (2018). Developing evaluative judgment: self-regulated learning perspective, in Developing Evaluative Judgement in Higher Education. Assessment for Knowing and Producing Quality Work, eds Boud D., Ajjwi R., Dawson P., Tai J., (London: Routledge), 81–89. 10.4324/9781315109251-9
41. Panadero E., Jönsson A., (2013). The use of scoring rubrics for formative assessment purposes revisited: a review. Educ. Res. Rev. 9, 129–144. 10.1016/j.edurev.2013.01.002
42. Panadero E., Romero M., (2014). To rubric or not to rubric? The effects of self-assessment on self-regulation, performance and self-efficacy. Assess. Educ. 21, 133–148. 10.1080/0969594X.2013.877872
43. Perlman C., (2002). An introduction to performance assessment scoring rubrics, in Understanding Scoring Rubrics, ed Boston C., (College Park, MD: ERIC Clearinghouse on Assessment and Evaluation), 5–13.
44. Reddy Y. M., Andrade H., (2010). A review of rubric use in higher education. Assess. Eval. High. Educ. 35, 435–488. 10.1080/02602930902862859
45. Sadler D. R., (1987). Specifying and promulgating achievement standards. Oxford Educ. Rev. 13:2. 10.1080/0305498870130207
46. Sadler D. R., (2009a). Indeterminacy in the use of preset criteria for assessment and grading in higher education. Assess. Eval. High. Educ. 34, 159–179. 10.1080/02602930801956059
47. Sadler D. R., (2009b). Transforming holistic assessment and grading into a vehicle for complex learning, in Assessment, Learning and Judgement in Higher Education, ed Joughin G., (Dordrecht: Springer, 45–63.
48. Sadler R. D., (2014). The futility of attempting to codify academic achievement standards. High. Educ. 67, 273–288. 10.1007/s10734-013-9649-1
49. Shaw S., Crisp V., (2012). An Approach to Validation: Developing and Applying an Approach for the Validation of General Qualifications. Research Matters: A Cambridge Assessment Publication, Special Issue 3. Available online at: https://www.cambridgeassessment.org.uk/Images/110003-research-matters-special-issue-3-an-approach-to-validation.pdf
50. Stevens D., Levi A., (2005). Introduction to Rubrics: An Assessment Tool to Save Grading Time, Convey Effective Feedback, and Promote Student Learning, 2nd Edn. Sterling: Stylus (VA). Available online at: https://www.amazon.es/Introduction-Rubrics-Assessment-Effective-Feedback/dp/1579225888
51. Suskie L., (2009). Using assessment results to inform teaching practice and promote lasting learning, in Assessment, Learning and Judgement in Higher Education, ed Joughin G., (Dordrecht: Springer Netherlands), 133–152. 10.1007/978-1-4020-8905-3_8
52. Suskie L., (2014). Five Dimensions of Quality: A Common Sense Guide to Accreditation and Accountability. San Francisco, CA: Jossey-Bass.
53. Suskie L., (2017). Rubric development, in Handbook On Measurement, Assessment and Evaluation in Higher Education, 2nd Edn., eds Secolsky C., Denison D. B., (London: Routledge), 545–559. 10.4324/9781315709307-43
54. Suto I., Nadas R., (2008). What determines GCSE marking accuracy? An exploration of expertise among maths and physics markers. Res. Papers Educ. 23, 477–497. 10.1080/02671520701755499
55. Torrance H., (2007). Assessment as learning? How the use of explicit learning objectives, assessment criteria and feedback in post-secondary education and training can come to dominate learning. Assess. Educ. 14, 281–294. 10.1080/09695940701591867
56. Utaberta N., Hassanpour B., (2012). Aligning assessment with learning outcomes. Soc. Behav. Sci. 60, 228–235. 10.1016/j.sbspro.2012.09.372
57. Woolf H., (2004). Assessment criteria: reflections on current practices. Assess. Eval. High. Educ. 29, 479–493. 10.1080/02602930310001689046
58. Yin R. K., (2002). Case Study Research: Design and Methods. Thousand Oaks, CA: SAGE Publications.
59. Yorke M., (2011). Summative assessment: dealing with the measurement fallacy. Stud. High. Educ. 36, 251–273. 10.1080/03075070903545082