Standard setting in student assessment: Is a defensible method yet to come?

Annals of the Academy of Medicine Singapore

Volumn 37, Issue 11, 2008, Pages 957-963

  Barman, Arunodaya ^a  

^a SCHOOL OF MEDICAL SCIENCES   (Malaysia)

Author keywords

Difficulty and discriminating indices; Judges and judgment; Legal issues; Practicability; Reliability; Validity

Indexed keywords

CURRICULUM; EVALUATION; LEARNING; LEGAL ASPECT; MEDICAL EDUCATION; MEDICAL STUDENT; RELIABILITY; REVIEW; STANDARD; VALIDITY;

EID: 57649157712     PISSN: 03044602     EISSN: 29724066     Source Type: Journal    
DOI: None     Document Type: Review

References (106)

1. Lilley PM, Harden RM. Standards and medical education. Med Teach 2003;25:349-51.
2. Senanayake MP, Mettananda DS. Standards medical students set for themselves when preparing for the final MBBS examination. Ann Acad Med Singapore 2005;34:483-5.
3. Kane M. Validating the performance standards associated with passing score. Rev Educ Res 1994;64:425-61.
4. Ricker KL. Setting cut-scores: A critical review of the Angoff and modified Angoff methods. Alberta J Educ Res 2006;52:53-64.
5. Berk RA. A consumer's guide to setting performance standard on criterion-referenced tests. Rev Educ Res 1986;56:137-72.
6. Kane M. Model-based practice analysis and test specifications. Appl Meas Educ 1997;10:5-18.
7. Andrew BJ, Hecht JT. A preliminary investigation of two procedures for setting examination standards. Educ Psychol Meas 1976;36:45-50.
8. Skakun EN, Kling S. Comparability of methods for setting standards. J Educ Meas1980;17:229-35.
9. Ebel RI. The case for minimum competency testing. Phi Delta Kappan 1978;59:546-9.
10. George S, Haque MS, Oyebode F. Standard setting: comparison of two methods. BMC Med Educ 2006;6:46.
11. Norcini JJ. Setting standards on educational tests. Med Educ 2003;37:464-9.
12. Carlson T, Macdonald D, Gorely T, Hanrahan S, Burgess-Limerick R. Implementing criterion-referenced assessment within a multidisciplinary university department. High Educ Res Dev 2000;19:103-16.
13. Truxillo DM, Donahue LM, Sulzer JL. Setting cutoff scores for personnel selection tests: issues, illustrations, and recommendations. Hum Perform 1996;9:275-85.
14. Hoover HD. Some common misconceptions about tests and testing. Educ Meas 2003;22:5-14.
15. Supernaw RB, Mehvar R. Methodology for the assessment of competence and the definition of deficiencies of students in all levels of the curriculum. Am J Pharmaceut Educ 2002;66:1-4.
16. Turnbull JM. What is ... normative versus criterion-referenced assessment. Med Teach 1989;11:145-50.
17. Black PJ. Formative and summative assessment by teachers. Stud Sci Educ 1993; 21:49-97.
18. Dunn L, Parry S, Morgan C. Seeking quality in criterion referenced assessment. Paper presented at the Learning Communities and assessment Cultures Conference organized by the EARLI Special Interest Group on Assessment and Evaluation, University of Northumbria, 28-30 August 2002. Available at: http://www.leeds.ac.uk/educol/documents/00002257.htm. Accessed 15 April 2007.
19. Harden RM. Ten questions to ask when planning a course or curriculum. Med Educ 1986;20:356-65.
20. Glaser R. Instructional technology and the measurement of learning outcomes: Some questions. Am Psychol 1963;18:519-21.
21. Gipps C. What do we mean by equity in relation to assessment? Assess Educ Princ Pol Pract 1995;2:271-81.
22. Schindler N, Corcoran J, DaRosa D. Description and impact of using a standard-setting method for determining pass/fail scores in a surgery clerkship. Am J Surg. 2007;193:252-7.
23. nd ed. Washington DC: Americal Council on Education, 1971:625-70.
24. Glass GV. Standard and criteria. J Educ Meas 1978;15:237-261.
25. Ben-David MF. Standard setting in student assessment. Med Teach 2000;22:120-30.
26. Hambleton RK. On the use of cut-off score with criterion-referenced tests in instructional settings. J Educ Meas 1978;15: 277-90.
27. Jolly B. Setting standard for tomorrow's doctors. Med Educ 1999;33: 792-3.
28. Berk RA. Standard setting: The next generation (Where few psychometricians have gone before!). Appl Meas Educ 1996;9:215-235.
29. Linn RL. Performance standards: Utility for different uses of assessments. Educ Pol Anal Arch 2003;11:1-20.
30. Boursicot KA, Roberts TE, Pell G. Standard setting for clinical competence at graduation from medical school: a comparison of passing scores across five medical schools. Adv Health Sci Educ Theory Pract 2006;11:173-83.
31. Halpin G, Halpin G. An analysis of the reliability and validity of procedures for setting minimum competency standards. Educ Psychol Meas 1987;47:977-83.
32. Popham W. As always provocative. J Educ Meas 1978;15:297-300.
33. Koffler SL. A comparison of approaches for setting proficiency standards. J Educ Meas 1980;17:167-78.
34. Angoff WH. Proposals for theoretical and applied development in measurement. Appl Meas Educ 1988;1:215-22.
35. Walter RA, Kapes JT. Development of a procedure for establishing occupational examination cut scores: A NOCTI example. J Ind Teach Educ 2003;40:25-45.
36. Kramer A, Muijtjens A, Jansen K, Düsman H, Tan L, van der Vleuten C. Comparison of a rational and an empirical standard setting procedure for an Objective Structured Clinical Examinations. Med Educ 2003;37:132-9.
37. Mills CN. A comparison of three methods of establishing cut-off scores on criterion-referenced tests. J Educ Meas 1983;20:283-92.
38. Livingston SA, Zieky MJ. A comparative study of standard-setting methods. Appl Meas Educ 1989;2:121-41.
39. Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teach Learn Med 2006;18:50-7.
40. Humphrey-Murto S, MacFadyen JC. Standard setting: a comparison of case-author and modified borderline-group methods in a small-scale OSCE. Acad Med 2002;77:729-32.
41. Jaeger RM. Certification of student competence. In: Linn RL, editor. Educational Measurement. 3rd ed. New York: ACE Macmillan, 1989:485-514.
42. Cizek GJ. Adapting testing technology to serve accountability aims: The case of vertically moderated standard setting. Appl Meas Educ 2005;18:1-9.
43. Reckase MD. Rejoinder: Evaluating standard setting methods using error models proposed by Schulz. Educ Meas 2006;25:14-17.
44. Cusimano MD. Standard setting in medical education. Acad Med 1996;71(10 suppl):S112-20.
45. Cizek GJ. Reconsidering standards and criteria. J Educ Meas 1993;30:93-106.
46. Cizek GJ. Standard setting guidelines. Educ Meas 1996;15:12-21.
47. Plake BS. Setting performance standards for professional licensure and certification. Appl Meas Educ 1998;11:650-80.
48. Cross LH, Impara JC, frary RB, Jaeger RM. A comparison of three methods for establishing minimum standards on the national teacher examinations. J Educ Meas 1984;21:113-29.
49. Chinn RN, Hertz NR. Alternative approaches to standard setting for licensing and certification examinations. Appl Meas Educ 2002;15:1-14.
50. Jaeger RM. Establishing standards for teacher certification tests. Educ Meas 1990;9:15-20.
51. Hobma SO, Ram PM, Muijtjens AM, Grol RP, van der Vleuten CP. Setting a standard for performance assessment of doctor-patient communication in general practice. Med Educ 2004;38:1244-52.
52. Boursicot KA, Roberts TE Pell G. Using borderline methods to compare passing standards for OSCEs at graduation across three medical schools. Med Educ 2007;41:1024-31.
53. Ahn DS, Ahn S. Reconsidering the cut score of Korean National Medical Licensing examination. J Educ Eval Health Prof 2007;4:1.
54. Stern DT, Friedman Ben-David M, Norcini J, Wojtczak A, Schwarz MR. Setting school-level outcome standards. Med Educ 2006;40:166-72.
55. Searle J. Defining competency - the role of standard setting. Med Educ 2000;34:363-6.
56. van der Linden WJ. A latent trait method for determining intrajudge inconsistency in the Angoff and Nedelsky techniques of standard setting. J Educ Meas 1982;19: 295-308.
57. Hambleton RK, Bernnan RL, Brown W, Dodd B, Forsyth RA, Mehrens WA, et al. A response to "Setting reasonable and useful standards" in the National Academy of Sciences' "Grading the Nation's Report Card". Educ Meas 2000;19:5-14.
58. Linn R. Demands, cautions and suggestions for setting standards. J Educ Meas 1978;15:301-8.
59. Norcini J, Guille R. Combining tests and setting standards. In: Norman GR, van der Vleuten CPM, Newble DI, editors. International Handbook of Research in Medical Education. Boston: Kluwer Academic Publishers, 2002:811-34.
60. Busch JC, Jaeger RM. Influence of type of judge, normative information, and discussion on standard recommended for the national teacher examinations. J Educ Meas 1990;27:145-63.
61. Stern DT, Ben-David MF, De Champlain A, Hodges B, Wojtczak A, Schwarz MR. Ensuring global standards for medical graduates: a pilot study of international standard-setting. Med Teach 2005;27:207-13.
62. Morrison H, McNally H, Wylie C, McFaul P, Thompson W. The passing score in the objective structured clinical examination. Med Educ 1996;30:345-8.
63. Fitzpatrick AR. Social influences in standard setting: The effects of social interaction on group judgments. Rev Educ Res 1989;59:315-28.
64. McGinty D. Illuminating the "Black Box" of standard setting: An exploratory qualitative study. Appl Meas Educ 2005;18:269-87.
65. Boursicot K, Roberts T. Setting standards in a professional higher education course: defining the concept of the minimally competent student in performance-based assessment at the level of graduation from medical school. High Educ Q 2006;60:74-90.
66. Boulet JR, De Champlain AF, McKinley DW. Setting defensible performance standards on OSCEs and standardized patient examinations. Med Teach 2003;25:245-9.
67. Wayne DB, Fudala MJ, Butter J, Siddall VJ, Feinglass J, Wade LD, et al. Comparison of two standard-setting methods for advanced cardiac life support training. Acad Med 2005;80(10 Suppl):S63-6.
68. Goodwin LD. Relations between observed item difficulty levels and Angoff minimum passing levels for a group of borderline examinees. Appl Meas Educ 1999;12:13-28.
69. Glass GV. 2003. Standards and criteria redux. Available at: http://glass.ed.asu.edu/gene/papers/standards/. Accessed 5 June 2007.
70. Verhoeven BH, van der Steeg AF, Scherpbier AJ, Muijtjens AM, Verwijnen GM, van der Vleuten CP. Reliability and credibility of an Angoff standard setting procedure in progress testing using recent graduates as judges. Med Educ 1999;33:832-7.
71. Shepard L, Glaser R., Linn R, Bohrnstedt G. Setting Performance Standards for Student Achievement. Stanford, CA: National Academy of Education, 1993.
72. Giraud G, Impara JC. Making the cut: The cut score setting process in a public school district. Appl Meas Educ 2005;18:289-312.
73. rd ed. Philadelphia, PA: National Board of Medical Examiners, 3750 Market Street, PA 19104, 2002:8-20. Available at: http://www.nbme.org/PDF/ItemWriting_2003/2003IWGwhole.pdf. Accessed 14 August 2007.
74. Kane MT. The validity of licensure examinations. Am Psychol 1982;37:911-8.
75. Raymond MR. A practical guide to practice analysis for credentialing examinations. Educ Meas 2002;21:25-36.
76. Levin HM. Educational performance standards: Image or substance? J Educ Meas 1978;15: 309-19.
77. Sireci SG, Green PC. Legal and psychometric criteria for evaluating teacher certification tests. Educ Meas 2000;19:22-34.
78. Mauer TJ, Alexander RA. Method of improving employment test critical scores derived by judging test content: A review and critique. Person Psychol 1992;45:727-62.
79. Crocker L. Assessing content representativeness of performance assessment exercises. Appl Meas Educ 1997;10: 83-95.
80. Wang N, Schnipke D, Witt EA. Use of knowledge, skill, and ability statements in developing licensure and certification examinations. Educ Meas 2005;24:15-22.
81. Chesser AM, Laing MR, Miedzybrodzka ZH, Brittenden J, Heys SD. Factor analysis can be a useful standard setting tool in a high stakes OSCE assessment. Med Educ 2004;38:825-31.
82. Wolf A, Silver R. Work Based Learning: Trainee Assessment by Supervisors. Research and Development Series Number 33 Sheffield: MSC, 1986.
83. Shepard LA. Why we need better assessments. Educ Leader 1989;46:4-9.
84. Berk RA. Something old, something new, something borrowed, a lot to do! Appl Meas Educ 1995;8:99-109.
85. Norcini JJ, Shea JA. The credibility and comparability of standards. Appl Meas Educ 1997;10:39-59.
86. Pell G, Roberts TE. Setting standard for students assessment. Int J Res Meth Educ 2006;29:91-103.
87. Bramley T. Accessibility, easiness and standards. Educ Res 2005;47:251-61.
88. Thorsteinson TJ. Framing effects on the setting of critical scores for content valid tests. Hum Perform 2006;19:201-17.
89. Ebel RL, Frisbie DA. Essentials of Educational Measurement. 5th ed. New Jersey: Prentice-Hall, 1991.
90. Linn RL, Gronlund NE. Measurement and Assessment in Teaching. 8th ed. New Jersey: Prentice-Hall, 2000.
91. Brown B. Trends in assessment. In: Harden R, Hart I, Mulholland H, editors. Approaches to the Assessment of Clinical Competence. Vol. 1. Dundee Centre for Medical Education, 1992.
92. Sireci SG, Parker P. Validity on trial: Psychometric and legal conceptualizations of validity. Educ Meas 2006;25:27-34.
93. Phillips SE. GI Forum V. Texas Education Agency: Psychometric evidence. Appl Meas Educ 2000;13:343-85.
94. Mehrens WA, Popham WJ. How to evaluate the legal defensibility of high-stake tests. Appl Meas Educ 1992;5:265-83.
95. Sireci SG. The construct of the content validity. Soc Indicat Res 1998;45:83-117.
96. Crocker LM, Miller MD, Franks EA. Quantitative methods for assessing the fit between test and curriculum. Appl Meas Educ 1989;2:179-94.
97. Cole NS, Nitko AJ. Measuring program effects. In: Berk RA, editor. Educational Evaluation Methodology: The State of the Art. Baltimore: Johns Hopkins University Press, 1981:32-63.
98. Wiggins G. Teaching to the (Authentic) test. Educ Leader 1989;46:41-7.
99. Monyooe LA. On shifting sands: Exploring the curriculum and assessment dichotomy. Int J Instructional Technol Distance Learning 2004:1:11-26.
100. La Marca P, Redfield D, Winter P, Bailey A, Despriet L. State Standards and State Assessment Systems: A Guide to Alignment. Series on Standards and Assessments. Washington, DC: Council of Chief State School Officers, 2000.
101. Newble D. Assessment. In: Jolly B, Rees L, editors. Medical Education in the Millennium. 1st ed. UK: Oxford University Press, 1998:129-42.
102. Brown B, Roberts J, Rankin J, Stevens B, Tompkins C, Patton D. The objective structured clinical examination: reliability and validity. In: Hart IR, Harden RM, Walton HJ, editors. Further Developments in Assessing Clinical Competence. International Conference Proceedings; Ottawa, Canada, 1987:563-71.
103. Fraser R, McKinley RK, Mulholland H. Consultation competence in general practice: establishing the face validity of prioritized criteria in the Leicester assessment package. Br J General Pract 1994;44:109-13.
104. Bhola DS, Impara JC, Buckendahl CW. Aligning tests with states' content standards: Methods and Issues. Educ Meas 2003;22:21-9.
105. Green BF. A primer of testing. Am Psychol 1981;36:1001-11.
106. Webb NL. Research monogram No. 6: Criteria for alignment of expectations and assessment in mathematics and science education. Washington, DC: Council of Chief State School Officers, 1997:1-46.