Standard setting in medical education

Academic Medicine

Volumn 71, Issue 10, 1996, Pages s112-s120

  Cusimano, Michael D ^a  

^a NONE

Author keywords

[No Author keywords available]

Indexed keywords

CERTIFICATION; CONFERENCE PAPER; ECONOMIC ASPECT; MEDICAL DECISION MAKING; MEDICAL EDUCATION; MEDICAL EXAMINATION; PRIORITY JOURNAL; STATISTICAL ANALYSIS;

EID: 0029859628     PISSN: 10402446     EISSN: 1938808X     Source Type: Journal    
DOI: 10.1097/00001888-199610000-00062     Document Type: Article

References (84)

1. Livingston S, Zicky M. Passing Scores: A Manual for Setting Standards of Performance on Educational and Occupational Tests. Princeton, NJ: Educational Testing Service, 1982.
2. Glass GV. Standards and criteria. J Educ Mcas. 1978;15:237-61.
3. Shepard L. Setting Standards. Washington, DC: National Council on Measurement in Education, 1979.
4. Hamblcton R, Eignor D. Competency Test Development, Validation and Standard Setting. Berkeley, CA: McCutchan, 1980.
5. Berk RA. A consumer’s guide to setting performance standards on criterion-referenced tests. Rev Educ Res. 1986;6:137-72.
6. Jaeger RM. Certification of student competence. In: Linn RL (ed). Educational measurement. New York: MacMillan, 1989:485-515.
7. Kane M. Validating the performance standards associated with passing scores. Rev Educ Res. 1994;64:425-61.
8. Meskauskas JA. Evaluation models for criterion-referenced testing: views regarding mastery and standard setting. Rev Educ Res. 1976;45:133-58.
9. Berk RA. A consumers’ guide to criterion-referenced test reliability. J Educ Meas. 1980;17:323-49.
10. Beuk CH. A method for reaching a compromise between absolute and relative standards in examinations. J Educ Meas. 1984;21:147-52.
11. Hofstee WKB. Caesuurproblemen opgelost. Onderzoek van Onderwijs. 1977;6: 6-7.
12. Hofstee WKB. The case for compromise in educational selection and grading. In: Anderson SB, Helmick, JS (eds). On Educational Testing. San Francisco: Jossey-Bass, 1983.
13. Shepard L. Standard setting issues and methods. Appl Psychol Meas. 1980;4: 447-67.
14. Hambleton RK. Contributions to criterion-referenced testing technology: an introduction. Appl Psychol Meas. 1980;4:421-4.
15. Nedelsky L. Absolute grading standards for objective tests. Educ Psychol Meas. 1954;14:13-19.
16. Angoff WA. Scales, norms and, equivalent scores. In: Thorndike RL (ed). Educational Measurement. Washington, DC: American Council on Education, 1971: 508-600.
17. Ebcl RL. Essentials of Educational Measurement. 3rd ed. Englewood Cliffs, NJ: Prentice-Hall, 1979.
18. Jaeger RM. An iterative structured judgment process for establishing standards on competency tests: theory’ and application. Educ Eval Policy Analysis. 1982;4: 461-75.
19. Poggio JP. Practical considerations when setting test standards: a look at the process used in Kansas. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA, 1984.
20. Melican TN. Identification of items that are hard to rate accurately in Angoff’s standard setting method. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA, 1984.
21. Cross, LH, Impara JC, Frary RB, Jaeger RM. A comparison of three methods for establishing minimum standards on the National Teacher Examinations. J Educ. Meas. 1984;21:113-29.
22. Norcini JJ, Lipner SR, Lynn LO, Strecker AC. A comparison of three variations on a standard-setting method. J Educ. Meas. 1987;24:56-64.
23. Norcini J, Shea J. The reproducibility of standards over groups and occasions. Appl Meas Educ. 1992;5:63-72.
24. Shepard LA. Setting performance standards. In: Berk RA (eds). A Guide To Criterion Reference Test Construction. Baltimore, MD: Johns Hopkins University Press, 1984:169-98.
25. Skakun EN, Kling, S. Comparability of methods for setting standards. J Educ Meas. 1980;17(3):229-35.
26. Glasnapp DR, Poggio JP, Eros DS. An analysis of the validity of judgmental methods used to set test standards. Paper presented at the annual meeting of the American Educational Research Association, 1983.
27. Zieky MJ, Livingston SA. Manual for Setting Standards on the Basic Skills Assessment Tests. Princeton, NJ: Educational Testing Service, 1977.
28. Poggio JP, Glasnapp DR, Eros DS. An empirical investigation of the Angoff, Ebel, and Nedelsky standard setting methods. Paper presented at the annual meeting of the American Educational Research Association, 1981.
29. Mills CN. A comparison of three methods of establishing cut-off scores in criterion-referenced tests. J Educ Meas. 1983;20:283-92.
30. Mills CN, Barr JE. A Comparison of Standard Setting Methods: Do the Same Judges Establish the Same Standards with Different Methods? Baton Rouge, LA: Louisiana Department of Education, 1983.
31. Koffler SL. A comparison of approaches for setting proficiency standards. J Educ Meas. 1980; 17(3): 167 — 78.
32. Skakun EN. A review of procedures for establishing cutting scores of multiple-choice examinations. Ann R Coll Phys Surg Canada. 1982;15(6):517-20.
33. de Gruijter DNM. Compromise models for establishing examination standards. J Educ Meas. 1985;22(4):263-9.
34. de Gruijter DNM. Accounting for uncertainty in performance standards. Paper presented at the International Symposium on Educational Testing. Antwerp, Belgium: 1980.
35. Ebel RL. The case for norm-referenced measurements. Educ Res. 1978:3-5.
36. Hofstee WKB. Een alternatief voor normhandhaving bij toetsen. Nederlands Tid-jschrift voorde Psychologic. 1973;28:215-27.
37. Jaeger RM. Establishing Standards on Tests Used for Certification of Teachers: Some Measurement Issues. Undated, unpublished manuscript.
38. Berk RA. Determination of optional cutting scores in criterion-referenced measurement. J Exp Educ. 1976;45:4-9.
39. Halpin G, Sigmon G. Minimum competency standards set by three divergent groups of raters using three judgmental procedures: implications for validity. Educ Psychol Meas. 1983;43:185-96.
40. Andrew BJ, Hecht JT. A preliminary' investigation of two procedures for setting examination standards. Educ Psychol Meas. 1976;36:45-50.
41. Hambleton RK, Rogers FIJ. Technical advances in credentialing examinations. Eval Health Professions. 1986;9:205-29.
42. Clauser B, Orr N, Clyman S. Models for making pass/fail decisions for performance assessment involving multiple cases. In: Rothman AI (ed). Proceedings of the Sixth Ottawa Conference on Medical Education. Toronto, Ontario, Canada; University of Toronto Bookstore Custom Publishing, 1995.
43. Clauser B, McKinley D, Subhiyah R, Nungester RJ, Ripkey D, Clyman S. Scoring performance-based assessment by modelling expert judgments. Paper presented at the annual meeting of the American Educational Research Association. New Orleans, LA, 1994.
44. Woehr DJ, Arthur W, Fehrmann LM. An empirical comparison of cutoff score methods for content-related and criterion-related validity settings. Educ Psychol Meas. 1991;51:1029-39.
45. Traub RE. Reliability for the Social Sciences: Theory and Applications. Thousand Oaks, CA: Sage Publications, 1994.
46. Meskauskas JA. Setting standards for credentialing examinations. Eval Health Professions. 1986;9:187-203.
47. Livingston SA, Zieky MJ. A Comparative Study of Standard-Setting Methods. Princeton, NJ: Educational Testing Service, 1983:1 -48.
48. Fehrmann LM, Woehr JD, Arthur JW. The Angoff cutoff score method: the impact of frame-of-reference rater training. Educ Psychol Meas. 1991;51: 857-72.
49. Hambleton R, Plake B. Using an extended Angoff procedure to set standards on complex performance assessments. Appl Meas Educ. 1995;8:41.
50. Busch JC, Jaeger RM. Influence of type of judge, normative information, and discussion on standards recommended for national teacher examinations. J Educ Meas. 1990;27:145-63.
51. Fitzpatrick AR. Social influences in standard setting: the effects of social interaction on group judgments. Rev Educ Res. 1989;59:315-28.
52. Hambleton RK, Powell S. A framework for viewing the process of standard setting. Eval Health Professions. 1983;6:3-24.
53. van der Linden WL. A latent trait method for determining intrajudge inconsistency in the Angoff and Nedelsky techniques of standard setting. J Educ Meas. 1982;19:295-308.
54. Secolsky C. Attitudinal and interpretive influences on difficulty ratings in setting standards for high school minimum competency tests. J Educ Res. 1987;80: 227-32.
55. Saunders JC, Ryan JP, Huynh H. A comparison of two approaches to setting passing scores based on the Nedelsky procedure. Appl Psychol Meas. 1981;5:209-17.
56. Plake BS, Melican GJ. Effects of item context on intrajudge consistency of expert judgment via the Nedelsky standard setting method. Educ Psychol Meas. 1980;49:45-51.
57. Follman J, Lowe AS, Miller C. Graphics variables and reliability and level of essay grades. Amer Educ Res J. 1971;8:365-73.
58. Hales LW, Tokar E. The effect of quality of preceding responses on grades assigned by subsequent responses to an essay question. J Educ Meas. 1975;12:115-7.
59. Huck S, Bounds W. Essay grades: An interaction between graders’ handwriting clarity and neatness of examination papers. Amer Educ Res J. 1972;9:279-83.
60. Cason GJ, Cason CL. A deterministic theory of clinical performance rating. Eval Health Professions. 1984;7:221-47.
61. Lunz ME, Linacre M, Wright BD. Impact of judge severity on examination scores. Paper presented at the annual meeting of the American Educational Research Association, 1988.
62. Lunz EM, Stahl AJ, Wright DB, Linacre MJ. Variation among examiners and protocols on oral examinations. Paper presented at the American Educational Research Association, 1989.
63. Bemadin HJ, Buckley MR. Strategics in rater training. Acad Manage Rev. 1981;6:205-12.
64. Case S, Fincher R, Becker D, Melnick D. Establishing pass/fail and honors performance standards for the NBME medicine clerkship exam: an application of the Hofstcc Method. Paper presented at the annual meeting of the American Educational Research Association. San Francisco, CA, 1995.
65. Mills C, Melican G. A Preliminary Investigation of Three Compromise Methods for Establishing Cut Off Scores. Princeton, NJ: Educational Testing Service, 1987.
66. Cusimano MD, Cohen R, Hutchison C, et al. Setting performance standards in medicine: an assessment of the Hofstee procedure. Paper presented at the Seventh International Ottawa Conference on Medical Education, Maastricht, The Netherlands, June 1996.
67. Norcini JJ, Hancock EW, Webster GD, Grosso LJ, Shea JA. A criterion-referenced examination of physician competence. Eval Health Prof. 1988; 11:98-112.
68. Brennan RL, Lockwood RE. A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory. Appl Psychol Meas. 1980;4: 219-40.
69. Reilly RR, Lockwood RE, Israelski EW. A comparison of direct and indirect methods for setting minimum passing scores. Appl Psychol Meas. 1984;8:421-9.
70. Swanson DB, Dillon GF, Ross LEP. Setting content-based standards for National Board Exams: initial research for the Comprehensive Part I Examination. Acad Med. 1990;65(9 suppl):Sl7-S18.
71. Brennan RL, Lockwood RE. A comparison of two cutting score procedures. ACT Tech Bull. 1979;33.
72. Cronbach LJ, Glescr GC, Nanda H, Rajaratnam N. The Dependability of Behavioral Measurements: Theory of Generalizability for Score and Profiles. New York: Wiley, 1972.
73. Brennan RL. Elements of Generalizability Theory. Iowa City, IA: American College Testing Program, 1983:609-25.
74. Reznick R, Smee S, Rothman A, et al. An objective structured clinical examination for the licentiate: report of the pilot project of the Medical Council of Canada. Acad Med. 1992;67:487-94.
75. Kopclow M, Schnabel G, Chochinov C, et al. Criterion referenced standard setting for a standardized patient examination. In: Rothman Al, Cohen R. (eds). Proceedings of the Sixth Ottawa Conference on Medical Education. Toronto, Ontario, Canada: University of Toronto Bookstore Custom Publishing, 1995.
76. Lescop J, Grand’Maison P, Brailovsky C. Setting minimal standards in the Quebec Licensing Exam. In: Rothman Al (ed). Proceedings of the Sixth Ottawa Conference on Medical Education. Toronto, Ontario, Canada: University of Toronto Bookstore Custom Publishing, 1995.
77. Dauphinec D, Blackmorc D. Setting minimal passing standards for the qualifying examination of the Medical Council of Canada: the transition from norm referencing to criterion referencing. In: Rothman Al (ed). Proceedings of the Sixth Ottawa Conference on Medical Education. Toronto, Ontario, Canada: University of Toronto Bookstore Custom Publishing, 1995.
78. Cohen R, Rothman AI, Poldre P, Dirks F, Ross J. Defining test cutting scores in a criterion referenced multistation clinical skills examination. In: Hart GR (ed). Current Developments in Assessing Clinical Competence. Montreal, Quebec, Canada: Can-Heal Publications, 1992.
79. Norcini J, Stillman P, Sutnick A, Regan M, Williams R, Friedman M. Scoring and standard setting with standardized patients. Eval Health Prof. 1993;16:322-32.
80. Sutnick A, Norcini J, Stillman P, Friedman M. Scoring and standard setting for the ECFMG Clinical Competence Assessment. In: Rothman Al (ed). Proceedings of the Sixth Ottawa Conference on Medical Education. Toronto, Ontario, Canada: University of Toronto Bookstore Custom Publishing, 1995.
81. Stillman P, Swanson D, et al. Assessment of clinical skills of residents utilizing standardized patients. Ann Intern Med. 1991;114:393-401.
82. Stillman, Paula. Office of the Curriculum Dean, University of Massachusetts Medical School, Amherst, MA, June 14, 1993.
83. Rothman AI, Blackmorc D, Cohen R, Reznick R. The consistency and uncertainty in examiners definitions of pass/fail performance on OSCE stations. Paper presented at the annual meeting of the American Educational Research Association. San Francisco, CA, 1995.
84. Eisner EW. Educational connoisscurship and criticism: their form and function in educational evaluation. J Acsthet Educ. 1976; 10(3): 135-150.