Rater training to support high-stakes simulation-based assessments

Journal of Continuing Education in the Health Professions

Volumn 32, Issue 4, 2012, Pages 279-286

  Feldman, Moshe ^a   Lazzara, Elizabeth H ^b   Vanderbilt, Allison A ^a   DiazGranados, Deborah ^a  

^a VIRGINIA COMMONWEALTH UNIVERSITY   (United States)

^b UNIVERSITY OF CENTRAL FLORIDA   (United States)

Author keywords

Assessment; Certification; Credentialing; Licensure; Rater training; Simulation

Indexed keywords

ACCREDITATION; ARTICLE; CERTIFICATION; CLINICAL COMPETENCE; COMPUTER SIMULATION; CURRICULUM; EDUCATION; HUMAN; MEDICAL EDUCATION; MEDICAL SOCIETY; METHODOLOGY; REPRODUCIBILITY; TEACHING;

ACCREDITATION; CERTIFICATION; CLINICAL COMPETENCE; COMPETENCY-BASED EDUCATION; COMPUTER SIMULATION; EDUCATION, MEDICAL, GRADUATE; EDUCATIONAL MEASUREMENT; HUMANS; REPRODUCIBILITY OF RESULTS; SOCIETIES, MEDICAL; SPECIALTY BOARDS; TEACHING;

EID: 84871650307     PISSN: 08941912     EISSN: 1554558X     Source Type: Journal    
DOI: 10.1002/chp.21156     Document Type: Article

References (70)

1. Blank L, Kimball H, McDonald W, Merino J. ABIM Foundation. ACP Foundation. European Federation of Internal Medicine. Medical professionalism in the new millennium: a physician charter 15 months later. Ann Intern Med. 2003;138:839-841.
2. Boulet JR, Murray DJ. Simulation-based assessment in anesthesiology. Anesthesiology. 2010;112:1041-1052.
3. Nahrwold DL. Continuing medical education reform for competency-based education and assessment. J Contin Educ Health Prof. 2005;25:168-173.
4. Melnick DE. Physician performance and assessment and their effect on continuing medical education and continuing professional development. J Contin Educ Health Prof. 2004;24:S38-S49.
5. Institute of Medicine (IOM). Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academies Press; 2001.
6. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296:1094-1102.
7. Prytowsky JB, Bordage G, Feinglass JM. Patient outcomes for segmental colon resection according to surgeon's training, certification, and experience. Surgery. 2002;132(4):663-672.
8. Goodman JC, Villarreal P, Jones B. The social cost of adverse medical events and what we can do about it. Health Aff. 2011;30(4):590-595.
9. Evans R, Elwyn G, Edwards A. Review of instruments for peer assessment of physicians. BMJ. 2004;328:1-5.
10. Holmboe ES. Assessment of the practicing physician: challenges and opportunities. J Contin Educ Health Prof. 2008;28(S1):S4-S10.
11. Institute of Medicine (IOM). Redesigning Continuing Education in the Health Professions. Washington, DC: National Academies Press; 2010.
12. Moore DE, Green JS, Gallis HA. Achieving the desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29(1):1-15.
13. van Hoof TJ, Meehan TP. Integrating essential components of quality improvement into a new paradigm for continuing education. J Contin Educ Health Prof. 2011;31(3):207-214.
14. Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the accreditation council for graduate medical education: a systematic review. Acad Med. 2009;84(3):301-309.
15. Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees. JAMA. 2009;302(12):1316-1326.
16. Gerbert B. Criteria for evaluating methods used to assess physician performance. Mobius. 1984;4(4):44-47.
17. Salas E, Rosen MA. Beyond the bells and whistles: when simulation-based team training works best. Harvard CRICO RMF Forum. 2008;26(4):6-7.
18. Lievens F, Patterson F. The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job-performance in advanced-level high-stakes selection. J Appl Psychol. 2011;96(5):927-940.
19. Gordon J, Wilkerson W, Shaffer D, Armstrong E. Practicing medicine without risk: students and educators responses to high-fidelity patient simulation. Acad Med. 2001;76:469-472.
20. Rosen MA, Salas E, Silvestri S, Wu TS, Lazzara EH. A measurement tool for simulation-based training in emergency medicine: the simulation module for assessment of resident targeted event responses (SMARTER) approach. Simul Healthc. 2008;3:170-179.
21. McGaghie W, Siddall VJ, Mazmanian PE, Myers J. Lessons for continuing medical education from simulation research in undergraduate and graduate medical education. Chest. 2009;135(3):62S-68S.
22. Levine AI, Schwartz AD, Bryson EO, Demaria S Jr. Role of simulation in US physician licensure and certification. Mt Sinai J Med. 2012;79:140-153.
23. Boulet JR, Jeffries PR, Hatala RA, Kornfdorffer JR, Feinstein DM, Roche JP. Research regarding methods of assessing learning outcomes. Simul Healthc. 2011;6:S48-S51.
24. Haladyna TM, Kramer GA. The validity of subscores for a credentialing test. Eval Health Prof. 2004;27(4):349-368.
25. Downing SM. Threats to the validity of clinical teaching assessments: what about rater error? Med Educ. 2005;39:350-355.
26. Downing SM, Haladyna SM. Validity threats: overcoming interference with proposed interpretations of assessment data. Med Educ. 2004;38:327-333.
27. Kogan JR, Conforti L, Bernabeo E, Iobst W, Holmboe E. Opening the black box of clinical skills assessment via observation: a conceptual model. Med Educ. 2011;45:1048-1060.
28. Hatala R, Scalese RJ, Cole G, Bacchus M, Kassen B, Issenberg SB. Development and validation of a cardiac findings checklist for use with simulator-based assessments of cardiac physical examination competence. Simul Healthc. 2009;4(1):17-22.
29. Graham J, Mudumbai SC, Gaba DM, Boulet J, Howard SK, Davies MF. External validation of simulation-based assessments with other performance measures of third-year anesthesiology residents. Simul Healthc. 2012;7(2):73-80.
30. Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995;123:795-799.
31. Holmboe ES, Hawkins RE, Huot SJ. Effects of training in direct observation of medial residents' clinical competence: a randomized trial. Ann Intern Med. 2004;140:874-881.
32. Cook DA, Dupras DM, Beckman TJ, Thomas KG, Pankratz VS. Effect of rater training on reliability and accuracy of mini-CEX scores: a randomized, controlled trial. J Intern Med. 2008;24(1):74-79.
33. Yule S, Rowley D, Flin R, Maran N, Youngson G, Duncan J, Paterson-Brown S. Experience matters: comparing novice and expert ratings of non-technical skills using the NOTSS system. Surg Educ. 2009;79:154-160.
34. van Zanten M, Boulet JR, McKinley D. Using standardized patients to assess the interpersonal skills of physicians: six years' experience with a high-stakes certification examination. Health Commun. 2007;22(3):195-205.
35. DeMaria S Jr, Levin AI, Bryson EO. The use of multimodality simulation in the retraining of the physician for medical licensure. J Clin Anesthesiol. 2010;22:294-299.
36. Dong Y, Suri HS, Cook DA, et al. Simulation-based objective assessment discerns clinical proficiency in central line placement: a construct validation. Chest. 2010;137:1050-1056.
37. Goldstein IL, Ford JK. Training in Organizations. 4th ed. Belmont, CA: Wadsworth; 2002.
38. DuBois CL, Sackett PR, Zedeck S, Fogli L. Further exploration of typical and maximum performance criteria: definitional issues, prediction, and White-Black differences. J Appl Psychol. 1993;78(2):205-211.
39. Kleinmann M. Are rating dimensions in assessment centers transparent for participants? Consequences for criterion and construct validity. J Appl Psych.. 1993;78:988-993.
40. Sackett PR, Zedeck S, Fogli l. Relations between measures of typical and maximum job performance. J Appl Psychol. 1988;73(3):482-486.
41. Baker R, Camosso-Stefinovic J, Gillies C, et al. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2010;3:CD005470.
42. Machin MA, Fogarty GJ. Perceptions of training-related factors and personal variables as predictors of transfer implementation intentions. J Bus Psychol. 2003;18:51-71.
43. Royer JM. Theories of the transfer of learning. Educ Psychol. 1979;14:53-70.
44. Landon BE, Normand SL, Blumenthal D, Daley J. Physician clinical performance assessment: prospects and barriers. JAMA. 2003;290:1183-1189.
45. American Board of Medical Specialties. http://www.abms.org/About_Board_Certification/means.aspx. Accessed March 27, 2012.
46. American Board of Medical Specialties. http://www.abms.org/newsandevents/medianewsroom/releases/releaseBoardEligibility02072012.aspx. Accessed March 27, 2012.
47. Havens C, Mallin J. Climate change: It's not about the weather-continuing medical education and maintenance of certification and licensure. Perm J. 2011;15(3):88-92.
48. Cassel CK, Holmboe ES. Professionalism and accountability: the role of specialty board certification. Trans Am Clin Climatol Assoc. 2008;119:295-304.
49. Maatsch JL. Assessment of clinical competence on the Emergency Medicine Specialty Certification Examination: the validity of examiner ratings of simulated clinical encounters. Ann Emerg Med. 1981;10(10):504-507.
50. Gallagher CJ, Tan JM. The current status of simulation in the maintenance of certification in anesthesia. Int Anesthesiol Clin. 2010;48:83-99.
51. Berkenstadt H, Ziv A, Gafni N, Sidi A. The validation process of incorporating simulation-based accreditation into the anesthesiology Israeli national board exam. Isr Med Assoc J. 2006;8:728-733.
52. Gallagher AG, Cates CU. Approval of virtual reality training for carotid stenting: what this means for procedural-based medicine. JAMA. 2004;292:3024-3025.
53. Amin Z, Boulet JR, Cook DA, et al. Technology-enabled assessment of health professions education: Consensus statement and recommendations from the Ottawa 2010 conference. Med Teach. 2011;33:364-369.
54. Nadler I, Sanderson P, Liley H. The accuracy of clinical assessments as a measure for teamwork effectiveness. Simul Healthc. 2011;6:260-268.
55. Pelgrim EM, Kramer AM, Mokkink HA, van den Elsen L, Grol RM, van der Vleuten CM. In-training assessment using direct observation of single-patient encounters: a literature review. Adv Health Sci Educ. 2011;16:131-142.
56. Evans LV, Morse JL, Hamann CJ, Osborne M, Lin Z, D'Onofrio G. The development of an independent rater system to assess residents' competence in invasive procedures. Acad Med. 2009;84(8):1135-1143.
57. Borman WC. Format and training effects on rating accuracy and rater errors. J Appl Psychol. 1979;64(4):410-421.
58. Gaugler BB, Rudolph AS. The influence of assessee performance variation on assessors' judgments. Pers Psychol. 1992;45(1):77-98.
59. Woehr DJ, Huffcutt AI. Rater training for performance appraisal: a quantitative review. J Occup Organ Psychol. 1994;67:189-205.
60. Pulakos ED. A comparison of rater training programs: error training and accuracy training. J Appl Psychol. 1984;69:581-588.
61. Bernardin HJ, Pence EC. Effects of rater training: creating new response sets and decreasing accuracy. J Appl Psychol. 1980;65:60-66.
62. Hedge JW, Kavanagh MJ. Improving the accuracy of performance evaluations: comparison of three methods of performance appraiser training. J Appl Psychol. 1988;73:68-73.
63. Fowlkes J, Dwyer DJ, Oser RL, Salas E. Event-based approach to training (EBAT). Int J Aviat Psychol. 1998;8(3):209-222.
64. Ludbrook J, Marshall VR. Examiner training for clinical examinations. Br J Med Educ. 1971;5:152-155.
65. Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to Their Development and Use. 3rd ed. New York, NY: Oxford University Press; 2003.
66. Iramaneerat C, Yudkowsky R, Myford CM, Downing SM. Quality control of an OSCE using generalizability theory and many-faceted Rasch measurement. Adv Health Sci Educ Theory Pract. 2008;13(4):479-493.
67. Williams RG, Klamen DA, McGaphie WC. Cognitive, social, and environmental sources of bias in clinical performance ratings. Teach Learn Med. 2003;15:270-292.
68. Iramaneerat C, Yudkowsky R. Rater errors in a clinical skills assessment of medical students. Eval Health Prof. 2007;30:266-283.
69. McLaughlin K, Ainslie M, Coderre S, Wright B, Violato C. The effect of differential rater function over time (DRIFT) on objective structured clinical examination ratings. Med Educ. 2009;43(1):989-992.
70. Ziv A, Rubin O, Sidi A, Berkenstadt H. Credentialing and certifying with simulation. Anesthesiol Clin. 2007;25:209-223.