Review article: Assessment in anesthesiology education

Canadian Journal of Anesthesia

Volumn 59, Issue 2, 2012, Pages 182-192

  Boulet, John R ^a,b   Murray, David ^c  

^a FOUNDATION FOR ADVANCEMENT OF INTERNATIONAL MEDICAL EDUCATION AND RESEARCH   (United States)

^b Educational Commission for Foreign Medical Graduates   (United States)

^c WASHINGTON UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANESTHESIOLOGY; ANESTHESIST; CERTIFICATION; CLINICAL ASSESSMENT; CLINICAL PRACTICE; COMMUNICATION SKILL; HUMAN; INTERPERSONAL COMMUNICATION; LEARNING STYLE; MEDICAL EDUCATION; MEDICAL PRACTICE; OUTCOME OF EDUCATION; PATIENT CARE; PERFORMANCE MEASUREMENT SYSTEM; POSTGRADUATE EDUCATION; PRACTICE BASED LEARNING; PRIORITY JOURNAL; PROFESSIONAL COMPETENCE; PROFESSIONAL KNOWLEDGE; PROFESSIONAL STANDARD; PROFESSIONALISM; RELIABILITY; REVIEW; SKILL; SYSTEM BASED PRACTICE; VALIDITY;

ANESTHESIOLOGY; CERTIFICATION; CLINICAL COMPETENCE; COMPETENCY-BASED EDUCATION; COMPUTER SIMULATION; EDUCATION, MEDICAL; EDUCATIONAL MEASUREMENT; EDUCATIONAL TECHNOLOGY; HUMANS; PEER GROUP; PSYCHOMETRICS;

EID: 84856963077     PISSN: 0832610X     EISSN: 14968975     Source Type: Journal    
DOI: 10.1007/s12630-011-9637-9     Document Type: Review

References (83)

1. TP Hofer RA Hayward 2002 Are bad outcomes from questionable clinical decisions preventable medical errors? A case of cascade iatrogenesis Ann Intern Med 137 5 Part 1 327 333 12204016 (Pubitemid 34977134)
2. Frank JR. The CanMEDS 2005 Physician Competency Framework. Better standards. Better physicians. Ottawa, Canada. The Royal College of Physicians and Surgeons of Canada. Available from URL: http://meds.queensu.ca/medicine/ obgyn/pdf/CanMEDS2005.booklet.pdf (accessed August 2011).
3. Accreditation Council for Graduate Medical Education. ACGME Outcome Project. Available from: http://www.acgme.org/acWebsite/home/home.asp (accessed August 2011).
4. Glavin R, Flin R. Review article: The influence of psychology and human factors on education in anesthesiology. Can J Anesth 2012; 59: this issue. DOI: 10.1007/s12630-011-9634-z .
5. Bould MD, Naik VN, Hamstra SJ. Review article: New directions in medical education related to anesthesiology and perioperative medicine. Can J Anesth 2012; 59: this issue. DOI: 10.1007/s12630-011-9633-0 .
6. JR Boulet DJ Murray 2010 Simulation-based assessment in anesthesiology: requirements for practical implementation Anesthesiology 112 1041 1052 20234313 10.1097/ALN.0b013e3181cea265
7. MH Davis 2003 OSCE: the Dundee experience Med Teach 25 255 261 12881046 10.1080/0142159031000100292 (Pubitemid 36806319)
8. JJ Norcini DW McKinley 2007 Assessment methods in medical education Teaching and Teacher Education 23 239 250 10.1016/j.tate.2006.12.021
9. RM Epstein 2007 Assessment in medical education N Engl J Med 356 387 396 17251535 10.1056/NEJMe078002 1:CAS:528:DC%2BD2sXhtVGrtb8%3D (Pubitemid 46193082)
10. JR Kogan ES Holmboe KE Hauer 2009 Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review JAMA 302 1316 1326 19773567 10.1001/jama.2009.1365 1:CAS:528:DC%2BD1MXhtFyltrvL
11. GE Miller 1990 The assessment of clinical skills/competence/performance Acad Med 65 9 Suppl S63 S67 2400509 10.1097/00001888-199009000-00045 1:STN:280:DyaK3czotVWluw%3D%3D
12. AA Edler RG Fanning MI Chen, et al. 2009 Patient simulation: a literary synthesis of assessment tools in anesthesiology J Educ Eval Health Prof 6 3 20046456 10.3352/jeehp.2009.6.3
13. JR Boulet SM Smee GF Dillon JR Gimpel 2009 The use of standardized patient assessments for certification and licensure decisions Simul Healthc 4 35 42 19212249 10.1097/SIH.0b013e318182fc6c
14. GF Dillon JR Boulet RE Hawkins DB Swanson 2004 Simulations in the United States Medical Licensing Examination (USMLE) Qual Saf Health Care 13 Suppl 1 i41 i45 15465954 10.1136/qshc.2004.010025 (Pubitemid 39389377)
15. D Nestel R Kneebone S Black 2006 Simulated patients and the development of procedural and operative skills Med Teach 28 390 391 16807185 (Pubitemid 43977028)
16. Z Amin JR Boulet DA Cook, et al. 2011 Technology-enabled assessment of health professions education: consensus statement and recommendations from the Ottawa 2010 conference Med Teach 33 364 369 21517684 10.3109/0142159X.2011. 565832
17. MD Bould NA Crabtree VN Naik 2009 Assessment of procedural skills in anaesthesia Br J Anaesth 103 472 483 19720612 10.1093/bja/aep241 1:STN:280:DC%2BD1MnhvVKhtQ%3D%3D
18. Leblanc VR. Review article: Simulation in anesthesia: state of the science and looking forward. Can J Anesth 2012; 59: this issue. DOI: 10.1007/s12630-011-9638-8 .
19. J Norcini B Anderson V Bollela, et al. 2011 Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference Med Teach 33 206 214 21345060 10.3109/0142159X.2011.551559
20. DA Cook TJ Beckman 2006 Current concepts in validity and reliability for psychometric instruments: theory and application Am J Med 119 166.e7 10.1016/j.amjmed.2005.10.036 (Pubitemid 43151485)
21. JH Devitt MM Kurrek MM Cohen D Cleave-Hogg 2001 The validity of performance assessments using simulation Anesthesiology 95 36 42 11465581 10.1097/00000542-200107000-00011 1:STN:280:DC%2BD3Mvhs1eltg%3D%3D (Pubitemid 32622994)
22. Boulet JR. Generalizability theory: basics. In: Everitt BS, Howell D (Eds). Encyclopedia of Statistics in Behavioral Science. Chichester: John Wiley & Sons, Ltd; 2005. 704-11.
23. Boulet JR, Swanson DB. Psychometric challenges of using simulations for high-stakes assessment. In: Dunn WF (Ed). Simulators in Critical Care Education and Beyond. Des Plains, IL: Society of Critical Care Medicine; 2004: 119-30.
24. Clauser BE, Margolis MJ, Swanson DB. Issues of validity and reliability for assessments in medical education. In: Holmboe ES, Hawkins RE (Eds). Practical Guide to the Evaluation of Clinical Competence, 1st ed. Philadelphia, PA: Mosby/Elsevier; 2008: 10-23
25. MR Raymond 2001 Job analysis and the specification of content for licensure and certification examinations Applied Measurement in Education 14 369 415 10.1207/S15324818AME1404-4 (Pubitemid 33610870)
26. RA Kearney P Sullivan E Skakun 2000 Performance on ABA-ASA in-training examination predicts success for RCPSC certification. American Board of Anesthesiology-American Society of Anesthesiologists. Royal College of Physicians and Surgeons of Canada Can J Anesth 47 914 918 10989866 10.1007/BF03019676 1:STN:280:DC%2BD3cvktF2hsw%3D%3D
27. TC Gale MJ Roberts PJ Sice, et al. 2010 Predictive validity of a selection centre testing non-technical skills for recruitment to training in anaesthesia Br J Anaesth 105 603 609 20736232 10.1093/bja/aeq228 1:STN:280:DC%2BC3cfpvVGntA%3D%3D
28. DJ Murray JR Boulet M Avidan, et al. 2007 Performance of residents and anesthesiologists in a simulation-based skill assessment Anesthesiology 107 705 713 18073544 10.1097/01.anes.0000286926.01083.9d (Pubitemid 351325961)
29. JC McClintock GP Gravlee 2010 Predicting success on the certification examinations of the American Board of Anesthesiology Anesthesiology 112 212 219 20010419 10.1097/ALN.0b013e3181c62e2f
30. The American Board of Anesthesiology. Maintenance of certfication in anesthesiology (MOCA). Available from: http://www.theaba.org/Home/ anesthesiology-maintenance (accessed July 2011).
31. E Ottestad JR Boulet GK Lighthall 2007 Evaluating the management of septic shock using patient simulation Crit Care Med 35 769 775 17235260 10.1097/01.CCM.0000256849.75799.20 (Pubitemid 46407789)
32. JR Wilkinson JG Crossley A Wragg P Mills G Cowan W Wade 2008 Implementing workplace-based assessment across the medical specialties in the United Kingdom Med Educ 42 364 373 18338989 10.1111/j.1365-2923.2008.03010.x (Pubitemid 351388738)
33. A Miller J Archer 2010 Impact of workplace based assessment on doctors' education and performance: a systematic review BMJ 341 c5064 20870696 10.1136/bmj.c5064
34. J Norcini V Burch 2007 Workplace-based assessment as an educational tool: AMEE Guide No. 31 Med Teach 29 855 871 18158655 10.1080/01421590701775453
35. H Berkenstadt D Erez Y Munz D Simon A Ziv 2007 Training and assessment of trauma management: the role of simulation-based medical education Anesthesiol Clin 25 65 74 17400156 10.1016/j.atc.2006.11.004 (Pubitemid 46497888)
36. EA Hunt AR Walker DH Shaffner MR Miller PJ Pronovost 2008 Simulation of in-hospital pediatric medical emergencies and cardiopulmonary arrests: highlighting the importance of the first 5 minutes Pediatrics 121 e34 e43 18166542 10.1542/peds.2007-0029
37. JB Cooper VR Taqueti 2004 A brief history of the development of mannequin simulators for clinical education and training Qual Saf Health Care 13 Suppl 1 i11 i18 15465949 10.1136/qshc.2004.009886 (Pubitemid 39389373)
38. R Flin R Patey R Glavin N Maran 2010 Anaesthetists' non-technical skills Br J Anaesth 105 38 44 20522911 10.1093/bja/aeq134 1:STN:280: DC%2BC3crgs1ektA%3D%3D
39. LW Schuwirth L Southgate GG Page, et al. 2002 When enough is enough: a conceptual basis for fair and defensible practice performance assessment Med Educ 36 925 930 12390459 10.1046/j.1365-2923.2002.01313.x 1:STN:280: DC%2BD38nlvVWlug%3D%3D (Pubitemid 35286314)
40. SM Downing TM Haladyna 2004 Validity threats: overcoming interference with proposed interpretations of assessment data Med Educ 38 327 333 14996342 10.1046/j.1365-2923.2004.01777.x (Pubitemid 38375761)
41. V Wass C Van der Vleuten J Shatzer R Jones 2001 Assessment of clinical competence Lancet 357 945 949 11289364 10.1016/S0140-6736(00)04221-5 1:STN:280:DC%2BD3M7pvFGiug%3D%3D (Pubitemid 32247289)
42. RM Epstein EM Hundert 2002 Defining and assessing professional competence JAMA 287 226 235 11779266 10.1001/jama.287.2.226 (Pubitemid 34251949)
43. GF Dillon SG Clyman BE Clauser MJ Margolis 2002 The introduction of computer-based case simulations into the United States Medical Licensing Examination Acad Med 77 10 Suppl S94 S96 12377716 10.1097/00001888-200210001- 00029
44. DE Moore Jr FC Pennington 2003 Practice-based learning and improvement J Contin Educ Health Prof 23 Suppl 1 S73 S80 14666835 10.1002/chp.1340230411
45. DC Lynch SR Swing SD Horowitz K Holt JV Messer 2004 Assessing practice-based learning and improvement Teach Learn Med 16 85 92 14987181 10.1207/s15328015tlm1601-17 (Pubitemid 38292938)
46. C Carraccio R Englander 2004 Evaluating competence using a portfolio: a literature review and web-based application to the ACGME competencies Teach Learn Med 16 381 387 15582877 10.1207/s15328015tlm1604-13 (Pubitemid 39574076)
47. RM Harden 2006 Trends and the future of postgraduate medical education Emerg Med J 23 798 802 16988312 10.1136/emj.2005.033738 1:STN:280: DC%2BD28rmtF2msA%3D%3D
48. J Graham G Hocking E Giles 2010 Anaesthesia non-technical skills: can anaesthetists be trained to reliably use this behavioural marker system in 1 day? Br J Anaesth 104 440 445 20190257 10.1093/bja/aeq032 1:STN:280: DC%2BC3c7otVejsg%3D%3D
49. M Canas R Moreno A Rhodes RM Grounds 2010 Patient safety in anesthesia Minerva Anestesiol 76 753 757 20820154 1:STN:280:DC%2BC3cfgtlylsQ%3D%3D
50. RG Evans A Edwards S Evans B Elwyn G Elwyn 2007 Assessing the practising physician using patient surveys: a systematic review of instruments and feedback methods Fam Pract 24 117 127 17264071 10.1093/fampra/cml072 (Pubitemid 47079260)
51. SJ Lurie CJ Mooney JM Lyness 2009 Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: a systematic review Acad Med 84 301 309 19240434 10.1097/ACM.0b013e3181971f08
52. FD Duffy GH Gordon G Whelan, et al. 2004 Assessing competence in communication and interpersonal skills: the Kalamazoo II report Acad Med 79 495 507 15165967 10.1097/00001888-200406000-00002 (Pubitemid 38691475)
53. AB Symons A Swanson D McGuigan S Orrange EA Akl 2009 A tool for self-assessment of communication skills and professionalism in residents BMC Med Educ 9 1 19133146 10.1186/1472-6920-9-1
54. LM Ong JC de Haes AM Hoos FB Lammes 1995 Doctor-patient communication: a review of the literature Soc Sci Med 40 903 918 7792630 10.1016/0277-9536(94) 00155-M 1:STN:280:DyaK2Mzht1yquw%3D%3D
55. J Nicolai R Demmel 2007 The impact of gender stereotypes on the evaluation of general practitioners' communication skills: an experimental study using transcripts of physician-patient encounters Patient Educ Couns 69 200 205 17942269 10.1016/j.pec.2007.08.013 (Pubitemid 350051689)
56. JM Schirmer L Mauksch F Lang, et al. 2005 Assessing communication competence: a review of current tools Fam Med 37 184 192 15739134 (Pubitemid 40354005)
57. SJ Mercer MJ Moneypenny A Guha 2009 Communication and simulation for anaesthetists Anaesthesia 64 1259 1260 19825066 10.1111/j.1365-2044.2009.06113.x 1:STN:280:DC%2BD1MnovVynsQ%3D%3D
58. CS Park 2011 Simulation and quality improvement in anesthesiology Anesthesiol Clin 29 13 28 21295750 10.1016/j.anclin.2010.11.010
59. GP Whelan DW McKinley JR Boulet J Macrae S Kamholz 2001 Validation of the doctor-patient communication component of the Educational Commission for Foreign Medical Graduates Clinical Skills Assessment Med Educ 35 757 761 11489103 10.1046/j.1365-2923.2001.00977.x 1:STN:280:DC%2BD3MvksVSrtA%3D%3D (Pubitemid 32679326)
60. AF Smith JD Greaves 2010 Beyond competence: defining and promoting excellence in anaesthesia Anaesthesia 65 184 191 20003114 10.1111/j.1365-2044. 2009.06162.x 1:STN:280:DC%2BC3c3ntFOrsQ%3D%3D
61. DC Lynch PM Surdyk AR Eiser 2004 Assessing professionalism: a review of the literature Med Teach 26 366 373 15203852 10.1080/01421590410001696434 (Pubitemid 38961364)
62. TJ Wilkinson WB Wade LD Knock 2009 A blueprint to assess professionalism: results of a systematic review Acad Med 84 551 558 19704185 10.1097/ACM.0b013e31819fbaa2
63. JJ Veloski SK Fields JR Boex LL Blank 2005 Measuring professionalism: a review of studies with instruments reported in the literature between 1982 and 2002 Acad Med 80 366 370 15793022 10.1097/00001888-200504000-00014 (Pubitemid 40446176)
64. BD Hodges S Ginsburg R Cruess, et al. 2011 Assessment of professionalism: recommendations from the Ottawa 2010 Conference Med Teach 33 354 363 21517683 10.3109/0142159X.2011.577300
65. JJ Norcini LL Blank FD Duffy GS Fortna 2003 The mini-CEX: a method for assessing clinical skills Ann Intern Med 138 476 481 12639081 (Pubitemid 38129026)
66. M van Zanten JR Boulet JJ Norcini D McKinley 2005 Using a standardised patient assessment to measure professional attributes Med Educ 39 20 29 15612897 10.1111/j.1365-2929.2004.02029.x
67. JJ Norcini 2003 Peer assessment of competence Med Educ 37 539 543 12787377 10.1046/j.1365-2923.2003.01536.x (Pubitemid 36718243)
68. TL Massagli JD Carline 2007 Reliability of a 360-degree evaluation to assess resident competence Am J Phys Med Rehabil 86 845 852 17885319 10.1097/PHM.0b013e318151ff5a (Pubitemid 47476368)
69. MG Heffron D Simspon MS Kochar 2007 Competency-based physician education, recertification, and licensure WMJ 106 215 218 17844712 (Pubitemid 47328164)
70. ST Hingle S Robinson JA Colliver RB Rosher N McCann-Stone 2011 Systems-based practice assessed with a performance-based examination simulated and scored by standardized participants in the health care system: feasibility and psychometric properties Teach Learn Med 23 148 154 21516602 10.1080/10401334.2011.561751
71. DM Gaba 2010 Crisis resource management and teamwork training in anaesthesia Br J Anaesth 105 3 6 20551023 10.1093/bja/aeq124 1:STN:280:DC%2BC3crgslaltw%3D%3D
72. MC Wright BG Phillips-Bute ER Petrusa KL Griffin GW Hobbs JM Taekman 2009 Assessing teamwork in medical education and practice: relating behavioural teamwork ratings and clinical performance Med Teach 31 30 38 18825572 10.1080/01421590802070853
73. MS Volk J Ward N Irias A Navedo J Pollart PH Weinstock 2011 Using Medical Simulation to Teach Crisis Resource Management and Decision-Making Skills to Otolaryngology Housestaff Otolaryngol Head Neck Surg 145 35 42 21493304 10.1177/0194599811400833
74. PJ Morgan R Pittini G Regehr C Marrs MF Haley 2007 Evaluating teamwork in a simulated obstetric environment Anesthesiology 106 907 915 17457121 10.1097/01.anes.0000265149.94190.04 (Pubitemid 46684056)
75. Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, Third Edition (revised). Philadelphia, PA: National Board of Medical Examiners; 2010 .
76. AA Edler 2008 The use of simulation education in competency assessment: more questions than answers Anesthesiology 108 167 18156902 10.1097/01.anes. 0000296648.16232.28
77. JM Weller B Jolly B Robinson 2008 Generalisability of behavioural skills in simulated anaesthetic emergencies Anaesth Intensive Care 36 185 189 18361009 1:STN:280:DC%2BD1c3gtlWmtQ%3D%3D (Pubitemid 351449823)
78. B Hodges G Regehr N McNaughton R Tiberius M Hanson 1999 OSCE checklists do not capture increasing levels of expertise Acad Med 74 1129 1134 10536636 10.1097/00001888-199910000-00017 1:STN:280:DC%2BD3c%2Fgs1Oiug%3D%3D (Pubitemid 29493389)
79. B Hodges N McNaughton G Regehr R Tiberius M Hanson 2002 The challenge of creating new OSCE measures to capture the characteristics of expertise Med Educ 36 742 748 12191057 10.1046/j.1365-2923.2002.01203.x (Pubitemid 35001998)
80. BM Scavone MT Sproviero RJ McCarthy, et al. 2006 Development of an objective scoring system for measurement of resident performance on the human patient simulator Anesthesiology 105 260 266 16871059 10.1097/00000542- 200608000-00008 (Pubitemid 44148170)
81. AF De Champlain 2004 Ensuring that the competent are truly competent: an overview of common methods and procedures used to set standards on high-stakes examinations J Vet Med Educ 31 61 65 15962251 10.3138/jvme.31.1.61 (Pubitemid 38748209)
82. SM Downing A Tekian R Yudkowsky 2006 Procedures for establishing defensible absolute passing scores on performance examinations in health professions education Teach Learn Med 18 50 57 16354141 10.1207/ s15328015tlm1801-11 (Pubitemid 43440082)
83. JR Boulet D Murray J Kras J Woodhouse 2008 Setting performance standards for mannequin-based acute-care scenarios: an examinee-centered approach Simul Healthc 3 72 81 19088645 10.1097/SIH.0b013e31816e39e2 (Pubitemid 351725506)