A BEME (Best Evidence in Medical Education) review of the use of workplace-based assessment in identifying and remediating underperformance among postgraduate medical trainees: BEME Guide No. 43

Medical Teacher

Volumn 38, Issue 12, 2016, Pages 1188-1198

  Barrett, Aileen ^a,b   Galvin, Rose ^c   Steinert, Yvonne ^d   Scherpbier, Albert ^e   O’Shaughnessy, Ann ^a   Horgan, Mary ^b   Horsley, Tanya ^f,g  

^a ROYAL COLLEGE OF PHYSICIANS OF IRELAND   (Ireland)

^b UNIVERSITY COLLEGE CORK   (Ireland)

^c UNIVERSITY OF LIMERICK   (Ireland)

^d MCGILL UNIVERSITY   (Canada)

^e MAASTRICHT UNIVERSITY   (Netherlands)

^f ROYAL COLLEGE OF PHYSICIANS AND SURGEONS OF CANADA   (Canada)

^g UNIVERSITY OF OTTAWA   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

CLINICAL EVALUATION; CONSENSUS; DATA BASE; DATA EXTRACTION; DECISION MAKING; EXERCISE; HUMAN; HUMAN EXPERIMENT; MEDICAL EDUCATION; POSTGRADUATE STUDENT; PUBLICATION; SCREENING; SYNTHESIS; WORKPLACE; CLINICAL COMPETENCE; CONSTRUCTIVE FEEDBACK; EDUCATION; OBSERVER VARIATION; PROCEDURES; STANDARDS;

CLINICAL COMPETENCE; EDUCATIONAL MEASUREMENT; FORMATIVE FEEDBACK; HUMANS; INTERNSHIP AND RESIDENCY; OBSERVER VARIATION; WORKPLACE;

EID: 84987625499     PISSN: 0142159X     EISSN: 1466187X     Source Type: Journal    
DOI: 10.1080/0142159X.2016.1215413     Document Type: Article

References (64)

1. Alves de Lima A, Conde D, Costabel J, Corso J, Vleuten C., 2013. A laboratory study on the reliability estimations of the mini-CEX. Adv Health Sci Educ Theory Pract. 18:5–13.
2. Archer JC, McAvoy P., 2011. Factors that might undermine the validity of patient and multi-source feedback. Med Educ. 45:886–893.
3. Barr H, D, Freeth M, Hammick I., Koppel S, Reeves. 2000. Evaluations of interprofessional education; a United Kingdom review of health and social care. London:CAIPE/BERA.
4. Barrett A, Galvin R, Steinert Y, Scherpbier A, O’Shaughnessy A, Horgan M, Horsley T., 2015. A BEME (Best Evidence in Medical Education) systematic review of the use of workplace-based assessment in identifying and remediating poor performance among postgraduate medical trainees. Syst Rev. 4:1–6.
5. Black D, Welch J., 2009. The under-performing trainee–concerns and challenges for medical educators. ClinTeach. 6:79–82.
6. Bok H, Teunissen P, Favier R, Rietbroek N, Theyse L, Brommer H, Haarhuis J, van Beukelen P, van der Vleuten C, Jaarsma D., 2013. Programmatic assessment of competency-based workplace learning:when theory meets practice. BMC Med Educ.13:123.
7. Booth A., 2006. “Brimful of STARLITE”:toward standards for reporting literature searches. J Med Libr Assoc. 94:421–e205.
8. Brown DJ, Thompson RE, Bhatti NI., 2008. Assessment of operative competency in otolaryngology residency:survey of US program directors. Laryngoscope. 118:1761–1764.
9. Buckley S, Coleman J, Davison I, Khan KS, Zamora J, Malick S, Morley D, Pollard D, Ashcroft T, Popovic C, et al. 2009. The educational effects of portfolios on undergraduate student learning:a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11. Med Teach. 31:282–298.
10. Bullock AD, Hassell A, Markham WA, Wall DW, Whitehouse AB., 2009. How ratings vary by staff group in multi-source feedback assessment of junior doctors. Med Educ. 43:516–520.
11. Burford B, Illing J, Kergon C, Morrow G, Livingston M., 2010. User perceptions of multi-source feedback tools for junior doctors. Med Educ. 44:165–176.
12. CASP:Critical Appraisal Skills Programme [Internet]. 2013. [cited 2016 Jan 14]. Available from:http://www.casp-uk.net/#!checklists/cb36.
13. Chipp E, Srinivasan K, Adil Abbas Khan M, Rayatt S., 2011. Incorporating multi-source feedback into a new clinically based revision course for the FRCS (Plast) exam. Med Teach. 33:e263–e266.
14. Cleland J, Mackenzie RK, Ross S, Sinclair HK, Lee AJ., 2010. A remedial intervention linked to a formative assessment is effective in terms of improving student performance in subsequent degree examinations. Med Teach. 32:e185–e190.
15. Cook D, Zendejas B, Hamstra S, Hatala R, Brydges R., 2014. What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ. 19:233–250.
16. Cook DA, Beckman TJ., 2008. Does scale length matter? A comparison of nine- versus five-point rating scales for the mini-CEX. Adv Health Sci Educ Theory Pract. 14:655–664.
17. Cook DA, Brydges R, Ginsburg S, Hatala R., 2015. A contemporary approach to validity arguments:a practical guide to Kane’s framework. Med Educ. 49:560–575.
18. De Houwer J, Barnes-Holmes D, Moors A., 2013. What is learning? On the nature and merits of a functional definition of learning. Psychon Bull Rev. 20:631–642.
19. Donato AA, Park YS, George DL, Schwartz A, Yudkowsky R., 2015. Validity and feasibility of the Minicard Direct Observation Tool in 1 Training Program. J Grad Med Educ. 7:225–229.
20. Dudek NL, Marks MB, Regehr G., 2005. Failure to fail:the perspectives of clinical supervisors. Acad Med. 80:S84–S87.
21. Ferguson J, Wakeling J, Bowie P., 2014. Factors influencing the effectiveness of multisource feedback in improving the professional practice of medical doctors:a systematic review. BMC Med Educ. 14:76–76.
22. Gingerich A, Regehr G, Eva KW., 2011. Rater-based assessments as social judgments:rethinking the etiology of rater errors. Acad Med. 86:S1–S7.
23. Gingerich A, van der Vleuten CPM, Eva KW, Regehr G., 2014. More consensus than idiosyncrasy:categorizing social judgments to examine variability in mini-CEX ratings. Acad Med. 89:1510–1519.
24. Govaerts M., 2015. Workplace-based assessment and assessment for learning:threats to validity. J Grad Med Educ. 7:265–267.
25. Govaerts M, Schuwirth L, Van der Vleuten C, Muijtjens A., 2011. Workplace-based assessment:effects of rater expertise. Adv Health Sci Educ Theory Pract. 16:151–165.
26. Hassell A, Bullock A, Whitehouse A, Wood L, Jones P, Wall D., 2012. Effect of rating scales on scores given to junior doctors in multi-source feedback. Postgrad Med J. 88:10–14.
27. Hatala R, Cook D, Brydges R, Hawkins R., 2015. Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS):a systematic review of validity evidence. Adv Health Sci Educ:20:1149–1175.
28. Hawkins RE, Margolis MJ, Durning SJ, Norcini JJ., 2010. Constructing a validity argument for the mini-clinical evaluation exercise:a review of the research. Acad Med. 85:1453–1461.
29. Hesketh EA, Anderson F, Bagnall GM, Driver CP, Johnston DA, Marshall D, Needham G, Orr G, Walker K., 2005. Using a 360 degrees diagnostic screening tool to provide an evidence trail of junior doctor performance throughout their first postgraduate year. Med Teach. 27:219–233.
30. Hiemstra E, Kolkman W, Wolterbeek R, Trimbos B, Jansen FW., 2011. Value of an objective assessment tool in the operating room. Can J Surg. 54:116–122.
31. Kogan JR, Conforti LN, Bernabeo EC, Durning SJ, Hauer KE, Holmboe ES., 2012. Faculty staff perceptions of feedback to residents after direct observation of clinical skills. Med Educ. 46:201–215.
32. Kogan JR, Holmboe ES, Hauer KE., 2009. Tools for direct observation and assessment of clinical skills of medical trainees:a systematic review. JAMA:JAm Med Assoc. 302:1316–1326.
33. Lo B., 2012. Informed consent. Ethical issues in clinical research:a practical guide. Philadelphia, PA:Lippincott Williams & Wilkins.
34. McHugh M., 2012. Interrater reliability:the kappa statistic. Biochem Med (Zagreb). 22:276–282.
35. Menon S, Winston M, Sullivan G., 2012. Workplace based assessment:attitudes and perceptions among consultant trainers and comparison with those of trainees. Psychiatrist. 36:16–24.
36. Miller A, Archer J., 2010. Impact of workplace based assessment on doctors’ education and performance:a systematic review. Bmj. 341:c5064.
37. Mitchell C, Bhat S, Herbert A, Baker P., 2011. Workplace-based assessments of junior doctors:do scores predict training difficulties? Med Educ. 45:1190–1198.
38. Mitchell C, Bhat S, Herbert A, Baker P., 2013. Workplace-based assessments in Foundation Programme training:do trainees in difficulty use them differently? Med Educ. 47:292–300.
39. Moher D, Liberati A, Tetzlaff J, Altman DG., 2009. Preferred reporting items for systematic reviews and meta-analyses:the PRISMA statement. AnnIntern Med. 151:264–269.
40. Norcini J, Burch V., 2007. Workplace-based assessment as an educational tool:AMEE Guide No. 31. Med Teach. 29:855–871.
41. Norcini JJ, Blank LL, Arnold GK, Kimball HR., 1995. The mini-CEX (Clinical Evaluation Exercise):a preliminary investigation. Ann InternMed. 123:795–799.
42. Overeem K, Wollersheim H, Driessen E, Lombarts K, Van De Ven G, Grol R, Arah O., 2009. Doctors’ perceptions of why 360-degree feedback does (not) work:a qualitative study. Med Educ. 43:874–882.
43. Pelgrim EAM, Kramer AWM, Mokkink HGA, den Elsen L, Grol RPTM, Vleuten CPM., 2011. In-training assessment using direct observation of single-patient encounters:a literature review. Adv Health Sci Educ. 16:131–142.
44. Rees CE, Cleland JA, Dennis A, Kelly N, Mattick K, Monrouxe LV., 2014. Supervised learning events in the Foundation Programme:a UK-wide narrative interview study. BMJ Open. 4:e005980.
45. Saedon H, Salleh S, Balakrishnan A, Imray C, Saedon M., 2012. The role of feedback in improving the effectiveness of workplace based assessments:a systematic review. BMC Med Educ. 12:25.
46. Sargeant J, McNaughton E, Mercer S, Murphy D, Sullivan P, Bruce DA., 2011. Providing feedback:exploring a model (emotion, content, outcomes) for facilitating multisource feedback. Med Teach. 33:744–749.
47. St-Onge C, Young M., 2015. Evolving conceptualisations of validity:impact on the process and outcome of assessment. Med Educ. 49:548–550.
48. Steinert Y, Mann K, Centeno A, Dolmans D, Spencer J, Gelula M, Prideaux D., 2006. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education:BEME Guide No. 8. Med Teach. 28:497–526.
49. Tong A, Palmer S, Craig JC, Strippoli GFM., 2014. A guide to reading and using systematic reviews of qualitative research. Nephrol Dial Transpl. doi:10.1093/ndt/gfu354.
50. Tong A, Sainsbury P, Craig J., 2007. Consolidated criteria for reporting qualitative research (COREQ):a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 19:349–357.
51. Vivekananda-Schmidt P, MacKillop L, Crossley J, Wade W., 2013. Do assessor comments on a multi-source feedback instrument provide learner-centred feedback? Med Educ.47:1080–1088.
52. Warm EJ, Schauer D, Revis B, Boex JR., 2010. Multisource feedback in the ambulatory setting. J Grad Med Educ. 2:269–277.
53. Watling C., 2014. Cognition, culture, and credibility:deconstructing feedback in medical education. Perspect Med Educ. 3:124–128.
54. Watling C, Driessen E, van der Vleuten CPM, Lingard L., 2012. Learning from clinical work:the roles of learning cues and credibility judgements. Med Educ. 46:192–200.
55. Watling C, Driessen E, van der Vleuten CPM, Vanstone M, Lingard L., 2012. Understanding responses to feedback:the potential and limitations of regulatory focus theory. Med Educ. 46:593–603.
56. Watling C, Driessen E, van der Vleuten CPM, Vanstone M, Lingard L., 2013a. Beyond individualism:professional culture and its influence on feedback. Med Educ. 47:585–594.
57. Watling C, Driessen E, van der Vleuten CPM, Vanstone M, Lingard L., 2013b. Music lessons:revealing medicine's learning culture through a comparison with that of music. Med Educ. 47:842–850.
58. Weller JM, Jones A, Merry AF, Jolly B, Saunders D., 2009. Investigation of trainee and specialist reactions to the mini-clinical evaluation exercise in anaesthesia:implications for implementation. BJA:British J Anaesth. 103:524–530.
59. Whitehouse A, Hassell A, Bullock A, Wood L, Wall D., 2007. 360 degree assessment (multisource feedback) of UK trainee doctors:field testing of team assessment of behaviours (TAB). Med Teach. 29:171–176.
60. Wood L, Wall D, Bullock A, Hassell A, Whitehouse A, Campbell I. 2006. “Team observation”:a six-year study of the development and use of multi-source feedback (360-degree assessment) in obstetrics and gynaecology training in the UK. Med Teach. 28:e177–e184.
61. Yao DC, Wright SM., 2000. National survey of internal medicine residency program directors regarding problem residents. JAMA. 284:1099–1104.
62. Yeates P, O'Neill P, Mann K, Eva KW., 2013a. Effect of exposure to good vs poor medical trainee performance on attending physician ratings of subsequent performances. JAMA. 30:2226–2232. [Erratum appears in JAMA. 2013 Jan 16;309(3):237].
63. Yeates P, O’Neill P, Mann K, Eva K., 2013b. Seeing the same thing differently:mechanisms that contribute to assessor differences in directly-observed performance assessments. Adv Health Sci Educ Theory Pract. 18:325–341.
64. Young JQ, Lieu S, O'Sullivan P, Tong L., 2011. Development and initial testing of a structured clinical observation tool to assess pharmacotherapy competence. Acad Psychiatr. 35(1):27–34.