The construct and criterion validity of the multi-source feedback process to assess physician performance: A meta-analysis

Advances in Medical Education and Practice

Volumn 5, Issue , 2014, Pages 39-51

  Ansari, Ahmed Al ^a   Donnon, Tyrone ^b   Khalifa, Khalid Al ^a   Darwish, Abdulla ^a   Violato, Claudio ^c  

^a BAHRAIN DEFENSE FORCE HOSPITAL   (Bahrain)

^b UNIVERSITY OF CALGARY   (Canada)

^c Ambrosiana University   (Italy)

Author keywords

Clinical performance; Construct validity; Criterion validity; Meta analysis; Multi source feedback system

Indexed keywords

EID: 84924289651     PISSN: None     EISSN: 11797258     Source Type: Journal    
DOI: 10.2147/AMEP.S57236     Document Type: Review

References (49)

1. Bracken DW, Timmreck CW, Church AH. Introduction: a multisource feedback process model. In: Bracken DW, Timmreck CW, Church AH, editors. The Handbook of Multisource Feedback: The Comprehensive Resource for Designing and Implementing MSF Processes. San Francisco, CA, USA; Jossey-Bass: 2001.
2. Lockyer J, Clyman S. Multisource feedback (360-degree evaluation). In: Holmboe ES, Hawkins RE, editors. Practical Guide to the Evaluation of Clinical Competence. Philadelphia, PA, USA; Mosby: 2008.
3. Hall W, Violato C, Lewkonia R, et al. Assessment of physician performance in Alberta: the Physician Achievement Review. CMAJ. 1999;161:52–57.
4. Fidler H, Lockyer J, Violato C. Changing physicians practice: the effect of individual feedback. Acad Med. 1999;74:702–714.
5. Violato C, Lockyer J, Fidler H. Multisource feedback: a method of assessing surgical practice. BMJ. 2003;326:546–548.
6. Miller A, Archer J. Impact of workplace based assessment on doctors education and performance: a systematic review. BMJ. 2010; 341:1–6.
7. Risucci DA, Tortolani AJ, Ward RJ. Ratings of surgical residents by self, supervisors and peers. Surg Gynecol Obstet. 1989;169:519–526.
8. Archer J, Norcini J, Southgate L, Heard S, Davies H. Mini-PAT (Peer Assessment Tool): a valid component of a national assessment programme in the UK? Adv Health Sci Educ Theory Pract. 2008;13:181–192.
9. Violato C, Lockyer J, Fidler H. Change in performance: a 5-year longitudinal study of participants in a multi-source feedback programme. Med Educ. 2008;42:1007–1013.
10. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;238:2008–2012.
11. Rosenthal R, Rubin D. A simple general purpose display of magnitude of experimental effect. J Educ Psychol. 1982;74:166–169.
12. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7:177–188.
13. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ, USA: Lawrence Earlbaum Associates; 1988.
14. Pollock RA, Donnelly MB, Plymale MA, Stewart DH, Vasconez HC. 360-degree evaluations of plastic surgery resident accreditation council for graduate medical education competencies: experience using a short form. Plast Reconstr Surg. 2008;122:639–649.
15. Archer J, McGraw M, Davies H. Assuring validity of multisource feedback in a national programme. Postgrad Med J. 2010;86:526–531.
16. Chandler N, Henderson G, Park B, Byerley J, Brown WD, Steiner MJ. Use of a 360-degree evaluation in the outpatient settings: the usefulness of nurse, faculty, patient/family, and resident self-evaluation. J Grad Med Educ. 2010;10:430–434.
17. Campbell J, Narayanan A, Burford B, Greco M. Validation of a multi-source feedback tool for use in general practice. Educ Prim Care. 2010;21:165–179.
18. Wood L, Wall D, Bullock A, Hassell A, Whitehouse A, Campbell I. ‘Team observation’: a six-year study of the development and use of multi-source feedback (360-degree assessment) in obstetrics and gyne-cology training in the UK. Med Teach. 2006;28:e177–e184.
19. Brinkman WB, Geraghty SR, Lanpher BP, et al. Effect of multisource feedback on resident communication skills and professionalism. Arch Pediatr Adolesc Med. 2007;161:44–49.
20. Archer JC, Norcini J, Davies HA. Use of SPRAT for peer review of paediatricians in training. BMJ. 2005;330:1251–1253.
21. Massagli TL, Carline JD. Reliability of a 360-degree evaluation to assess resident competence. Am J Phys Med Rehabil. 2007;86:845–852.
22. Lockyer J, Violato C, Fidler H. What multisource feedback factors influence physicians’ self-assessments? A five-year longitudinal study. Acad Med. 2007;82:77–80.
23. Qu B, Zhao YH, Sun BZ. Assessment of resident physicians in pro-fessionalism, interpersonal and communication skills: a multisource feedback. Int J Med Sci. 2012;9:228–236.
24. Yang YY, Lee FY, Hsu HC, et al. Assessment of first-year post-graduate residents: usefulness of multiple tools. J Chin Med Assoc. 2011;74:531–538.
25. Hatala R, Norman GR. In-training evaluation during an internal medicine clerkship. Acad Med. 1999;74(Suppl 10):118S–120S.
26. Hill F, Kendall K, Galbraith K, Crossley J. Implementing the undergraduate mini CEX: a tailored approach at Southampton University. Med Educ. 2009;43:326–334.
27. Wood J, Collins J, Burnside ES, et al. Patient, faculty, and self-assessment of radiology resident performance: a 360-degree method of measuring professionalism and interpersonal/communication skills. Acad Radiol. 2004;11:931–939.
28. Dominick P, Reilly R, McGourty J. The effects of peer feedback on team member Behaviour. Group and Organization Management. 1997;22:508–520.
29. Edwards M, Ewen A. 360 Feedback: The Powerful New Model for Employee Assessment and Performance Improvement. New York, NY, USA: Amacom; 1996.
30. Waldman D. Predictors of employee preferences for multi-rater and group-based performance appraisal. Group and Organization Management. 1997;22:264–287.
31. Violato C, Hecker K. How to use structural equation modeling in medical education research: a brief guide. Teach Learn Med. 2008;19: 362–371.
32. Violato C, Lockyer J, Fidler H. Assessment of psychiatrists in practice through multisource feedback. Can J Psychiatry. 2008;53:525–533.
33. Campbell JL, Richards SH, Dickens A, Greco M, Narayanan A, Brearley S. Assessing the professional performance of UK doctors: an evaluation of the utility of the General Medical Council patient and colleague questionnaires. Qual Saf Health Care. 2008;17: 187–193.
34. Meng L, Metro DG, Patel RM. Evaluating professionalism and interpersonal and communication skills: implementing a 360-degree evaluation instrument in an anesthesiology residency program. J Grad Med Educ. 2009;10:216–220.
35. Lockyer J, Violato C, Fidler H, Alakija P. The assessment of pathologists/ laboratory medicine physicians through a multisource feedback tool. Arch Pathol Lab Med. 2009;133:1301–1308.
36. Lockyer J, Violato C, Fidler H. The assessment of emergency physicians by a regulatory authority. Acad Emerg Med. 2006;13:1296–1303.
37. Archer J, McAvoy P. Factors that might undermine the validity of patient and multi-source feedback. Med Educ. 2011;45:886–893.
38. Overeem K, Wollersheim HC, Arah OA, Cruijsberg JK, Grol RP, Lombarts KM. Evaluation of physicians’ professional performance: an iterative development and validation study of multisource feedback instruments. BMC Health Serv Res. 2012;12:1–11.
39. Lockyer J, Violato C, Wright B, Fidler H, Chan R. Long-term outcomes for surgeons from 3-and 4-year medical school curricula. Can J Surg. 2012;55:1–5.
40. Davies H, Archer J, Bateman A, et al. Specialty-specific multi-source feedback: assuring validity, information training. Med Educ. 2008;42:1014–1020.
41. Wenrich MD, Carline JD, Giles LM, Ramsey PG. Ratings of the per-formances of practicing internists by hospital-based registered nurses. Acad Med. 1993;68:680–687.
42. Lelliott P, Williams R, Mears A, et al. Questionnaires for 360-degree assessment of consultant psychiatrists: development and psychometric properties. Br J Psychiatry. 2008;193:156–160.
43. Violato C, Marini A, Towes J, et al. Feasibility and psychometric properties of using peers, consulting physicians, co-workers, and patients to assess physicians. Acad Med. 1997;72:82–84.
44. Thomas PA, Gebo KA, Hellmann DB. A pilot study of peer review in residency training. J Gen Intern Med. 1999;14:551–554.
45. Lipner RS, Blank LL, Leas BF, Fortna GS. The value of patient and peer ratings in recertification. Acad Med. 2002;77 Suppl 10:64–66.
46. Joshi R, Ling FW, Jaeger J. Assessment of a 360-degree instrument to evaluate residents’ competency in interpersonal and communication skills. Acad Med. 2004;79:458–463.
47. Lockyer J, Violato C, Fidler H. A multi-source feedback program for anesthesiology. Can J Anesth. 2006;53:33–39.
48. Violato C, Lockyer J, Fidler H. Assessment of pediatricians by a regulatory authority. Pediatrics. 2006;117:796–802.
49. Lockyer JL, Blackmore D, Fidler H, et al. A study of a multi-source feedback system for international medical graduates holding defined licences. Med Educ. 2006;40:340–347.