Reconceptualizing variable rater assessments as both an educational and clinical care problem

Academic Medicine

Volumn 89, Issue 5, 2014, Pages 721-727

  Kogan, Jennifer R ^a   Conforti, Lisa N ^b   Iobst, William F ^c   Holmboe, Eric S ^d  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b AMERICAN BOARD OF INTERNAL MEDICINE   (United States)

^c Geisinger Health Geisinger Commonwealth School of Medicine   (United States)

^d Accreditation Council for Graduate Medical Education   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLINICAL COMPETENCE; CLINICAL MEDICINE; EDUCATION; EVALUATION STUDY; FEMALE; HEALTH CARE DELIVERY; HEALTH CARE QUALITY; HUMAN; MALE; MEDICAL EDUCATION; MEDICAL SCHOOL; OBSERVER VARIATION; ORGANIZATION AND MANAGEMENT; PATIENT CARE; REVIEW; UNITED STATES;

CLINICAL COMPETENCE; CLINICAL MEDICINE; DELIVERY OF HEALTH CARE; EDUCATION, MEDICAL; EDUCATIONAL MEASUREMENT; FACULTY, MEDICAL; FEMALE; HUMANS; MALE; OBSERVER VARIATION; PATIENT-CENTERED CARE; QUALITY ASSURANCE, HEALTH CARE; UNITED STATES;

EID: 84900513141     PISSN: 10402446     EISSN: 1938808X     Source Type: Journal    
DOI: 10.1097/ACM.0000000000000221     Document Type: Review

References (77)

1. Frank JR, Mungroo R, Ahmad Y, Wang M, De Rossi S, Horsley T. Toward a defnition of competency-based education in medicine: A systematic review of published defnitions. Med Teach. 2010;32:631-637.
2. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system-rationale and benefts. N Engl J Med. 2012;366:1051-1056.
3. Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. JAMA. 2009;302:1316-1326.
4. Pelgrim EA, Kramer AW, Mokkink HG, van den Elsen L, Grol RP, van der Vleuten C P. In-training assessment using direct observation of single-patient encounters: A literature review. Adv Health Sci Educ Theory Pract. 2011;16:131-142.
5. Norcini JJ. Current perspectives in assessment: The assessment of performance at work. Med Educ. 2005;39:880-889.
6. Gingerich A, Regehr G, Eva KW. Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Acad Med. 2011;86(10 suppl):S1-S7.
7. Govaerts MJ, van der Vleuten CP, Schuwirth LW, Muijtjens AM. Broadening perspectives on clinical performance assessment: Rethinking the nature of in-training assessment. Adv Health Sci Educ Theory Pract. 2007;12:239-260.
8. van der Vleuten CP, Schuwirth LW. Assessing professional competence: From methods to programmes. Med Educ. 2005;39:309-317.
9. Ramani S, Leinster S. AMEE guide no. 34: Teaching in the clinical environment. Med Teach. 2008;30:347-364.
10. Kilminster SM, Jolly BC. Effective supervision in clinical practice settings: A literature review. Med Educ. 2000;34:827-840.
11. Rowe JW, chair; Institute of Medicine Committee on the Future Health Care Workforce for Older Americans. Retooling for an Aging America: Building the Health Care Workforce. Washington, DC: National Academies Press; 2008.
12. Pelgrim EA, Kramer AW, Mokkink HG, van der Vleuten CP. The process of feedback in workplace-based assessment: Organisation, delivery, continuity. Med Educ. 2012;46: 604-612.
13. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32:676-682.
14. Albanese MA. Challenges in using rater judgements in medical education. J Eval Clin Pract. 2000;6:305-319.
15. Williams RG, Klamen DA, McGaghie WC. Cognitive, social and environmental sources of bias in clinical performance ratings. Teach Learn Med. 2003;15:270-292.
16. Alves de Lima A, Conde D, Costabel J, Corso J, van der Vleuten C. A laboratory study on the reliability estimations of the mini-CEX. Adv Health Sci Educ Theory Pract. 2013;18:5-13.
17. Margolis MJ, Clauser BE, Cuddy MM, et al. Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: a validity study. Acad Med. 2006;81(10 suppl):S56-S60.
18. Wilkinson JR, Crossley JG, Wragg A, Mills P, Cowan G, Wade W. Implementing workplace-based assessment across the medical specialties in the United Kingdom. Med Educ. 2008;42:364-373.
19. Weller JM, Jolly B, Misur MP, et al. Mini-clinical evaluation exercise in anaesthesia training. Br J Anaesth. 2009;102:633-641.
20. Govaerts MJ, Schuwirth LW, Van der Vleuten C P, Muijtjens AM. Workplace-based assessment: Effects of rater expertise. Adv Health Sci Educ Theory Pract. 2011;16:151-165.
21. Yeates P, O'Neill P, Mann K, Eva K. Seeing the same thing differently: Mechanisms that contribute to assessor differences in directly-observed performance assessments. Adv Health Sci Educ Theory Pract. 2013;18:325-341.
22. Yeates P, O'Neill P, Mann K, Eva KW. Effect of exposure to good vs poor medical trainee performance on attending physician ratings of subsequent performances. JAMA. 2012;308:2226-2232.
23. Holmboe ES, Huot S, Chung J, Norcini J, Hawkins RE. Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. 2003;78:826-830.
24. Schuh LA, London Z, Neel R, et al. Education research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination. Neurology. 2009;73:904-908.
25. Kogan JR, Conforti L, Bernabeo E, Iobst W, Holmboe E. Opening the black box of clinical skills assessment via observation: A conceptual model. Med Educ. 2011;45:1048-1060.
26. Kogan JR, Hess BJ, Conforti LN, Holmboe ES. What drives faculty ratings of residents' clinical skills? The impact of faculty's own clinical skills. Acad Med. 2010;85(10 suppl):S25-S28.
27. Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA. 1993;269:1655-1660.
28. Paauw DS, Wenrich MD, Curtis JR, Carline JD, Ramsey PG. Ability of primary care physicians to recognize physical fndings associated with HIV infection. JAMA. 1995;274:1380-1382.
29. Vukanovic-Criley JM, Criley S, Warde CM, et al. Competency in cardiac examination skills in medical students, trainees, physicians, and faculty: A multicenter study. Arch Intern Med. 2006;166:610-616
30. Braddock CH 3rd, Fihn SD, Levinson W, Jonsen AR, Pearlman RA. How doctors and patients discuss routine clinical decisions. Informed decision making in the outpatient setting. J Gen Intern Med. 1997;12:339-345.
31. Levinson W. Patient-centred communication: A sophisticated procedure. BMJ Qual Saf. 2011;20:823-825.
32. Frankel RM, Eddins-Folensbee F, Inui TS. Crossing the patient-centered divide: Transforming health care quality through enhanced faculty development. Acad Med. 2011;86:445-452.
33. Donato AA, Pangaro L, Smith C, et al. Evaluation of a novel assessment form for observing medical residents: A randomised, controlled trial. Med Educ. 2008;42:1234-1242.
34. Cook DA, Beckman TJ. Does scale length matter? A comparison of nine-versus fve-point rating scales for the mini-CEX. Adv Health Sci Educ Theory Pract. 2009;14:655-664.
35. Holmboe ES, Hawkins RE, Huot SJ. Effects of training in direct observation of medical residents' clinical competence: A randomized trial. Ann Intern Med. 2004;140:874-881.
36. Cook DA, Dupras DM, Beckman TJ, Thomas KG, Pankratz VS. Effect of rater training on reliability and accuracy of mini-CEX scores: A randomized, controlled trial. J Gen Intern Med. 2009;24:74-79.
37. Moonen-van Loon JM, Overeem K, Donkers HH, van der Vleuten CP, Driessen EW. Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Adv Health Sci Educ Theory Pract. 2013;18:1087-1102
38. Braddock CH 3rd, Edwards KA, Hasenberg NM, Laidley TL, Levinson W. Informed decision making in outpatient practice: Time to get back to basics. JAMA. 1999;282:2313-2320.
39. Kennedy TJ, Regehr G, Baker GR, Lingard L. Point-of-care assessment of medical trainee competence for independent clinical work. Acad Med. 2008;83(10 suppl):S89-S92.
40. Kennedy TJ, Lingard L, Baker GR, Kitchen L, Regehr G. Clinical oversight: Conceptualizing the relationship between supervision and safety. J Gen Intern Med. 2007;22:1080-1085.
41. Eva KW, Regehr G. Knowing when to look it up: A new conception of self-assessment ability. Acad Med. 2007;82(10 suppl):S81-S84.
42. Sargeant J, Armson H, Chesluk B, et al. The processes and dimensions of informed self-assessment: A conceptual model. Acad Med. 2010;85:1212-1220.
43. Sklar D P, Lee R. Commentary: What if high-quality care drove medical education? A multiattribute approach. Acad Med. 2010;85:1401-1404.
44. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academies Press; 2001.
45. Asch DA, Nicholson S, Srinivas S, Herrin J, Epstein AJ. Evaluating obstetrical residency programs using patient outcomes. JAMA. 2009;302:1277-1283.
46. Murray CJ, Frenk J. Health metrics and evaluation: Strengthening the science. Lancet. 2008;371:1191-1199.
47. Dudek NL, Marks MB, Regehr G. Failure to fail: The perspectives of clinical supervisors. Acad Med. 2005;80(10 suppl):S84-S87.
48. Weiner SJ, Schwartz A, Sharma G, et al. Patient-centered decision making and health care outcomes: An observational study. Ann Intern Med. 2013;158:573-579.
49. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: Theory to practice. Med Teach. 2010;32: 638-645.
50. Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): A tool to assess surgical competence. Acad Med. 2012;87:1401-1407.
51. Crossley J, Jolly B. Making sense of work-based assessment: Ask the right questions, in the right way, about the right things, of the right people. Med Educ. 2012;46:28-37.
52. Alexander EK, Osman NY, Walling JL, Mitchell VG. Variation and imprecision of clerkship grading in U.S. medical schools. Acad Med. 2012;87:1070-1076.
53. Carraccio CL, Benson BJ, Nixon LJ, Derstine PL. From the educational bench to the clinical bedside: Translating the Dreyfus developmental model to the learning of clinical skills. Acad Med. 2008;83:761-767.
54. Van Der Vleuten CP. The assessment of professional competence: Developments, research and practical implications. Adv Health Sci Educ Theory Pract. 1996;1:41-67.
55. Chandratilake MN, Davis MH, Ponnamperuma G. Evaluating and designing assessments for medical education: The utility formula. Internet J Med Educ. 2010;1(1). http://ispub.com/IJME/1/1/10810. Accessed January 23, 2014.
56. Norcini J, Anderson B, Bollela V, et al. Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 conference. Med Teach. 2011;33:206-214.
57. Shepard L. The role of assessment in a learning culture. Educ Res. 2000;29:4-14.
58. Ten Cate O, Snell L, Carraccio C. Medical competence: The interplay between individual ability and the health care environment. Med Teach. 2010;32:669-675.
59. Hodges BD. A tea-steeping or i-Doc model for medical education? Acad Med. 2010;85(9 suppl):S34-S44.
60. Sterkenburg A, Barach P, Kalkman C, Gielen M, ten Cate O. When do supervising physicians decide to entrust residents with unsupervised tasks? Acad Med. 2010;85:1408-1417.
61. U.S. Offce of Compliance. Policy DC-323. Medicare's primary care exception. http://ooc.usc.edu/policy-dc-323-medicare%E2%80%99s-primary-care- exception. Accessed January 14, 2014.
62. Green ML, Aagaard EM, Caverzagie KJ, et al. Charting the road to competence: Developmental milestones for internal medicine residency training. J Grad Med Educ. 2009;1:5-20.
63. Dijksterhuis MG, Voorhuis M, Teunissen PW, et al. Assessment of competence and progressive independence in postgraduate clinical training. Med Educ. 2009;43: 1156-1165.
64. Kennedy TJ. Towards a tighter link between supervision and trainee ability. Med Educ. 2009;43:1126-1128.
65. Lurie SJ. History and practice of competency-based assessment. Med Educ. 2012;46:49-57.
66. Lypson ML, Frohna JG, Gruppen LD, Woolliscroft JO. Assessing residents' competencies at baseline: Identifying the gaps. Acad Med. 2004;79:564-570.
67. Sachdeva AK, Loiacono LA, Amiel GE, Blair PG, Friedman M, Roslyn JJ. Variability in the clinical skills of residents entering training programs in surgery. Surgery. 1995;118: 300-308.
68. Bernabeo EC, Holtman MC, Ginsburg S, Rosenbaum JR, Holmboe ES. Lost in transition: The experience and impact of frequent changes in the inpatient learning environment. Acad Med. 2011;86:591-598.
69. Moulton C, Epstein RM. Self-monitoring in surgical practice: Slowing down when you should. Adv Med Educ. 2011;2:169-182.
70. Lyles JS, Dwamena FC, Lein C, Smith RC. Evidence-based patient-centered interviewing. J Clin Outcomes Manage. 2001;8:28-34.
71. Ptacek JT, Eberhardt TL. Breaking bad news. A review of the literature. JAMA. 1996;276:496-502.
72. Searight R. Realistic approaches to counseling in the offce setting. Am Fam Physician. 2009;79:277-284.
73. McGee S. Evidence-Based Physical Diagnosis. St. Louis, Mo: Saunders Elsevier; 2007.
74. Makoul G, Curry RH, Tang PC. The use of electronic medical records: Communication patterns in outpatient encounters. J Am Med Inform Assoc. 2001;8:610-615.
75. Elsawy B, Higgins KE. The geriatric assessment. Am Fam Physician. 2011; 83:48-56.
76. Quinn TJ, McArthur K, Ellis G, Stott DJ. Functional assessment in older people. BMJ. 2011;343:d4681.
77. Haan CK, Edwards FH, Poole B, Godley M, Genuardi FJ, Zenni EA. A model to begin to use clinical outcomes in medical education. Acad Med. 2008;83:574-580.