Seeing the 'black box' differently: Assessor cognition from three research perspectives

Medical Education

Volumn 48, Issue 11, 2014, Pages 1055-1068

  Gingerich, Andrea ^a   Kogan, Jennifer ^b   Yeates, Peter ^c   Govaerts, Marjan ^d   Holmboe, Eric ^e  

^a UNIVERSITY OF NORTHERN BRITISH COLUMBIA   (Canada)

^b UNIVERSITY OF PENNSYLVANIA   (United States)

^c UNIVERSITY OF MANCHESTER   (United Kingdom)

^d MAASTRICHT UNIVERSITY   (Netherlands)

^e ACCREDITATION COUNCIL FOR GRADUATE MEDICAL EDUCATION   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COGNITION; EDUCATION; HUMAN; MEDICAL EDUCATION; RESEARCH; STANDARDS;

COGNITION; EDUCATION, MEDICAL; EDUCATIONAL MEASUREMENT; HUMANS; RESEARCH;

EID: 84907879328     PISSN: 03080110     EISSN: 13652923     Source Type: Journal    
DOI: 10.1111/medu.12546     Document Type: Article

References (127)

1. McGaghie WC, Lipson L. Competency-based Curriculum Development in Medical Education: An Introduction. Geneva: World Health Organization 1978.
2. Frenk J, Chen L, Bhutta ZA et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet 2010;376 (9756):1923-58.
3. Hodges BD. A tea-steeping or i-doc model for medical education? Acad Med 2010;85 (9) (Suppl):34-44.
4. Irby DM, Cooke M, O'Brien BC. Calls for reform of medical education by the Carnegie Foundation for the advancement of teaching: 1910 and 2010. Acad Med 2010;85 (2):220-7.
5. van der Vleuten CPM, Schuwirth LWT. Assessing professional competence: from methods to programmes. Med Educ 2005;39:309-17.
6. ten Cate TJO, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach 2010;32 (8):669-75.
7. Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: a systematic review. Acad Med 2009;84 (3):301-9.
8. Albanese M. Rating education quality: factors in the erosion of professional standards. Acad Med 1999;74 (6):652-8.
9. Williams RG, Klamen DA, McGaghie WC. Cognitive, social and environmental sources of bias in clinical performance ratings. Teach Learn Med 2003;15 (4):270-92.
10. Elliot DL, Hickam DH. Evaluation of physical examination skills. Reliability of faculty observers and patient instructors. JAMA 1987;258 (23):3405-8.
11. Noel GL, Herbers JE, Caplow MP, Cooper GS, Pangaro LN, Harvey J. How well do internal medicine faculty members evaluate the clinical skills of residents? Ann Intern Med 1992;117 (9):757-65.
12. Herbers JE Jr, Noel GL, Cooper GS, Harvey J, Pangaro LN, Weaver MJ. How accurate are faculty evaluations of clinical competence? J Gen Intern Med 1989;4 (3):202-8.
13. Cook DA, Beckman TJ, Mandrekar JN, Pankratz VS. Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalisability. Adv Health Sci Educ Theory Pract 2010;15 (5):633-45.
14. Margolis MJ, Clauser BE, Cuddy MM, Ciccone A, Mee J, Harik P, Hawkins RE. Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: a validity study. Acad Med 2006;81 (10 Suppl):56-60.
15. Hill F, Kendall K, Galbraith K, Crossley J. Implementing the undergraduate mini-CEX: a tailored approach at Southampton University. Med Educ 2009;43:326-34.
16. Tyler R. Basic Principles of Curriculum and Instruction. Chicago, IL: University of Chicago Press 1949.
17. Ertmer PA, Newby TJ. Behaviourism, cognitivism, constructivism: comparing critical features from a design perspective. Perform Improve Q 1993;6 (4):50-72.
18. Saettler P. The Evolution of American Educational Technology. Englewood, CO: Libraries Unlimited 1990.
19. Smith P, Ragan T. Instructional Design, 2nd edn. New York, NY: John Wiley & Sons 1999.
20. Torre DM, Daley BJ, Sebastian JL, Elnicki DM. Overview of current learning theories for medical educators. Am J Med 2006;119:903-7.
21. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: Institute of Medicine 2001.
22. World Health Organization. Quality of Care: A Process for Making Strategic Choices in Health Systems. 2006. http://www.who.int/management/quality/assurance/QualityCare_B.Def.pdf [Accessed 22 November 2013.]
23. Lyles JS, Dwamena FC, Lein C, Smith RC. Evidence-based patient-centred interviewing. J Clin Outcomes Manage 2001;8 (7):28-34.
24. Braddock CH III, Edwards KA, Hasenberg NM, Laidley TL, Levinson W. Informed decision making in outpatient practice: time to get back to basics. JAMA 1999;282 (24):2313-20.
25. Murray CJ, Frenk J. Health metrics and evaluation: strengthening the science. Lancet 2008;371 (9619):1191-9.
26. Ptacek JT, Eberhardt TL. Breaking bad news. A review of the literature. JAMA 1996;276 (6):496-502.
27. Searight R. Realistic approaches to counselling in the office setting. Am Fam Physician 2009;79 (4):277-84.
28. McGee S. Evidence-Based Physical Diagnosis. St Louis, MO: Saunders Elsevier 2007.
29. Sklar DP, Lee R. Commentary: what if high-quality care drove medical education? A multiattribute approach Acad Med 2010;85 (9):1401-4.
30. Asch DA, Nicholson S, Srinivas S, Herrin J, Epstein AJ. Evaluating obstetrical residency programmes using patient outcomes. JAMA 2009;302 (12):1277-83.
31. Haan CK, Edwards FH, Poole B, Godley M, Genuardi FJ, Zenni EA. A model to begin to use clinical outcomes in medical education. Acad Med 2008;83 (6):574-80.
32. Kogan JR, Conforti L, Bernabeo E, Iobst W, Holmboe E. Opening the black box of clinical skills assessment via observation: a conceptual model. Med Educ 2011;45:1048-60.
33. Yeates P, O'Neill P, Mann K, Eva K. Seeing the same thing differently: mechanisms that contribute to assessor differences in directly observed performance assessments. Adv Health Sci Educ Theory Pract 2013a;18 (3):325-41.
34. Yeates P, O'Neill P, Mann K, Eva KW. Effect of exposure to good vs poor medical trainee performance on attending physician ratings of subsequent performances. JAMA 2012;308 (21):2226-32.
35. Yeates P, O'Neill P, Mann K, Eva KW. 'You're certainly relatively competent': assessor bias due to recent experiences. Med Educ 2013;47:910-22.
36. Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA 2009;302 (12):1316-26.
37. Kogan JR, Hess BJ, Conforti LN, Holmboe ES. What drives faculty ratings of residents' clinical skills? The impact of faculty's own clinical skills. Acad Med 2010;85 (10 Suppl):25-8.
38. Vukanovic-Criley JM, Criley S, Warde CM, Broker JR, Guevara-Mathews L, Churchill WH, Nelson WP, Criley JM. Competency in cardiac examination skills in medical students, trainees, physicians, and faculty: a multicentre study. Arch Intern Med 2006;166 (6):610-6.
39. Paauw DS, Wenrich MD, Curtis JR, Carline JD, Ramsey PG. Ability of primary care physicians to recognise physical findings associated with HIV infection. JAMA 1995;274 (17):1380-2.
40. Ramsey PG, Wenrich MD. Use of peer ratings to evaluate physician performance. JAMA 1993;269 (13):1655-60.
41. Braddock CH III, Fihn SD, Levinson W, Jonsen AR, Pearlman RA. How doctors and patients discuss routine clinical decisions. Informed decision making in the outpatient setting. J Gen Intern Med 1997;12 (6):339-45.
42. Govaerts MJB, van de Wiel MWJ, van der Vleuten CPM. Quality of feedback following performance assessments: does assessor expertise matter? Eur J Train Dev 2013a;37 (1):105-25.
43. Govaerts MJB, Schuwirth L, van der Vleuten CP, Muijtjens AMM. Workplace-based assessment: effects of rater expertise. Adv Health Sci Educ Theory Pract 2011;16 (2):151-65.
44. Dudek NL, Marks MB, Regehr G. Failure to fail: the perspectives of clinical supervisors. Acad Med 2005;80 (10) (Suppl):84-7.
45. Cleland JA, Knight LV, Rees CE, Tracey S, Bond CM. Is it me or is it them? Factors that influence the passing of underperforming students. Med Educ 2008;42:800-9.
46. Nasca TJ, Brigham T, Philibert I, Flynn TC. The next GME accreditation system - rationale and benefits. New Engl J Med 2012;366 (11):1051-6.
47. Carraccio CL, Englander R. From Flexner to competencies: reflections on a decade and the journey ahead. Acad Med 2013;88 (8):1067-73.
48. Frankel RM, Eddins-Folensbee F, Inui TS. Crossing the patient-centred divide: transforming health care quality through enhanced faculty development. Acad Med 2011;86 (4):445-52.
49. Landy FJ, Farr JL. Performance rating. Psychol Bull 1980;87 (1):72-107.
50. Ilgen DR, Barnes-Farrell JL, McKellin DB. Performance appraisal process research in the 1980s: what has it contributed to appraisals in use? Organ Behav Hum Dec 1993;54 (3):321-68.
51. Baddeley A. The magical number seven: still magical after all these years? Psychol Rev 1994;101 (2):353.
52. van Merriënboer JJG, Sweller J. Cognitive load theory in health professional education: design principles and strategies. Med Educ 2010;44:85-93.
53. Macrae CN, Bodenhausen GV. Social cognition: categorical person perception. Br J Psychol 2001;92 (Pt 1):239-55.
54. Schmidt HG. Foundations of problem-based learning: some explanatory notes. Med Educ 1993;27:422-32.
55. Gingerich A, Regehr G, Eva KW. Rater-based assessments as social judgements: rethinking the aetiology of rater errors. Acad Med 2011;86 (10) (Suppl):1-7.
56. Tversky A, Kahneman D. Judgement under uncertainty: heuristics and biases. Science 1974;185 (4157):1124-31.
57. Bodenhausen GV, Wyer JRS. Effects of stereotypes on decision making and information-processing strategies. J Pers Soc Psychol 1985;48 (2):267-82.
58. Bodenhausen GV. Stereotypic biases in social decision making and memory: testing process models of stereotype use. J Pers Soc Psychol 1988;55 (5):726-37.
59. Dijksterhuis A, van Knippenberg A. Memory for stereotype-consistent and stereotype-inconsistent information as a function of processing pace. Eur J Soc Psychol 1995;25 (6):689-93.
60. MacRae CN, Schloerscheidt AM, Bodenhausen GV, Milne AB. Creating memory illusions: expectancy-based processing and the generation of false memories. Memory 2002;10 (1):63-80.
61. Macrae CN, Bodenhausen GV, Milne AB. The dissection of selection in person perception: inhibitory processes in social stereotyping. J Pers Soc Psychol 1995;69 (3):397-407.
62. Nisbett RE, Wilson TD. Telling more than we can know: verbal reports on mental processes. Psychol Rev 1977;84 (3):231-59.
63. Bargh JA, Chartrand TL. The unbearable automaticity of being. Am Psychol 1999;54 (7):462-79.
64. Bodenhausen GV, Sheppard LA, Kramer GP. Negative affect and social judgement: the differential impact of anger and sadness. Eur J Soc Psychol 1994;24 (1):45-62.
65. Bodenhausen GV. Stereotypes as judgemental heuristics: evidence of circadian variations in discrimination. Psychol Sci 1990;1 (5):319-22.
66. Kunda Z, Spencer SJ. When do stereotypes come to mind and when do they colour judgement? A goal-based theoretical framework for stereotype activation and application. Psychol Bull 2003;129 (4):522-44.
67. Crawford MT, Skowronski JJ. When motivated thought leads to heightened bias: high need for cognition can enhance the impact of stereotypes on memory. Pers Soc Psychol Bull 1998;24 (10):1075-88.
68. Macrae CN, Bodenhausen GV, Milne AB, Jetten J. Out of mind but back in sight: stereotypes on the rebound. J Pers Soc Psychol 1994;67 (5):808-17.
69. Woolf K, Cave J, Greenhalgh T, Dacre J. Ethnic stereotypes and the underachievement of UK medical students from ethnic minorities: qualitative study. BMJ 2008;337 (7670):611-5.
70. van den Bergh L, Denessen E, Hornstra L, Voeten M, Holland RW. The implicit prejudiced attitudes of teachers: relations to teacher expectations and the ethnic achievement gap. Am Educ Res J 2010;47 (2):497-527.
71. Tweed M, Ingham C. Observed consultation: confidence and accuracy of assessors. Adv Health Sci Educ Theory Pract 2010;15 (1):31-43.
72. Stewart N, Brown GDA, Chater N. Absolute identification by relative judgement. Psychol Rev 2005;112 (4):881-911.
73. Mussweiler T. Comparison processes in social judgement: mechanisms and consequences. Psychol Rev 2003;110 (3):472-89.
74. Wood TJ. Mental workload as a tool for understanding dual processes in rater-based assessments. Adv Health Sci Educ Theory Pract 2013;18 (3):523-5.
75. Tavares W, Eva KW. Exploring the impact of mental workload on rater-based assessments. Adv Health Sci Educ Theory Pract 2013;18 (2):291-303.
76. Wood TJ. Exploring the role of first impressions in rater-based assessments. Adv Health Sci Educ Theory Pract 2014;19 (3):409-27.
77. Byrne A, Tweed N, Halligan C. A pilot study of the mental workload of objective structured clinical examination examiners. Med Educ 2014;48:262-7.
78. Moskowitz GB, Li P. Egalitarian goals trigger stereotype inhibition: a proactive form of stereotype control. J Exp Soc Psychol 2011;47 (1):103-16.
79. Todd AR, Bodenhausen GV, Richeson JA, Galinsky AD. Perspective taking combats automatic expressions of racial bias. J Pers Soc Psychol 2011;100 (6):1027.
80. Watson RA. Use of a machine learning algorithm to classify expertise: analysis of hand motion patterns during a simulated surgical task. Acad Med 2014;89 (8):1163-1167.
81. Colliver JA. Educational theory and medical education practice: a cautionary note for medical school faculty. Acad Med 2002;77 (12):1217-20.
82. Baig LA, Violato C, Crutcher RA. Assessing clinical communication skills in physicians: are the skills context specific or generalisable. BMC Med Educ 2009;9:22.
83. Keller L, Mazor KM, Swaminathan H, Pugnaire MP. An investigation of the impacts of different generalisability study designs on estimates of variance components and generalisability coefficients. Acad Med 2000;75 (10) (Suppl):21-4.
84. Durning SJ, Artino AR Jr, Pangaro LN, van der Vleuten C, Schuwirth L. Redefining context in the clinical encounter: implications for research and training in medical education. Acad Med 2010;85 (5):894-901.
85. Durning SJ, Artino AR. Situativity theory: a perspective on how participants and the environment can interact: AMEE Guide no. 52. Med Teach 2011;33 (3):188-99.
86. Richter Lagha RA, Boscardin CK, May W, Fung C-C. A comparison of two standard-setting approaches in high-stakes clinical performance assessment using generalisability theory. Acad Med 2012;87 (8):1077-82.
87. Hager P, ed. Theories of Workplace Learning. Los Angeles, CA: Sage Publications 2011.
88. Engeström Y, Sannino A. Studies of expansive learning: foundations, findings and future challenges. Educ Res Rev 2010;5 (1):1-24.
89. Kuper A, Reeves S, Albert M, Hodges BD. Assessment: do we need to broaden our methodological horizons? Med Educ 2007;41:1121-3.
90. Ginsburg S, Bernabeo E, Ross KM, Holmboe ES. 'It depends': results of a qualitative study investigating how practising internists approach professional dilemmas. Acad Med 2012;87 (12):1685-93.
91. Hodges B. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach 2013;35 (7):564-8.
92. Lingard L. What we see and don't see when we look at 'competence': notes on a god term. Adv Health Sci Educ Theory Pract 2009;14 (5):625-8.
93. Delandshere G, Petrosky AR. Assessment of complex performances: limitations of key measurement assumptions. Educ Res 1998;27 (2):14-24.
94. Hodges B. Medical education and the maintenance of incompetence. Med Teach 2006;28 (8):690-6.
95. Pangaro L, ten Cate O. Frameworks for learner assessment in medicine: AMEE Guide no. 78. Med Teach 2013;35 (6):e1197-210.
96. Read SJ, Jones DK, Miller LC. Traits as goal-based categories: the importance of goals in the coherence of dispositional categories. J Pers Soc Psychol 1990;58 (6):1048-61.
97. Reeder GD, Kumar S, Hesson-McInnis M, Trafimow D. Inferences about the morality of an aggressor: the role of perceived motive. J Pers Soc Psychol 2002;83 (4):789-803.
98. Malle BF, Pearce GE. Attention to behavioural events during interaction: two actor-observer gaps and three attempts to close them. J Pers Soc Psychol 2001;81 (2):278-94.
99. Weiner B. Inferences of responsibility and social motivation. In: Zanna MP, ed. Advances in Experimental Social Psychology, vol 27. San Diego, CA: Academic Press 1995;1-47.
100. Berendonk C, Stalmeijer RE, Schuwirth LWT. Expertise in performance assessment: assessors' perspectives. Adv Health Sci Educ Theory Pract 2013;18 (4):559-71.
101. Govaerts MJB, Wiel MWJ, Schuwirth LWT, van der Vleuten CPM, Muijtjens AMM. Workplace-based assessment: raters' performance theories and constructs. Adv Health Sci Educ Theory Pract 2013b;18 (3):375-96.
102. Gruppen LD, Frohna AZ. Clinical reasoning. In: Norman GR, van der Vleuten CPM, Newble DI, eds. International Handbook of Research in Medical Education. Dordrecht: Kluwer Academic Publishers 2002;205-30.
103. Durning SJ, Artino AR, Boulet JR, Dorrance K, van der Vleuten C, Schuwirth L. The impact of selected contextual factors on experts' clinical reasoning performance (does context impact clinical reasoning performance in experts?). Adv Health Sci Educ Theory Pract 2012;17 (1):65-79.
104. Hofstadter D. I am a Strange Loop. New York, NY: Basic Books 2007.
105. Klein G. Streetlights and Shadows: Searching for the Keys to Adaptive Decision Making. Cambridge, MA: MIT Press 2009.
106. Chi MTH, Feltovich PJ, Glaser R. Categorisation and representation of physics problems by experts and novices. Cognit Sci 1981;5 (2):121-52.
107. Norman GR, Tugwell P, Feightner JW, Muzzin LJ, Jacoby LL. Knowledge and clinical problem-solving. Med Educ 1985;19:344-56.
108. Webster-Wright A. Reframing professional development through understanding authentic professional learning. Rev Educ Res 2009;79 (2):702-39.
109. Gipps C. Socio-cultural aspects of assessment. Rev Res Educ 1999;24:355-92.
110. Delandshere G, Petrosky AR. Capturing teachers' knowledge: performance assessment a) and post-structuralist epistemology, b) from a post-structuralist perspective, c) and post-structuralism, d) none of the above. Educ Res 1994;23 (5):11-8.
111. Elstein AS, Schwarz A. Clinical problem solving and diagnostic decision making: selective review of the cognitive literature. BMJ 2002;324 (7339):729-32.
112. Schuwirth LW, der van Vleuten CP. Assessing competence. In: Hodges BD, Lingard LA, eds. The Question of Competence: Reconsidering Medical Education in the Twenty-First Century. Ithaca, NY; London: ILR Press, Cornell University Press 2012;113-30.
113. Ross S, Poth CN, Donoff M, Humphries P, Steiner I, Schipper S, Janke F, Nichols D. Competency-based achievement system: using formative feedback to teach and assess family medicine residents' skills. Can Fam Physician 2011;57 (9):e323-30.
114. Kahneman D, Klein G. Strategic decisions: when can you trust your gut? [Interview]. McKinsey Quartley & Co. Inc. 2010. http://www.mckinsey.com/insights/strategy/strategic_decisions_when_can_you_trust_your_gut. [Accessed 28 October 2013.]
115. Crossley J, Jolly B. Making sense of work-based assessment: ask the right questions, in the right way, about the right things, of the right people. Med Educ 2012;46:28-37.
116. Nutter D, Whitcomb ME. The AAMC project on the clinical education of medical students. 2008. https://www.aamc.org/download/68522/data/clinicalskillsnutter.pdf. [Accessed 16 December 2013.]
117. Daelmans HEM, Hoogenboom RJI, Donker AJM, Scherpbier AJJA, Stehouwer CDA, van der Vleuten CPM. Effectiveness of clinical rotations as a learning environment for achieving competences. Med Teach 2004;26 (4):305-12.
118. Holmboe ES. Faculty and the observation of trainees' clinical skills: problems and opportunities. Acad Med 2004;79 (1):16-22.
119. Bindal N, Goodyear H, Bindal T, Wall D. DOPS assessment: a study to evaluate the experience and opinions of trainees and assessors. Med Teach 2013;35 (6):e1230-4.
120. Bindal T, Wall D, Goodyear HM. Trainee doctors' views on workplace-based assessments: are they just a tick box exercise? Med Teach 2011;33 (11):919-27.
121. Weston PSJ, Smith CA. The use of mini-CEX in UK Foundation Training six years following its introduction: lessons still to be learned and the benefit of formal teaching regarding its utility. Med Teach 2014;36 (2):155-63.
122. McKavanagh P, Smyth A, Carragher A. Hospital consultants and workplace based assessments: how foundation doctors view these educational interactions? Postgrad Med J 1037;2012 (88):119-24.
123. Sabey A, Harris M. Training in hospitals: what do GP specialist trainees think of workplace-based assessments? Educ Primary Care 2011;22 (2):90-9.
124. Tokode OM, Dennick R. A qualitative study of foundation doctors' experiences with mini-CEX in the UK. Int J Med Educ 2013;4:83-92.
125. Pangaro L. A new vocabulary and other innovations for improving descriptive in-training evaluations. Acad Med 1999;74 (11):1203-7.
126. Surowiecki J. The Wisdom of Crowds. New York, NY: Random House Digital 2005.
127. Baile WF, Buckman R, Lenzi R, Glober G, Beale EA, Kudelka AP. SPIKES - a six-step protocol for delivering bad news: application to the patient with cancer. Oncologist 2000;5 (4):302-11.