Assessing diagnostic reasoning: A consensus statement summarizing theory, practice, and future needs

Academic Emergency Medicine

Volumn 19, Issue 12, 2012, Pages 1454-1461

  Ilgen, Jonathan S ^a   Humbert, Aloysius J ^b   Kuhn, Gloria ^c   Hansen, Matthew L ^d   Norman, Geoffrey R ^e   Eva, Kevin W ^f   Charlin, Bernard ^g   Sherbino, Jonathan ^e  

^a UNIVERSITY OF WASHINGTON SCHOOL OF MEDICINE   (United States)

^b INDIANA UNIVERSITY SCHOOL OF MEDICINE   (United States)

^c WAYNE STATE UNIVERSITY SCHOOL OF MEDICINE   (United States)

^d OREGON HEALTH AND SCIENCE UNIVERSITY   (United States)

^e MCMASTER UNIVERSITY   (Canada)

^f UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^g UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

CLINICAL ASSESSMENT TOOL; CLINICAL DECISION MAKING; CLINICAL PRACTICE; CONSENSUS; DIAGNOSTIC ACCURACY; DIAGNOSTIC REASONING; EMERGENCY MEDICINE; EMERGENCY PHYSICIAN; HUMAN; MINI MENTAL STATE EXAMINATION; PATIENT SAFETY; PRIORITY JOURNAL; PROFESSIONAL COMPETENCE; PSYCHOMETRY; RELIABILITY; REVIEW; SIMULATION; STAFF TRAINING; STRUCTURED QUESTIONNAIRE; THEORY; VERBAL COMMUNICATION; VIRTUAL REALITY; WRITTEN LANGUAGE;

CLINICAL COMPETENCE; CONSENSUS DEVELOPMENT CONFERENCES AS TOPIC; EDUCATIONAL MEASUREMENT; EMERGENCY MEDICINE; HUMANS;

EID: 84871739209     PISSN: 10696563     EISSN: 15532712     Source Type: Journal    
DOI: 10.1111/acem.12034     Document Type: Review

References (113)

1. Kohn LT, Corrigan J, Donaldson MS, Institute of Medicine (U.S.). Committee on Quality of Health Care in America. To Err is Human: Building a Safer Health System. Washington, DC: National Academies Press, 2000.
2. Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Resident Education in Emergency Medicine, 2007. Available at: http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/ 110emergencymed07012007.pdf. Accessed Sep 9, 2012.
3. Liaison Committee on Medical Education. Functions and Structure of a Medical School. Available at: http://www.lcme.org/functions.pdf. Accessed Sep 9, 2012.
4. Royal College of Physician and Surgeons of Canada. General Standards Applicable to All Residency Programs: B Standards. Available at: http://www.cfpc.ca/uploadedFiles/Education/-PDFs/Blue-Book-B-Standards-January- %202011-English-Final.pdf. Accessed Sep 9, 2012.
5. National Health Service. The UK Foundation Programme Curriculum. Available at: http://www.foundationprogramme.nhs.uk/download.asp?file= Foundation-Curriculum-2011-WEB.pdf. Accessed Sep 9, 2012.
6. Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med. 1991; 324: 377-84.
7. Croskerry P, Sinclair D,. Emergency medicine: a practice prone to error? CJEM. 2001; 3: 271-6.
8. Kachalia A, Gandhi TK, Puopolo AL, et al. Missed and delayed diagnoses in the emergency department: a study of closed malpractice claims from 4 liability insurers. Ann Emerg Med. 2007; 49: 196-205.
9. Chellis M, Olson J, Augustine J, Hamilton G,. Evaluation of missed diagnoses for patients admitted from the emergency department. Acad Emerg Med. 2001; 8: 125-30.
10. Brown TW, McCarthy ML, Kelen GD, Levy F,. An epidemiologic study of closed emergency department malpractice claims in a national database of physician malpractice insurers. Acad Emerg Med. 2010; 17: 553-60.
11. Wears RL, Nemeth CP,. Replacing hindsight with insight: toward better understanding of diagnostic failures. Ann Emerg Med. 2007; 49: 206-9.
12. Tversky A, Kahneman D,. Judgment under uncertainty: heuristics and biases. Science. 1974; 185: 1124-31.
13. Tversky A, Kahneman D,. The framing of decisions and the psychology of choice. Science. 1981; 211: 453-8.
14. Flavell JH,. Metacognitive Aspects of Problem Solving. In:, Resnick LB, (ed.). The Nature of Intelligence. Hillsdale, NJ: Lawrence Erlbaum Associates, 1976, pp 231-5.
15. Eva KW,. What every teacher needs to know about clinical reasoning. Med Educ. 2005; 39: 98-106.
16. Norman G,. Dual processing and diagnostic errors. Adv Health Sci Educ Theory Pract. 2009; 14 (Suppl 1): 37-49.
17. Norman GR, Eva KW,. Diagnostic error and clinical reasoning. Med Educ. 2010; 44: 94-100.
18. Schmidt HG, Norman GR, Boshuizen HP,. A cognitive perspective on medical expertise: theory and implication. Acad Med. 1990; 65: 611-21.
19. Coderre S, Mandin H, Harasym PH, Fick GH,. Diagnostic reasoning strategies and diagnostic success. Med Educ. 2003; 37: 695-703.
20. Elstein AS, Shulman LS, Sprafka SA,. Medical Problem Solving: An Analysis of Clinical Reasoning. Cambridge, MA: Harvard University Press, 1978.
21. Mandin H, Jones A, Woloschuk W, Harasym P,. Helping students learn to think like experts when solving clinical problems. Acad Med. 1997; 72: 173-9.
22. Eva KW,. On the generality of specificity. Med Educ. 2003; 37: 587-8.
23. Eva KW, Neville AJ, Norman GR,. Exploring the etiology of content specificity: factors influencing analogic transfer and problem solving. Acad Med. 1998; 73: S1-5.
24. Norman GR, Tugwell P, Feightner JW, Muzzin LJ, Jacoby LL,. Knowledge and clinical problem-solving. Med Educ. 1985; 19: 344-56.
25. Eva KW, Hatala RM, Leblanc VR, Brooks LR,. Teaching from the clinical reasoning literature: combined reasoning strategies help novice diagnosticians overcome misleading information. Med Educ. 2007; 41: 1152-8.
26. Boshuizen HP, Schmidt HG,. On the role of biomedical knowlege in clinical reasoning by experts, intermediates, and novices. Cogn Sci. 1992; 16: 153-84.
27. Patel VL, (ed.). The Psychology of Learning and Motivation: Advances in Research and Theory. San Diego, CA: Academic Press, 1994.
28. Barrows HS, Feltovich PJ,. The clinical reasoning process. Med Educ. 1987; 21: 86-91.
29. Charlin B, Boshuizen HP, Custers EJ, Feltovich PJ,. Scripts and clinical reasoning. Med Educ. 2007; 41: 1178-84.
30. Charlin B, Tardif J, Boshuizen HP,. Scripts and medical diagnostic knowledge: theory and applications for clinical reasoning instruction and research. Acad Med. 2000; 75: 182-90.
31. Brooks L,. The matching game. New Physician. 1978; 27: 34-6.
32. Custers EJ, Boshuizen HP, Schmidt HG,. The role of illness scripts in the development of medical diagnostic expertise: results from an interview study. Cogn Instr. 1998; 16: 367-98.
33. Brooks LR, Norman GR, Allen SW,. Role of specific similarity in a medical diagnostic task. J Exp Psychol Gen. 1991; 120: 278-87.
34. Stanovich KE, West RF,. Individual differences in reasoning: implications for the rationality debate? Behav Brain Sci. 2000; 23: 645-65.
35. Evans JS,. Heuristic and analytic processes in reasoning. Br J Psychol. 1984; 75: 451-68.
36. Evans JS,. Dual-processing accounts of reasoning, judgment, and social cognition. Annu Rev Psychol. 2008; 59: 255-78.
37. Kahneman D,. Thinking, Fast and Slow. New York, NY: Farrar, Straus and Giroux, 2011.
38. Sloman SA,. The empirical case for two systems of reasoning. Psychol Bull. 1996; 119: 3-22.
39. Croskerry P,. A universal model of diagnostic reasoning. Acad Med. 2009; 84: 1022-8.
40. Eva KW, Cunnington JP,. The difficulty with experience: does practice increase susceptibility to premature closure? J Contin Educ Health Prof. 2006; 26: 192-8.
41. Ark TK, Brooks LR, Eva KW,. Giving learners the best of both worlds: do clinical teachers need to guard against teaching pattern recognition to novices? Acad Med. 2006; 81: 405-9.
42. Mamede S, Schmidt HG, Penaforte JC,. Effects of reflective practice on the accuracy of medical diagnoses. Med Educ. 2008; 42: 468-75.
43. Mamede S, Schmidt HG, Rikers RM, Penaforte JC, Coelho-Filho JM,. Influence of perceived difficulty of cases on physicians' diagnostic reasoning. Acad Med. 2008; 83: 1210-6.
44. Moulton CA, Regehr G, Lingard L, Merritt C, MacRae H,. Slowing down to stay out of trouble in the operating room: remaining attentive in automaticity. Acad Med. 2010; 85: 1571-7.
45. Gonzalez C,. Learning to make decisions in dynamic environments: effects of time constraints and cognitive abilities. Hum Factors. 2004; 46: 449-60.
46. Chisholm CD, Weaver CS, Whenmouth L, Giles B,. A task analysis of emergency physician activities in academic and community settings. Ann Emerg Med. 2011; 58: 117-22.
47. Westbrook JI, Coiera E, Dunsmuir WT, et al. The impact of interruptions on clinical task completion. Qual Saf Health Care. 2010; 19: 284-9.
48. Chisholm CD, Dornfeld AM, Nelson DR, Cordell WH,. Work interrupted: a comparison of workplace interruptions in emergency departments and primary care offices. Ann Emerg Med. 2001; 38: 146-51.
49. Coiera EW, Jayasuriya RA, Hardy J, Bannan A, Thorpe ME,. Communication loads on clinical staff in the emergency department. Med J Aust. 2002; 176: 415-8.
50. Spencer R, Coiera E, Logan P,. Variation in communication loads on clinical staff in the emergency department. Ann Emerg Med. 2004; 44: 268-73.
51. Brixey JJ, Tang Z, Robinson DJ, et al. Interruptions in a level one trauma center: a case study. Int J Med Inform. 2008; 77: 235-41.
52. Bos MW, Dijksterhuis A, van Baaren R,. Food for thought? Trust your unconscious when energy is low. J Neurosci Psychol Econ. 2012; 5: 24-130.
53. Masicampo EJ, Baumeister RF,. Toward a physiology of dual-process reasoning and judgment: lemonade, willpower, and expensive rule-based analysis. Psychol Sci. 2008; 19: 255-60.
54. van Merrienboer JJ, Sweller J,. Cognitive load theory in health professional education: design principles and strategies. Med Educ. 2010; 44: 85-93.
55. Sherbino J, Frank JR, (eds). Educational Design: A CanMEDS Guide for the Health Professions. Ottawa, Canada: The Royal College of Physicians & Surgeons, 2011.
56. Norcini J, Anderson B, Bollela V, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011; 33: 206-14.
57. Schuwirth LW, van der Vleuten CP,. Programmatic assessment and Kane's validity perspective. Med Educ. 2012; 46: 38-48.
58. Jacoby LL,. A process dissociation framework: separating automatic from intentional uses of memory. J Mem Lang. 1991; 30: 513-41.
59. Moulton CA, Regehr G, Mylopoulos M, MacRae HM,. Slowing down when you should: a new model of expert judgment. Acad Med. 2007; 82: S109-16.
60. Crossley J, Johnson G, Booth J, Wade W,. Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales. Med Educ. 2011; 45: 560-9.
61. Gingerich A, Regehr G, Eva KW,. Rater-based assessments as social judgments: rethinking the etiology of rater errors. Acad Med. 2011; 86: S1-7.
62. Holmboe ES, Hawkins RE, Huot SJ,. Effects of training in direct observation of medical residents' clinical competence: a randomized trial. Ann Intern Med. 2004; 140: 874-81.
63. Kogan JR, Conforti L, Bernabeo E, Iobst W, Holmboe E,. Opening the black box of clinical skills assessment via observation: a conceptual model. Med Educ. 2011; 45: 1048-60.
64. Kogan JR, Hess BJ, Conforti LN, Holmboe ES,. What drives faculty ratings of residents' clinical skills? The impact of faculty's own clinical skills. Acad Med. 2010; 85: S25-8.
65. Epstein RM,. Assessment in medical education. N Engl J Med. 2007; 356: 387-96.
66. Miller GE,. The assessment of clinical skills/competence/performance. Acad Med. 1990; 65: S63-7.
67. Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA. 2007; 298: 993-1001.
68. Wenghofer E, Klass D, Abrahamowicz M, et al. Doctor scores on national qualifying examinations predict quality of care in future practice. Med Educ. 2009; 43: 1166-73.
69. Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD,. Predictive validity of certification by the American Board of Internal Medicine. Ann Intern Med. 1989; 110: 719-26.
70. Hatala R, Norman GR, Brooks LR,. Impact of a clinical scenario on accuracy of electrocardiogram interpretation. J Gen Intern Med. 1999; 14: 126-9.
71. Young M, Brooks L, Norman G,. Found in translation: the impact of familiar symptom descriptions on diagnosis in novices. Med Educ. 2007; 41: 1146-51.
72. Page G, Bordage G, Allen T,. Developing key-feature problems and examinations to assess clinical decision-making skills. Acad Med. 1995; 70: 194-201.
73. Page G, Bordage G,. The Medical Council of Canada's key features project: a more valid written examination of clinical decision-making skills. Acad Med. 1995; 70: 104-10.
74. Charlin B, Roy L, Brailovsky C, Goulet F, van der Vleuten C,. The Script Concordance Test: a tool to assess the reflective clinician. Teach Learn Med. 2000; 12: 189-95.
75. Fournier JP, Demeester A, Charlin B,. Script Concordance Tests: guidelines for construction. BMC Med Inform Decis Mak. 2008; 8: e18.
76. Humbert AJ, Besinger B, Miech EJ,. Assessing clinical reasoning skills in scenarios of uncertainty: convergent validity for a Script Concordance Test in an emergency medicine clerkship and residency. Acad Emerg Med. 2011; 18: 627-34.
77. Goulet F, Jacques A, Gagnon R, Charlin B, Shabah A,. Poorly performing physicians: does the Script Concordance Test detect bad clinical reasoning? J Contin Educ Health Prof. 2010; 30: 161-6.
78. Bianchi L, Gallagher EJ, Korte R, Ham HP,. Interexaminer agreement on the American Board of Emergency Medicine oral certification examination. Ann Emerg Med. 2003; 41: 859-64.
79. Maatsch JL,. Assessment of clinical competence on the Emergency Medicine Specialty Certification Examination: the validity of examiner ratings of simulated clinical encounters. Ann Emerg Med. 1981; 10: 504-7.
80. Bandiera G, Sherbino J, Frank JR,. The CanMEDS Assessment Tools Handbook. An Introductory Guide to Assessment Methods for the CanMEDS Competencies. Ottawa, Canada: Royal College of Physicians and Surgeons of Canada, 2006.
81. Wong ML, Fones CS, Aw M, et al. Should non-expert clinician examiners be used in objective structured assessment of communication skills among final year medical undergraduates? Med Teach. 2007; 29: 927-32.
82. Chenot JF, Simmenroth-Nayda A, Koch A, et al. Can student tutors act as examiners in an objective structured clinical examination? Med Educ. 2007; 41: 1032-8.
83. Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M,. OSCE checklists do not capture increasing levels of expertise. Acad Med. 1999; 74: 1129-34.
84. Schuwirth LW, Van Der Vleuten CP,. The use of clinical simulations in assessment. Med Educ. 2003; 37: 65-71.
85. Brailovsky C, Charlin B, Beausoleil S, Cote S, Van der Vleuten C,. Measurement of clinical reflective capacity early in training as a predictor of clinical reasoning performance at the end of residency: an experimental study on the Script Concordance Test. Med Educ. 2001; 35: 430-6.
86. Schuwirth L, Gorter S, Van der Heijde D, et al. The role of a computerised case-based testing procedure in practice performance assessment. Adv Health Sci Educ Theory Pract. 2005; 10: 145-55.
87. Dillon GF, Clauser BE,. Computer-delivered patient simulations in the United States Medical Licensing Examination (USMLE). Simul Healthc. 2009; 4: 30-4.
88. Huang G, Reynolds R, Candler C,. Virtual patient simulation at US and Canadian medical schools. Acad Med. 2007; 82: 446-51.
89. Harik P, Cuddy MM, O'Donovan S, Murray CT, Swanson DB, Clauser BE,. Assessing potentially dangerous medical actions with the computer-based case simulation portion of the USMLE step 3 examination. Acad Med. 2009; 84: S79-82.
90. Sydor DT, Naik VN,. Simulation and competency-based medical education: "showing how." In:, Sherbino J, Frank JR, eds. Educational Design: A CanMEDS Guide for the Health Professions. Ottawa, Canada: Royal College of Physicians & Surgeons, 2011.
91. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011; 306: 978-88.
92. Norman G, Dore K, Grierson L,. The minimal relationship between simulation fidelity and transfer of learning. Med Educ. 2012; 46: 636-47.
93. McGaghie WC, Issenberg SB, Petrusa ER, Scalese RJ,. A critical review of simulation-based medical education research: 2003-2009. Med Educ. 2010; 44: 50-63.
94. Kassirer JP,. Teaching clinical reasoning: case-based and coached. Acad Med. 2010; 85: 1118-24.
95. Kogan JR, Holmboe ES, Hauer KE,. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009; 302: 1316-26.
96. Mitchell C, Bhat S, Herbert A, Baker P,. Workplace-based assessments of junior doctors: do scores predict training difficulties? Med Educ. 2011; 45: 1190-8.
97. Norcini JJ, Blank LL, Duffy FD, Fortna GS,. The mini-CEX: a method for assessing clinical skills. Ann Intern Med. 2003; 138: 476-81.
98. LaMantia J, Kane B, Yarris L, et al. Real-time inter-rater reliability of the Council of Emergency Medicine Residency Directors standardized direct observation assessment tool. Acad Emerg Med. 2009; 16 (Suppl 2): S51-7.
99. Shayne P, Gallahue F, Rinnert S, Anderson CL, Hern G, Katz E,. Reliability of a core competency checklist assessment in the emergency department: the Standardized Direct Observation Assessment Tool. Acad Emerg Med. 2006; 13: 727-32.
100. Bandiera G, Lendrum D,. Daily encounter cards facilitate competency-based feedback while leniency bias persists. CJEM. 2008; 10: 44-50.
101. Dudek NL, Marks MB, Wood TJ, Lee AC,. Assessing the quality of supervisors' completed clinical evaluation reports. Med Educ. 2008; 42: 816-22.
102. Norcini J, Burch V,. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007; 29: 855-71.
103. Goulet F, Jacques A, Gagnon R, Racette P, Sieber W,. Assessment of family physicians' performance using patient charts: interrater reliability and concordance with chart-stimulated recall interview. Eval Health Prof. 2007; 30: 376-92.
104. Accreditation Council of Graduate Medical Education, American Board of Medical Specialties. Toolbox of Assessment Methods. Available at: http://www.acgme.org/outcome/assess/ toolbox.asp. Accessed Jan 15, 2001 (site subsequently removed). Now: http://www.partners.org/Assets/Documents/Graduate- Medical-Education/ToolTable.pdf. Accessed Sep 9, 2012.
105. Ramsey PG, Carline JD, Blank LL, Wenrich MD,. Feasibility of hospital-based use of peer ratings to evaluate the performances of practicing physicians. Acad Med. 1996; 71: 364-70.
106. Ramsey PG, Wenrich MD,. Peer ratings. An assessment tool whose time has come. J Gen Intern Med. 1999; 14: 581-2.
107. Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP,. Use of peer ratings to evaluate physician performance. JAMA. 1993; 269: 1655-60.
108. Wenrich MD, Carline JD, Giles LM, Ramsey PG,. Ratings of the performances of practicing internists by hospital-based registered nurses. Acad Med. 1993; 68: 680-7.
109. Arnold L, Willoughby L, Calkins V, Gammon L, Eberhart G,. Use of peer evaluation in the assessment of medical students. Acad Med. 1981; 56: 35-42.
110. Archer JC, Norcini J, Davies HA,. Use of SPRAT for peer review of paediatricians in training. BMJ. 2005; 330: 1251-3.
111. Whitehouse A, Walzman M, Wall D,. Pilot study of 360 degrees assessment of personal skills to inform record of in training assessments for senior house officers. Hosp Med. 2002; 63: 172-5.
112. Wright C, Richards SH, Hill JJ, et al. Multisource feedback in medical regulation: the example of the UK GMC Patient and Colleague Questionnaires. Acad Med. 2012;(in press).
113. Rethans JJ, Sturmans F, Drop R, van der Vleuten C,. Assessment of the performance of general practitioners by the use of standardized (simulated) patients. Br J Gen Pract. 1991; 41: 97-9.