Towards a program of assessment for health professionals: from training into practice

Advances in Health Sciences Education

Volumn 21, Issue 4, 2016, Pages 897-913

  Eva, Kevin W ^a   Bordage, Georges ^b   Campbell, Craig ^c   Galbraith, Robert ^d   Ginsburg, Shiphra ^e   Holmboe, Eric ^f   Regehr, Glenn ^a  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^b UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

^c ROYAL COLLEGE OF PHYSICIANS AND SURGEONS OF CANADA   (Canada)

^d NATIONAL BOARD OF MEDICAL EXAMINERS   (United States)

^e UNIVERSITY OF TORONTO   (Canada)

^f ACCREDITATION COUNCIL FOR GRADUATE MEDICAL EDUCATION   (United States)

Author keywords

Assessment; Competency based education; Continuing professional development; Health professional education

Indexed keywords

CURRICULUM; EDUCATION; HUMAN; MEDICAL PROFESSION; PATIENT SAFETY; TOTAL QUALITY MANAGEMENT;

COMPETENCY-BASED EDUCATION; EDUCATIONAL MEASUREMENT; HEALTH OCCUPATIONS; HUMANS; PATIENT SAFETY; QUALITY IMPROVEMENT;

EID: 84947705073     PISSN: 13824996     EISSN: 15731677     Source Type: Journal    
DOI: 10.1007/s10459-015-9653-6     Document Type: Article

References (100)

1. ABA: American Board of Anesthesiology. (2014). MOCA minute. http://www.theaba.org/MOCA/MOCA-Minute. Last accessed November 2, 2015.
2. AFMC: Association of Faculties of Medicine in Canada. (2010). The Future of Medical Education in Canada (FMEC): A collective vision for MD education. Retrieved from http://www.afmc.ca/fmec/pdf/collective_vision.pdf.
3. Bernabeo, E., Hood, S., Iobst, W., Holmboe, E., & Caverzagie, K. (2013). Optimizing the implementation of practice improvement modules in training: Lessons from educators. Journal of Graduate Medical Education, 5(1), 74–80.
4. Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. P. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). Cambridge, MA: MIT Press.
5. Bogo, M., Regehr, C., Logie, C., et al. (2011). Adapting objective structured clinical examinations to assess social work students’ performance and reflections. Journal of Social Work Education, 47, 5–18.
6. Bordage, G., Meguerditchian, A. N., & Tamblyn, R. (2013). Avoidable adverse events: A content analysis of a national qualifying examination. Academic Medicine, 88, 1493–1498.
7. Boud, D., & Molloy, E. (Eds.). (2013). Feedback in higher and professional education: Understanding it and doing it well. London: Routledge.
8. Butler, R. (1987). Task-involving and ego-involving properties of evaluation: Effects of different feedback conditions on motivational perceptions, interest, and performance. Journal of Educational Psychology, 79, 474–482.
9. Cadieux, G., Tamblyn, R., Dauphinee, D., & Libman, M. (2007). Predictors of inappropriate antibiotic prescribing among primary care physicians. CMAJ, 177(8), 877–883.
10. Choudhry, N. K., Fletcher, R. H., & Soumerai, S. B. (2005). Systematic review: The relationship between clinical experience and quality of health care. Annals of Internal Medicine, 142(4), 260–273.
11. Cizek, G. J. (2012). Defining and distinguishing validity: Interpretations of score meaning and justification on test use. Psychological Methods, 17, 31–43.
12. Colliver, J. A. (2002). Educational theory and medical education practice: A cautionary note for medical school faculty. Academic Medicine, 77(12), 1217–1220.
13. Cook, D. A. (2014). When I say… validity. Medical Education, 48(10), 948–949.
14. Cook, D. A., Brydges, R., Ginsburg, S., & Hatala, R. (2015). A contemporary approach to validity arguments: A practical guide to Kane’s framework. Medical Education, 49, 560–575.
15. Cote, L., & Bordage, G. (2012). Content and conceptual frameworks of preceptor feedback in response to residents’ educational needs. Academic Medicine, 87(9), 1274–1281.
16. Cruess, R., & Cruess, S. (2014). Updating the Hippocratic Oath to include medicine’s social contract. Medical Education, 48(1), 95–100.
17. Custers, E. (2010). Long-term retention of basic science knowledge: A review study. Advances in Health Sciences Education, 15(1), 109–128.
18. Downing, S. M. (2003). Validity: On the meaningful interpretation of assessment data. Medical Education, 37, 830–837.
19. Ellaway, R. H., Pusic, M. V., Galbraith, R. M., & Cameron, T. (2014). Developing the role of big data and analytics in health professional education. Medical Teacher, 36(3), 216–222.
20. Ericsson, K. A. (2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine, 79, S70–S81.
21. Eva, K. W. (2002). The aging physician: Changes in cognitive processing and their impact on medical practice. Academic Medicine, 77, S1–S6.
22. Eva, K. W. (2003). On the generality of specificity. Medical Education, 37, 587–588.
23. Eva, K. W. (2009). Diagnostic error in medical education: Where wrongs can make rights. Advances in Health Sciences Education, 14, 71–81.
24. Eva, K. W., Bordage, G., Campbell, C., Galbraith, R., Ginsburg, S., Holmboe, E., & Regehr, G. (2013). Medical Education Assessment Advisory Committee report to the Medical Council of Canada on Current Issues in Health Professional and Health Professional Trainee Assessment. Retrieved from http://mcc.ca/wp-content/uploads/Reports-MEAAC.pdf.
25. Eva, K. W., & Cunnington, J. P. (2006). The difficulty with experience: Does practice increase susceptibility to premature closure? Journal of Continuing Education in the Health Professions, 26(3), 192–198.
26. Eva, K. W., & Hodges, B. D. (2012). Scylla or Charbydis? Can we navigate between objectification and judgment in assessment? Medical Education, 46, 914–919.
27. Eva, K. W., Munoz, J., Hanson, M. D., Walsh, A., & Wakefield, J. (2010). Which factors, personal or external, most influence students’ generation of learning goals? Academic Medicine, 85, S102–S105.
28. Eva, K. W., & Regehr, G. (2013). Effective feedback for maintenance of competence: From data delivery to trusting dialogues. CMAJ, 185, 463–464.
29. Eva, K. W., Regehr, G., & Gruppen, L. D. (2012). Blinded by ‘insight’: Self-assessment and its role in performance improvement. In B. D. Hodges & L. Lingard (Eds.), The question of competence: Reconsidering medical education in the twenty-first century (pp. 131–154). Ithaca, NY: Cornell University Press.
30. Farmer, E. A., & Page, G. (2005). A practical guide to assessing clinical decision-making skills using the key features approach. Medical Education, 39, 1188–1194.
31. Frank, J. R., Snell, L. S., Cate, O. T., Holmboe, E. S., Carraccio, C., Swing, S. R., et al. (2010). Competency-based medical education: theory to practice. Medical Teacher, 32(8), 638–645.
32. Galbraith, R. M., Clyman, S., & Melnick, D. E. (2011). Conceptual perspectives: Emerging changes in the assessment paradigm. In J. P. Hafler (Ed.), Extraordinary learning in the workplace (pp. 87–100). Berlin: Springer.
33. Galbraith, R. M., Hawkins, R. E., & Holmboe, E. S. (2008). Making self-assessment more effective. Journal of Continuing Education in the Health Professions, 28(1), 20–24.
34. Gierl, M. J., & Lai, H. (2013). Evaluating the quality of medical multiple-choice items created with automated processes. Medical Education, 47(7), 726–733.
35. Gierl, M. J., Lai, H., & Turner, S. R. (2012). Using automatic item generation to create multiple-choice test items. Medical Education, 46(8), 757–765.
36. Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014). Seeing the ‘black box’ differently: Assessor cognition from three research perspectives. Medical Education, 48(11), 1055–1068.
37. Ginsburg, S., Eva, K., & Regehr, G. (2013). Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Academic Medicine, 88(10), 1539–1544.
38. Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780–786.
39. Ginsburg, S., Regehr, G., & Lingard, L. (2004). Basing the evaluation of professionalism on observable behaviours: A cautionary tale. Academic Medicine, 79(10, Suppl), S1–S4.
40. Goldszmidt, M., Minda, J. P., & Bordage, G. (2013). What physicians reason about during clinical encounters: Time to be more explicit. Academic Medicine, 88(3), 390–394.
41. Guadagnoli, M., Morin, M. P., & Dubrowski, A. (2012). The application of the challenge point framework in medical education. Medical Education, 46(5), 447–453.
42. Harrison, C. J., Könings, K. D., Schuwirth, L., Wass, V., & van der Vleuten, C. (2015). Barriers to the uptake and use of feedback in the context of summative assessment. Advances in Health Sciences Education, 20(1), 229–245.
43. Hatala, R., Marr, S., Cuncic, C., & Bacchus, C. M. (2011). Modification of an OSCE format to enhance patient continuity in a high-stakes assessment of clinical performance. BMC Medical Education, 11, 23.
44. Hawkins, et al. (under review). The ABMS MOC Part III examination: Value, concerns and alternative formats.
45. Hays, R., & Gay, S. (2011). Reflection or ‘pre-reflection’: What are we actually measuring in reflective practice? Medical Education, 45(2), 116–118.
46. Hodges, B. (2003). OSCE! variations on a theme by Harden. Medical Education, 37(12), 1134–1140.
47. Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., & Frank, J. R. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32(8), 676–682.
48. James, J. T. (2013). A new, evidence-based estimate of patient harms associated with hospital care. Journal of Patient Safety, 9(3), 122–128.
49. Jarvis-Selinger, S., Pratt, D. D., & Regehr, G. (2012). Competency is not enough: integrating identity formation into the medical education discourse. Academic Medicine, 87(9), 1185–1190.
50. Kane, M. T. (1992). An argument-based approach to validation. Psychological Bulletin, 112, 527–535.
51. Karpicke, J. D., & Roediger, H. L, I. I. I. (2008). The critical importance of retrieval for learning. Science, 319, 966–968.
52. Kennedy, T. J., Regehr, G., Baker, G. R., & Lingard, L. A. (2009). ‘It’s a cultural expectation…’ The pressure on medical trainees to work independently in clinical practice. Medical Education, 43(7), 645–653.
53. Klass, D. A. (2007). Performance-based conception of competence is changing the regulation of physicians’ professional behavior. Academic Medicine, 82(6), 529–535.
54. Kluger, A. N., & van Dijk, D. (2010). Feedback, the various tasks of the doctor, and the feedforward alternative. Medical Education, 44, 1166–1174.
55. Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. S. (2011). Opening the black box of postgraduate trainee assessment in the clinical setting via observation: A conceptual model. Medical Education, 45, 1048–1060.
56. Kogan, J. R., & Holmboe, E. (2013). Realizing the promise and importance of performance-based assessment. Teaching and Learning in Medicine, 25(Suppl 1), S68–S74.
57. Kogan, J. R., Holmboe, E. S., & Hauer, K. R. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. JAMA, 302, 1316–1326.
58. Kohn, L. T., Corrigan, J. M., & Donaldson, M. S. (Eds.). (1999). To err is human: building a safer health system. Washington, DC: National Academy Press, Institute of Medicine.
59. Kornell, N., & Son, L. K. (2009). Learners’ choices and beliefs about self-testing. Memory, 17, 493–501.
60. Kromann, C. B., Bohnstedt, C., Jensen, M. L., & Ringsted, C. (2010). The testing effect on skills learning might last 6 months. Advances in Health Sciences Education, 15(3), 395–401.
61. Krumholz, et al. (under review). Recommendations to the American Board of Internal Medicine (ABIM): A vision for certification in internal medicine in 2020.
62. Larsen, D. P., Butler, A. C., & Roediger, H. L, 3rd. (2008). Test-enhanced learning in medical education. Medical Education, 42(10), 959–966.
63. MacRae, H. M., Cohen, R., Regehr, G., Reznick, R., & Burnstein, M. (1997). A new assessment tool: the patient assessment and management examination. Surgery, 122(2), 335–343.
64. Mann, K., Gordon, J., & MacLeod, A. (2009). Reflection and reflective practice in health professions education: A systematic review. Advances in Health Sciences Education, 14(4), 595–621.
65. Mann, K. V., van der Vleuten, C., Eva, K., Armson, H., Chesluk, B., Dornan, T., et al. (2011). Tensions in informed self-assessment: How the desire for feedback and reticence to collect and use it conflict. Academic Medicine, 86, 1120–1127.
66. Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52, 1187–1197.
67. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–104). New York: American Council on Education and Macmillan.
68. Morcke, A. M., Dornan, T., & Elka, B. (2013). Outcome (competency) based education: an exploration of its origins, theoretical basis and empirical evidence. Advances in Health Sciences Education, 18, 851–863.
69. Mutabdzic, D., Mylopoulos, M., Murnaghan, M. L., Patel, P., Zilbert, N., Seemann, N., et al. (2015). Coaching surgeons: Is culture limiting our ability to improve? Annals of Surgery, 262(2), 213–216.
70. Mylopoulos, M., & Regehr, G. (2011). Putting the expert together again. Medical Education, 45(9), 920–926.
71. Mylopoulos, M., & Scardamalia, M. (2008). Doctors’ perspectives on their innovations in daily practice: implications for knowledge building in health care. Medical Education, 42(10), 975–981.
72. Neve, H., & Hanks, S. (2016). When I say … capability. Medical Education, 50 (in press).
73. Newble, D. I., & Jaeger, K. (1983). The effect of assessments and examinations on the learning of medical students. Medical Education, 17(3), 165–171.
74. Newell, K. M., Liu, Y., & Mayer-Kress, G. (2001). Time scales in motor learning and development. Psychological Review, 108, 57–82.
75. Norcini, J. J. (2005). Current perspectives in assessment: The assessment of performance at work. Medical Education, 39(9), 880–889.
76. Norcini, J., Anderson, B., Bollela, V., Burch, V., Costa, M. J., Duvivier, R., et al. (2011). Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 conference. Medical Teacher, 33(3), 206–214.
77. Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138(6), 476–481.
78. Norcini, J., & Burch, V. (2007). Workplace-based assessment as an educational tool: AMEE Guide No. 31. Medical Teacher, 29(9), 855–871.
79. Norman, G., Dore, K., & Grierson, L. (2012). The minimal relationship between simulation fidelity and transfer of learning. Medical Education, 46(7), 636–647.
80. Norman, G., Neville, A., Blake, J. M., & Mueller, B. (2010). Assessment steers learning down the right road: Impact of progress testing on licensing examination performance. Medical Teacher, 32(6), 496–499.
81. Norman, G. R., Norcini, J., & Bordage, G. (2014). Competency-based education: Milestones or millstones. Journal of Graduate Medical Education, 6(1), 1–6.
82. Page, G., & Bordage, G. (1995). The medical council of Canada’s key feature project: A more valid written exam. of clinical decision-making skills. Academic Medicine, 70, 104–110.
83. Pugh, D., Hamstra, S. J., Wood, T. J., Humphrey-Murto, S., Touchie, C., Yudkowsky, R., Bordage, G. (2014). A procedural skills OSCE: Assessing technical and non-technical skills of internal medicine residents. Advances in health sciences education. Retrieved from http://link.springer.com/article/10.1007/s10459-014-9512-x?sa_campaign=email/event/articleAuthor/onlineFirst.
84. Razack, S., Hodges, B., Steinert, Y., & Maguire, M. (2015). Seeking inclusion in an exclusive process: Discourses of medical school student selection. Medical Education, 49, 36–47.
85. RCPSC: Royal College of Physicians and Surgeons of Canada. (2011). Assessment strategies within the revised maintenance of certification program, draft recommendations.
86. Regehr, G. (1994). Chickens and children do not an expert make. Academic Medicine, 69, 970–971.
87. Regehr, G., Eva, K., Ginsburg, S., Halwani, Y., & Sidhu, R. (2011). Future of medical education in Canada postgraduate project environmental scan. Paper 13. Assessment in postgraduate medical education: Trends and issues in assessment in the workplace. Retrieved from http://www.afmc.ca/pdf/fmec/13_Regehr_Assessment.pdf.
88. Rohrer, D., & Pashler, H. (2010). Recent research on human learning challenges conventional instructional strategies. Educational Research, 38, 406–412.
89. Sargeant, J., Eva, K. W., Armson, H., Chesluk, B., Dornan, T., Holmboe, E., et al. (2011). Features of assessment learners use to make informed self-assessments of clinical performance. Medical Education, 45, 636–647.
90. Schön, D. (1983). The reflective practitioner: How professionals think in action. London: Temple Smith.
91. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78, 153–189.
92. Swanson, D., & Roberts, T. (2016). Trends in national licensing examinations. Medical Education, 50(1) (in press).
93. Tamblyn, R., Abrahamowicz, M., Dauphinee, D., et al. (2007). Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA, 298(9), 993–1001.
94. Teunissen, P. W., & Westerman, M. (2011). Opportunity or threat: The ambiguity of the consequences of transitions in medical education. Medical Education, 45(1), 51–59.
95. van der Vleuten, C. (1996). The assessment of professional competence: Developments, research and practical implications. Advances in Health Sciences Education, 1, 41–67.
96. van der Vleuten, C. P., & Schuwirth, L. W. (2005). Assessing professional competence: From methods to programmes. Medical Education, 39(3), 309–317.
97. van Tartwijk, J., & Driessen, E. W. (2009). Portfolios for assessment and learning: AMEE Guide no. 45. Medical Teacher, 31(9), 790–801.
98. Watling, C., Driessen, E., van der Vleuten, C. P., & Lingard, L. (2014). Learning culture and feedback: An international study of medical athletes and musicians. Medical Education, 48(7), 713–723.
99. Wenghofer, E., Klass, D., Abrahamowicz, M., et al. (2009). Doctor scores on national qualifying examinations predict quality of care in future practice. Medical Education, 43(12), 1166–1173.
100. Williams, R. G., Klamen, D. L., Markwell, S. J., Cianciolo, A. T., Colliver, J. A., & Verhulst, S. J. (2014). Variations in senior medical student diagnostic justification ability. Academic Medicine, 89(5), 790–798.