Achieving the desired transformation: Thoughts on next steps for outcomes-based medical education

Academic Medicine

Volumn 90, Issue 9, 2015, Pages 1215-1223

  Holmboe, Eric S ^a   Batalden, Paul ^b  

^a ACCREDITATION COUNCIL FOR GRADUATE MEDICAL EDUCATION   (United States)

^b DARTMOUTH INSTITUTE FOR HEALTH POLICY AND CLINICAL PRACTICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLINICAL COMPETENCE; CURRICULUM; ECONOMICS; EDUCATION; HUMAN; MEDICAL EDUCATION; STANDARDS; TOTAL QUALITY MANAGEMENT; UNITED STATES;

CLINICAL COMPETENCE; CURRICULUM; EDUCATION, MEDICAL, GRADUATE; EDUCATIONAL MEASUREMENT; HUMANS; QUALITY IMPROVEMENT; TRAINING SUPPORT; UNITED STATES;

EID: 84944240454     PISSN: 10402446     EISSN: 1938808X     Source Type: Journal    
DOI: 10.1097/ACM.0000000000000779     Document Type: Review

References (168)

1. Frank J, Jabbour M, Tugwell P, for the Societal Needs Working Group Skills for the New Millenium: Report of the Societal Needs Working Group 1996 Ottawa, Ontario, Canada Royal College of Physicians and Surgeons of Canada
2. Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S., General competencies and accreditation in graduate medical education. Health Aff (Millwood) 2002 21 103 111
3. Talbot M., Monkey see, monkey do: A critique of the competency model in graduate medical education. Med Educ 2004 38 587 592
4. Lurie SJ, Mooney CJ, Lyness JM., Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: A systematic review. Acad Med 2009 84 301 309
5. Norman G, Norcini J, Bordage G., Competency-based education: Milestones or millstones? J Grad Med Educ 2014 6 1 6
6. Glass JM., Competency based training is a framework for incompetence. BMJ 2014 348 g2909
7. Institute of Medicine Graduate Medical Education That Meets the Nation's Health Needs 2014 Washington, DC National Academies Press
8. Association of American Medical Colleges IOM's vision of GME will not meet real-world patient needs [news release]. 2014. https://www.aamc.org/newsroom/newsreleases/381882/07292014.html. Accessed April 24, 2015
9. McGaghie WC, Miller GE, Sajid AW, Telder TV, Lipson L., Competency-Based Curriculum Development in Medical Education: An Introduction 1978 Geneva, Switzerland World Health Organization. http://whqlibdoc.who.int/php/WHO-PHP-68.pdf. Accessed April 24, 2015
10. Frenk J, Chen L, Bhutta ZA, Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. Lancet 2010 376 1923 1958
11. Frank JR, Snell LS, Cate OT, Competency-based medical education: Theory to practice. Med Teach 2010 32 638 645
12. Graham DW., Heraclitus. Internet Encyclopedia of Philosophy. http://www.iep.utm.edu/heraclit/. Accessed May 1, 2015
13. Heifetz RA, Linsky M., Leadership on the Line: Staying Alive Through the Dangers of Leading 2002 Boston, Mass Harvard Business School Press
14. Deming WE., The New Economics 2000 2nd ed Cambridge, Mass MIT Press
15. Kegan R, Lahey LL., How the Way We Talk Can Change the Way We Work 2001 San Francisco, Calif Jossey-Bass
16. Lehmann LS, Brancati FL, Chen MC, Roter D, Dobs AS., The effect of bedside case presentations on patients' perceptions of their medical care. N Engl J Med 1997 336 1150 1155
17. Engel GL., Are medical schools neglecting clinical skills? JAMA 1976 236 861 863
18. Gonzalo JD, Chuang CH, Huang G, Smith C., The return of bedside rounds: An educational intervention. J Gen Intern Med 2010 25 792 798
19. Martin RL., The Opposable Mind: How Successful Leaders Win Through Integrative Thinking 2007 Boston, Mass Harvard Business School Publishing
20. Holcombe RF., Improving health care quality: A polarity management perspective. Acad Med 2015 90 259
21. Scharmer CO., Theory U: Leading From the Future As It Emerges 2007 Boston, Mass Society for Organizational Learning
22. Freire K, Sangiorgi D., Service design and healthcare innovation: From consumption to co-production and co-creation. 2010 Linkoping, Sweden Second Nordic Conference on Service Design and Service Innovation. http://www.servdes.org/pdf/freire-sangiorgi.pdf. Accessed April 24, 2015
23. Glouberman S, Zimmerman B., Complicated and complex systems: What would successful reform of Medicare look like? Discussion paper no. 8. Commission on the Future of Healthcare in Canada 2002. http://c.ymcdn.com/sites/www.plexusinstitute.org/resource/collection/6528ED29-9907-4BC7-8D00-8DC907679FED/ComplicatedAndComplexSystems-ZimmermanReport-Medicare-reform.pdf. Accessed April 24, 2015
24. Fuchs VR., The Service Economy 1968 Cambridge, Mass National Bureau of Economic Research. http://www.nber.org/books/fuch68-1. Accessed April 24, 2015
25. Bate P, Robert G., Experience-based design: From redesigning the system around the patient to co-designing services with the patient. Qual Saf Health Care 2006 15 307 310
26. Sabadosa KA, Batalden PB., The interdependent roles of patients, families and professionals in cystic fibrosis: A system for the coproduction of healthcare and its improvement. BMJ Qual Saf 2014 23 suppl 1 i90 i94
27. Normann R., Reframing Business: When the Map Changes the Landscape 2001 Chichester, UK Wiley Publishing
28. Batalden M, Leach D, Batalden P., Batalden P, Foster T, Chapter 4: Better professional development: Competence, mastery, pride and joy. Sustainably Improving Health Care: Creatively Linking Care Outcomes, System Performance, and Professional Development 2012 London, UK Radcliffe Publishing
29. Cooke M, Irby DM, O'Brien BC, Carnegie Foundation for the Advancement of Teaching Educating Physicians: A Call for Reform of Medical School and Residency 2010 San Francisco, Calif Jossey-Bass
30. Institute of Medicine Health Professions Education: A Bridge to Quality 2003 Washington, DC National Academies Press
31. MedPAC Graduate medical education financing: Focusing on educational priorities. Report to the Congress: Aligning Incentives in Medicare 2010 Washington, DC MedPAC 103 128. http://www.medpac.gov/documents/reports/Jun10-Ch04.pdf. Accessed April 24, 2015
32. Antiel RM, Van Arendonk KJ, Reed DA, Surgical training, duty-hour restrictions, and implications for meeting the Accreditation Council for Graduate Medical Education core competencies: Views of surgical interns compared with program directors. Arch Surg 2012 147 536 541
33. Ten Cate O., The false dichotomy of quality and quantity in the discourse around assessment in competency-based education [published online June 8, 2014]. Adv in Health Sci Educ
34. Division of Orthopedics, University of Toronto A practical approach to competency-based education in your residency program. http://www.competency-based-training.com/. May 1, 2015
35. Long DM., Competency-based residency training: The next advance in graduate medical education. Acad Med 2000 75 1178 1183
36. Hafferty FW., Duty hours, distant voices, and an alternative future. Arch Surg 2012 147 496 497
37. Holmboe ES., Faculty and the observation of trainees' clinical skills: Problems and opportunities. Acad Med 2004 79 16 22
38. Crosson FJ, Leu J, Roemer BM, Ross MN., Gaps in residency training should be addressed to better prepare doctors for a twenty-first-century delivery system. Health Aff (Millwood) 2011 30 2142 2148
39. Lypson ML, Frohna JG, Gruppen LD, Woolliscroft JO., Assessing residents' competencies at baseline: Identifying the gaps. Acad Med 2004 79 564 570
40. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR., The role of assessment in competency-based medical education. Med Teach 2010 32 676 682
41. Di Francesco L, Pistoria MJ, Auerbach AD, Nardino RJ, Holmboe ES., Internal medicine training in the inpatient setting. A review of published educational interventions. J Gen Intern Med 2005 20 1173 1180
42. Bernabeo EC, Holtman MC, Ginsburg S, Rosenbaum JR, Holmboe ES., Lost in transition: The experience and impact of frequent changes in the inpatient learning environment. Acad Med 2011 86 591 598
43. Zafar MA, Diers T, Schauer DP, Warm EJ., Connecting resident education to patient outcomes: The evolution of a quality improvement curriculum in an internal medicine residency. Acad Med 2014 89 1341 1347
44. Warm EJ., Interval examination: The ambulatory long block. J Gen Intern Med 2010 25 750 752
45. Hauer KE, Hirsh D, Ma I, The role of role: Learning in longitudinal integrated and traditional block clerkships. Med Educ 2012 46 698 710
46. Gaufberg E, Hirsh D, Krupat E, Into the future: Patient-centredness endures in longitudinal integrated clerkship graduates. Med Educ 2014 48 572 582
47. Commonwealth Medical College Longitudinal model. http://www.tcmc.edu/oth/Page.asp?PageID=OTH000478. Accessed April 24, 2015
48. Ostrom E., Crossing the great divide: Coproduction, synergy, and development. World Dev 1996 24 1073 1087
49. Garn HA., Models for Indicator Development: A Framework for Policy Analysis 1976 Washington, DC Urban Institute
50. Wagner EH, Austin BT, Von Korff M., Improving outcomes in chronic illness. Manag Care Q 1996 4 12 25
51. Molloy E, Boud D., Boud D, Molloy E, Changing conceptions of feedback. Feedback in Higher and Professional Education: Understanding It and Doing It Well 2013 New York, NY Routledge 11 33
52. Tamblyn R, McLeod P, Hanley JA, Girard N, Hurley J., Physician and practice characteristics associated with the early utilization of new prescription drugs. Med Care 2003 41 895 908
53. Asch DA, Nicholson S, Srinivas S, Herrin J, Epstein AJ., Evaluating obstetrical residency programs using patient outcomes. JAMA 2009 302 1277 1283
54. Sirovich BE, Lipner RS, Johnston M, Holmboe ES., The association between residency training and internists' ability to practice conservatively. JAMA Intern Med 2014 174 1640 1648
55. Duffy FD, Holmboe ES., Self-assessment in lifelong learning and improving performance in practice: Physician know thyself. JAMA 2006 296 1137 1139
56. Cook DA, Holmboe ES, Sorensen KJ, Berger RA, Wilkinson JM., Getting maintenance of certification to work: A grounded theory study of physicians' perceptions. JAMA Intern Med 2015 175 35 42
57. Sargeant J, Armson H, Chesluk B, Processes and dimensions of informed self-assessment. Acad Med 2010 85 1212 1220
58. Eva KW, Regehr G., Effective feedback for maintenance of competence: From data delivery to trusting dialogues. CMAJ 2013 185 463 464
59. Weise MR, Christensen CM., Hire education. Mastery, modularization, and the workforce revolution. http://www.christenseninstitute.org/publications/hire/. Accessed April 24, 2015
60. Khan Academy Health and medicine. http://www.khanacademy.org/science/health-and-medicine. Accessed April 24, 2015
61. Rohrer D, Pashler H., Recent research on human learning challenges conventional instructional strategies. Educ Res 2010 39 406 412
62. Karpicke JD, Blunt JR., Retrieval practice produces more learning than elaborative studying with concept mapping. Science 2011 331 772 775
63. Mangen A, Walgermo BR, Bronnick K., Reading linear texts on paper versus computer screen: Effects on reading comprehension. Int J Educ Res 2013 58 61 68
64. Holmboe ES, Durning SJ., Assessing clinical reasoning: Moving from in vitro to in vivo. Diagnosis 2014 1 111 117
65. Parmelee DX., Michaelsen LK, Parmelee D, McMahon KK, Levine RE, Team-based learning for health professions education: Why is it a good fit? Team-Based Learning for Health Professions Education 2008 Sterling, Va Stylus 3 8
66. Durning SJ, Artino AR., Situativity theory: A perspective on how participants and the environment can interact: AMEE guide no. 52. Med Teach 2011 33 188 199
67. Kogan JR, Holmboe ES., Preparing residents for practice in new systems of care by preparing their teachers. Acad Med 2014 89 1436 1437
68. Hodges B., Assessment in the post-psychometric era: Learning to love the subjective and collective. Med Teach 2013 35 564 568
69. Kogan JR, Holmboe E., Realizing the promise and importance of performance-based assessment. Teach Learn Med 2013 25 suppl 1 S68 S74
70. Kogan JR, Conforti LN, Iobst WF, Holmboe ES., Reconceptualizing variable rater assessments as both an educational and clinical care problem. Acad Med 2014 89 721 727
71. National Cardiovascular Data Registry (NCDR) Published manuscripts based on NCDR registries. http://cvquality.acc.org/en/NCDR-Home/Research/cvquality.acc.org/∼/media/QII/NCDR/Published%20Research%20Page/NCDR%20manuscripts%20by%20registry.ashx. Accessed April 24, 2015
72. Barsuk JH, Cohen ER, Potts S, Dissemination of a simulation-based mastery learning intervention reduces central line-associated bloodstream infections. BMJ Qual Saf 2014 23 749 756
73. Holmboe E, Rizzolo MA, Sachdeva AK, Rosenberg M, Ziv A., Simulation-based assessment and the regulation of healthcare professionals. Simul Healthc 2011 6 suppl S58 S62
74. Johnson TR, Lyons R, Kopper R, Johnsen KJ, Lok BC, Cendan JC., Virtual patient simulations and optimal social learning context: A replication of an aptitude-treatment interaction effect. Med Teach 2014 36 486 494
75. Saleh N., The value of virtual patients in medical education. Ann Behav Sci Med Educ 2010 16 29 31
76. Folkman J., Commencement address. Geisel School of Medicine at Dartmouth
77. Nasca TJ, Philibert I, Brigham T, Flynn TC., The next GME accreditation system-rationale and benefits. N Engl J Med 2012 366 1051 1056
78. Weiss KB, Bagian JP, Nasca TJ., The clinical learning environment: The foundation of graduate medical education. JAMA 2013 309 1687 1688
79. Green ML, Aagaard EM, Caverzagie KJ, Charting the road to competence: Developmental milestones for internal medicine residency training. J Grad Med Educ 2009 1 5 20
80. Philibert I, Brigham T, Edgar L, Swing S., Organization of the educational milestones for use in the assessment of educational outcomes. J Grad Med Educ 2014 6 177 182
81. Engestrom Y, Sannino A., Studies of expansive learning: Foundations, findings and future challenges. Educ Res Rev 2010 5 1 24
82. Engestrom Y., New forms of expansive learning at work: The landscape of co-configuration. http://www.edu.helsinki.fi/activity/people/engestro/files/The-Finnish-proposal.pdf. Accessed April 24, 2015
83. Pawson R, Tilley N., Realistic Evaluation 1997 London, UK Sage Publications
84. Alford J., The multiple facets of co-production: Building on the work of Elinor Ostrom. Public Manage Rev 2014 16 299 316
85. Frank J, Jabbour M, Tugwell P, for the Societal Needs Working Group Skills for the New Millenium: Report of the Societal Needs Working Group 1996 Ottawa, Ontario, Canada Royal College of Physicians and Surgeons of Canada
86. Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S., General competencies and accreditation in graduate medical education. Health Aff (Millwood) 2002 21 103 111
87. Talbot M., Monkey see, monkey do: A critique of the competency model in graduate medical education. Med Educ 2004 38 587 592
88. Lurie SJ, Mooney CJ, Lyness JM., Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: A systematic review. Acad Med 2009 84 301 309
89. Norman G, Norcini J, Bordage G., Competency-based education: Milestones or millstones? J Grad Med Educ 2014 6 1 6
90. Glass JM., Competency based training is a framework for incompetence. BMJ 2014 348 g2909
91. Institute of Medicine Graduate Medical Education That Meets the Nation's Health Needs 2014 Washington, DC National Academies Press
92. Association of American Medical Colleges IOM's vision of GME will not meet real-world patient needs [news release]. 2014. https://www.aamc.org/newsroom/newsreleases/381882/07292014.html. Accessed April 24, 2015
93. McGaghie WC, Miller GE, Sajid AW, Telder TV, Lipson L., Competency-Based Curriculum Development in Medical Education: An Introduction 1978 Geneva, Switzerland World Health Organization. http://whqlibdoc.who.int/php/WHO-PHP-68.pdf. Accessed April 24, 2015
94. Frenk J, Chen L, Bhutta ZA, Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. Lancet 2010 376 1923 1958
95. Frank JR, Snell LS, Cate OT, Competency-based medical education: Theory to practice. Med Teach 2010 32 638 645
96. Graham DW., Heraclitus. Internet Encyclopedia of Philosophy. http://www.iep.utm.edu/heraclit/. Accessed May 1, 2015
97. Heifetz RA, Linsky M., Leadership on the Line: Staying Alive Through the Dangers of Leading 2002 Boston, Mass Harvard Business School Press
98. Deming WE., The New Economics 2000 2nd ed Cambridge, Mass MIT Press
99. Kegan R, Lahey LL., How the Way We Talk Can Change the Way We Work 2001 San Francisco, Calif Jossey-Bass
100. Lehmann LS, Brancati FL, Chen MC, Roter D, Dobs AS., The effect of bedside case presentations on patients' perceptions of their medical care. N Engl J Med 1997 336 1150 1155
101. Engel GL., Are medical schools neglecting clinical skills? JAMA 1976 236 861 863
102. Gonzalo JD, Chuang CH, Huang G, Smith C., The return of bedside rounds: An educational intervention. J Gen Intern Med 2010 25 792 798
103. Martin RL., The Opposable Mind: How Successful Leaders Win Through Integrative Thinking 2007 Boston, Mass Harvard Business School Publishing
104. Holcombe RF., Improving health care quality: A polarity management perspective. Acad Med 2015 90 259
105. Scharmer CO., Theory U: Leading From the Future As It Emerges 2007 Boston, Mass Society for Organizational Learning
106. Freire K, Sangiorgi D., Service design and healthcare innovation: From consumption to co-production and co-creation. 2010 Linkoping, Sweden Second Nordic Conference on Service Design and Service Innovation. http://www.servdes.org/pdf/freire-sangiorgi.pdf. Accessed April 24, 2015
107. Glouberman S, Zimmerman B., Complicated and complex systems: What would successful reform of Medicare look like? Discussion paper no. 8. Commission on the Future of Healthcare in Canada 2002. http://c.ymcdn.com/sites/www.plexusinstitute.org/resource/collection/6528ED29-9907-4BC7-8D00-8DC907679FED/ComplicatedAndComplexSystems-ZimmermanReport-Medicare-reform.pdf. Accessed April 24, 2015
108. Fuchs VR., The Service Economy 1968 Cambridge, Mass National Bureau of Economic Research. http://www.nber.org/books/fuch68-1. Accessed April 24, 2015
109. Bate P, Robert G., Experience-based design: From redesigning the system around the patient to co-designing services with the patient. Qual Saf Health Care 2006 15 307 310
110. Sabadosa KA, Batalden PB., The interdependent roles of patients, families and professionals in cystic fibrosis: A system for the coproduction of healthcare and its improvement. BMJ Qual Saf 2014 23 suppl 1 i90 i94
111. Normann R., Reframing Business: When the Map Changes the Landscape 2001 Chichester, UK Wiley Publishing
112. Batalden M, Leach D, Batalden P., Batalden P, Foster T, Chapter 4: Better professional development: Competence, mastery, pride and joy. Sustainably Improving Health Care: Creatively Linking Care Outcomes, System Performance, and Professional Development 2012 London, UK Radcliffe Publishing
113. Cooke M, Irby DM, O'Brien BC, Carnegie Foundation for the Advancement of Teaching Educating Physicians: A Call for Reform of Medical School and Residency 2010 San Francisco, Calif Jossey-Bass
114. Institute of Medicine Health Professions Education: A Bridge to Quality 2003 Washington, DC National Academies Press
115. MedPAC Graduate medical education financing: Focusing on educational priorities. Report to the Congress: Aligning Incentives in Medicare 2010 Washington, DC MedPAC 103 128. http://www.medpac.gov/documents/reports/Jun10-Ch04.pdf. Accessed April 24, 2015
116. Antiel RM, Van Arendonk KJ, Reed DA, Surgical training, duty-hour restrictions, and implications for meeting the Accreditation Council for Graduate Medical Education core competencies: Views of surgical interns compared with program directors. Arch Surg 2012 147 536 541
117. Ten Cate O., The false dichotomy of quality and quantity in the discourse around assessment in competency-based education [published online June 8, 2014]. Adv in Health Sci Educ
118. Division of Orthopedics, University of Toronto A practical approach to competency-based education in your residency program. http://www.competency-based-training.com/. May 1, 2015
119. Long DM., Competency-based residency training: The next advance in graduate medical education. Acad Med 2000 75 1178 1183
120. Hafferty FW., Duty hours, distant voices, and an alternative future. Arch Surg 2012 147 496 497
121. Holmboe ES., Faculty and the observation of trainees' clinical skills: Problems and opportunities. Acad Med 2004 79 16 22
122. Crosson FJ, Leu J, Roemer BM, Ross MN., Gaps in residency training should be addressed to better prepare doctors for a twenty-first-century delivery system. Health Aff (Millwood) 2011 30 2142 2148
123. Lypson ML, Frohna JG, Gruppen LD, Woolliscroft JO., Assessing residents' competencies at baseline: Identifying the gaps. Acad Med 2004 79 564 570
124. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR., The role of assessment in competency-based medical education. Med Teach 2010 32 676 682
125. Di Francesco L, Pistoria MJ, Auerbach AD, Nardino RJ, Holmboe ES., Internal medicine training in the inpatient setting. A review of published educational interventions. J Gen Intern Med 2005 20 1173 1180
126. Bernabeo EC, Holtman MC, Ginsburg S, Rosenbaum JR, Holmboe ES., Lost in transition: The experience and impact of frequent changes in the inpatient learning environment. Acad Med 2011 86 591 598
127. Zafar MA, Diers T, Schauer DP, Warm EJ., Connecting resident education to patient outcomes: The evolution of a quality improvement curriculum in an internal medicine residency. Acad Med 2014 89 1341 1347
128. Warm EJ., Interval examination: The ambulatory long block. J Gen Intern Med 2010 25 750 752
129. Hauer KE, Hirsh D, Ma I, The role of role: Learning in longitudinal integrated and traditional block clerkships. Med Educ 2012 46 698 710
130. Gaufberg E, Hirsh D, Krupat E, Into the future: Patient-centredness endures in longitudinal integrated clerkship graduates. Med Educ 2014 48 572 582
131. Commonwealth Medical College Longitudinal model. http://www.tcmc.edu/oth/Page.asp?PageID=OTH000478. Accessed April 24, 2015
132. Ostrom E., Crossing the great divide: Coproduction, synergy, and development. World Dev 1996 24 1073 1087
133. Garn HA., Models for Indicator Development: A Framework for Policy Analysis 1976 Washington, DC Urban Institute
134. Wagner EH, Austin BT, Von Korff M., Improving outcomes in chronic illness. Manag Care Q 1996 4 12 25
135. Molloy E, Boud D., Boud D, Molloy E, Changing conceptions of feedback. Feedback in Higher and Professional Education: Understanding It and Doing It Well 2013 New York, NY Routledge 11 33
136. Tamblyn R, McLeod P, Hanley JA, Girard N, Hurley J., Physician and practice characteristics associated with the early utilization of new prescription drugs. Med Care 2003 41 895 908
137. Asch DA, Nicholson S, Srinivas S, Herrin J, Epstein AJ., Evaluating obstetrical residency programs using patient outcomes. JAMA 2009 302 1277 1283
138. Sirovich BE, Lipner RS, Johnston M, Holmboe ES., The association between residency training and internists' ability to practice conservatively. JAMA Intern Med 2014 174 1640 1648
139. Duffy FD, Holmboe ES., Self-assessment in lifelong learning and improving performance in practice: Physician know thyself. JAMA 2006 296 1137 1139
140. Cook DA, Holmboe ES, Sorensen KJ, Berger RA, Wilkinson JM., Getting maintenance of certification to work: A grounded theory study of physicians' perceptions. JAMA Intern Med 2015 175 35 42
141. Sargeant J, Armson H, Chesluk B, Processes and dimensions of informed self-assessment. Acad Med 2010 85 1212 1220
142. Eva KW, Regehr G., Effective feedback for maintenance of competence: From data delivery to trusting dialogues. CMAJ 2013 185 463 464
143. Weise MR, Christensen CM., Hire education. Mastery, modularization, and the workforce revolution. http://www.christenseninstitute.org/publications/hire/. Accessed April 24, 2015
144. Khan Academy Health and medicine. http://www.khanacademy.org/science/health-and-medicine. Accessed April 24, 2015
145. Rohrer D, Pashler H., Recent research on human learning challenges conventional instructional strategies. Educ Res 2010 39 406 412
146. Karpicke JD, Blunt JR., Retrieval practice produces more learning than elaborative studying with concept mapping. Science 2011 331 772 775
147. Mangen A, Walgermo BR, Bronnick K., Reading linear texts on paper versus computer screen: Effects on reading comprehension. Int J Educ Res 2013 58 61 68
148. Holmboe ES, Durning SJ., Assessing clinical reasoning: Moving from in vitro to in vivo. Diagnosis 2014 1 111 117
149. Parmelee DX., Michaelsen LK, Parmelee D, McMahon KK, Levine RE, Team-based learning for health professions education: Why is it a good fit? Team-Based Learning for Health Professions Education 2008 Sterling, Va Stylus 3 8
150. Durning SJ, Artino AR., Situativity theory: A perspective on how participants and the environment can interact: AMEE guide no. 52. Med Teach 2011 33 188 199
151. Kogan JR, Holmboe ES., Preparing residents for practice in new systems of care by preparing their teachers. Acad Med 2014 89 1436 1437
152. Hodges B., Assessment in the post-psychometric era: Learning to love the subjective and collective. Med Teach 2013 35 564 568
153. Kogan JR, Holmboe E., Realizing the promise and importance of performance-based assessment. Teach Learn Med 2013 25 suppl 1 S68 S74
154. Kogan JR, Conforti LN, Iobst WF, Holmboe ES., Reconceptualizing variable rater assessments as both an educational and clinical care problem. Acad Med 2014 89 721 727
155. National Cardiovascular Data Registry (NCDR) Published manuscripts based on NCDR registries. http://cvquality.acc.org/en/NCDR-Home/Research/cvquality.acc.org/∼/media/QII/NCDR/Published%20Research%20Page/NCDR%20manuscripts%20by%20registry.ashx. Accessed April 24, 2015
156. Barsuk JH, Cohen ER, Potts S, Dissemination of a simulation-based mastery learning intervention reduces central line-associated bloodstream infections. BMJ Qual Saf 2014 23 749 756
157. Holmboe E, Rizzolo MA, Sachdeva AK, Rosenberg M, Ziv A., Simulation-based assessment and the regulation of healthcare professionals. Simul Healthc 2011 6 suppl S58 S62
158. Johnson TR, Lyons R, Kopper R, Johnsen KJ, Lok BC, Cendan JC., Virtual patient simulations and optimal social learning context: A replication of an aptitude-treatment interaction effect. Med Teach 2014 36 486 494
159. Saleh N., The value of virtual patients in medical education. Ann Behav Sci Med Educ 2010 16 29 31
160. Folkman J., Commencement address. Geisel School of Medicine at Dartmouth
161. Nasca TJ, Philibert I, Brigham T, Flynn TC., The next GME accreditation system-rationale and benefits. N Engl J Med 2012 366 1051 1056
162. Weiss KB, Bagian JP, Nasca TJ., The clinical learning environment: The foundation of graduate medical education. JAMA 2013 309 1687 1688
163. Green ML, Aagaard EM, Caverzagie KJ, Charting the road to competence: Developmental milestones for internal medicine residency training. J Grad Med Educ 2009 1 5 20
164. Philibert I, Brigham T, Edgar L, Swing S., Organization of the educational milestones for use in the assessment of educational outcomes. J Grad Med Educ 2014 6 177 182
165. Engestrom Y, Sannino A., Studies of expansive learning: Foundations, findings and future challenges. Educ Res Rev 2010 5 1 24
166. Engestrom Y., New forms of expansive learning at work: The landscape of co-configuration. http://www.edu.helsinki.fi/activity/people/engestro/files/The-Finnish-proposal.pdf. Accessed April 24, 2015
167. Pawson R, Tilley N., Realistic Evaluation 1997 London, UK Sage Publications
168. Alford J., The multiple facets of co-production: Building on the work of Elinor Ostrom. Public Manage Rev 2014 16 299 316