Linking simulation-based educational assessments and patient-related outcomes: A systematic review and meta-analysis

Academic Medicine

Volumn 90, Issue 2, 2015, Pages 246-256

  Brydges, Ryan ^a,b   Hatala, Rose ^c   Zendejas, Benjamin ^d   Erwin, Patricia J ^d   Cook, David A ^d  

^a UNIVERSITY OF TORONTO   (Canada)

^b UNIVERSITY HEALTH NETWORK   (Canada)

^c UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^d MAYO CLINIC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLINICAL COMPETENCE; COMPUTER SIMULATION; HUMAN; META ANALYSIS; OUTCOME ASSESSMENT; REPRODUCIBILITY; TEACHING;

CLINICAL COMPETENCE; COMPUTER SIMULATION; HUMANS; PATIENT OUTCOME ASSESSMENT; REPRODUCIBILITY OF RESULTS; TEACHING MATERIALS;

EID: 84922407219     PISSN: 10402446     EISSN: 1938808X     Source Type: Journal    
DOI: 10.1097/ACM.0000000000000549     Document Type: Review

References (83)

1. Royal College of Physicians and Surgeons of Canada. CanMEDS 2015: Planned updates to the CanMEDS framework. 2013. http://www.royalcollege.ca/portal/page/portal/rc/canmeds/canmeds2015/updates. Accessed September 8, 2014
2. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: Theory to practice. Med Teach. 2010;32: 638-645
3. Green ML, Aagaard EM, Caverzagie KJ, et al. Charting the road to competence: Developmental milestones for internal medicine residency training. J Grad Med Educ. 2009;1:5-20
4. American Board of Internal Medicine. Milestones. 2013. http://www.abim. org/program-directors-administrators/milestones.aspx. Accessed September 8, 2014
5. Wayne DB, Butter J, Siddall VJ, et al. Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice. J Gen Intern Med. 2006;21:251-256
6. Weinberger SE, Pereira AG, Iobst WF, Mechaber AJ, Bronze MS; Alliance for Academic Internal Medicine Education Redesign Task Force II. Competency-based education and training in internal medicine. Ann Intern Med. 2010;153:751-756
7. Cook DA, West CP. Perspective: Reconsidering the focus on outcomes research in medical education: A cautionary note. Acad Med. 2013;88:162-167
8. Norcini JJ, McKinley DW. Assessment methods in medical education. Teach Teach Educ. 2007;23:239-250
9. Shea JA. Mind the gap: Some reasons why medical education research is different from health services research. Med Educ. 2001;35:319-320
10. Dijksterhuis MG, Voorhuis M, Teunissen PW, et al. Assessment of competence and progressive independence in postgraduate clinical training. Med Educ. 2009;43: 1156-1165
11. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE guide no. 31. Med Teach. 2007;29:855-871
12. Yudkin JS, Lipska KJ, Montori VM. The idolatry of the surrogate. BMJ. 2011;343:d7995
13. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system-rationale and benefits. N Engl J Med. 2012;366:1051-1056
14. Philibert I, Nasca T, Brigham T, Shapiro J. Duty-hour limits and patient care and resident outcomes: Can high-quality studies offer insight into complex relationships?. Annu Rev Med. 2013;64:467-483
15. Kane MT. Validation as a pragmatic, scientific activity. J Educ Meas. 2013;50:115-122
16. Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R. Technology-enhanced simulation to assess health professionals: A systematic review of validity evidence, research methods, and reporting quality. Acad Med. 2013;88:872-883
17. Zendejas B, Brydges R, Wang AT, Cook DA. Patient outcomes in simulation-based medical education: A systematic review. J Gen Intern Med. 2013;28:1078-1089
18. Schmidt E, Goldhaber-Fiebert SN, Ho LA, McDonald KM. Simulation exercises as a patient safety strategy: A systematic review. Ann Intern Med. 2013;158(5 pt 2):426-432
19. Bucher HC, Guyatt GH, Cook DJ, Holbrook A, McAlister FA. Users' guides to the medical literature: XIX. Applying clinical trial results A. How to use an article measuring the effect of an intervention on surrogate end points. JAMA. 1999;282:771-778
20. Fleming TR, DeMets DL. Surrogate end points in clinical trials: Are we being misled?. Ann Intern Med. 1996;125:605-613
21. Boulet JR, Jeffries PR, Hatala RA, Korndorffer JR Jr, Feinstein DM, Roche JP. Research regarding methods of assessing learning outcomes. Simul Healthc. 2011;6(suppl): S48-S51
22. Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R. What counts as validity evidence?. Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract. 2014;19: 233-250
23. Downing SM. Validity: On meaningful interpretation of assessment data. Med Educ. 2003;37:830-837
24. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Ann Intern Med. 2009;151:264-269, W64
25. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: A systematic review and meta-analysis. JAMA. 2011;306:978-988
26. Dieckmann P, Phero JC, Issenberg SB, Kardong-Edgren S, Ostergaard D, Ringsted C. The first research consensus summit of the Society for Simulation in Healthcare: Conduction and a synthesis of the results. Simul Healthc. 2011;6(suppl):S1-S9
27. Hamstra SJ, Dubrowski A. Effective training and assessment of surgical skills, and the correlates of performance. Surg Innov. 2005;12:71-77
28. Issenberg SB, Ringsted C, Ostergaard D, Dieckmann P. Setting a research agenda for simulation-based healthcare education: A synthesis of the outcome from an Utstein style meeting. Simul Healthc. 2011;6:155-167
29. McGaghie WC. Medical education research as translational science. Sci Transl Med. 2010;2:19cm8
30. McGaghie WC, Draycott TJ, Dunn WF, Lopez CM, Stefanidis D. Evaluating the impact of simulation on translational patient outcomes. Simul Healthc. 2011;6(suppl):S42-S47
31. McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Translational educational research: A necessity for effective health-care improvement. Chest. 2012;142:1097-1103
32. Wayne DB, McGaghie WC. Use of simulation-based medical education to improve patient care quality. Resuscitation. 2010;81:1455-1456
33. Ahmed K, Jawad M, Abboudi M, et al. Effectiveness of procedural simulation in urology: A systematic review. J Urol. 2011;186:26-34
34. Byrne AJ, Greaves JD. Assessment instruments used during anaesthetic simulation: Review of published studies. Br J Anaesth. 2001;86:445-450
35. Edler AA, Fanning RG, Chen MI, et al. Patient simulation: A literary synthesis of assessment tools in anesthesiology. J Educ Eval Health Prof. 2009;6:3
36. Fitzgerald TN, Duffy AJ, Bell RL, Berman L, Longo WE, Roberts KE. Computer-based endoscopy simulation: Emerging roles in teaching and professional skills assessment. J Surg Educ. 2008;65:229-235
37. Kardong-Edgren S, Adamson KA, Fitzgerald C. A review of currently published evaluation instruments for human patient simulation. Clin Simul Nurs. 2010;6:e25-e35
38. Van Nortwick SS, Lendvay TS, Jensen AR, Wright AS, Horvath KD, Kim S. Methodologies for establishing validity in surgical simulation studies. Surgery. 2010;147:622-630
39. Cleland JA, Abe K, Rethans JJ. The use of simulated patients in medical education: AMEE guide no 42. Med Teach. 2009;31:477-486
40. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: A metaanalysis. JAMA. 2008;300:1181-1196
41. Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas. 1973;33:613-619
42. Cohen J. Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge Academic; 1988
43. Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298: 1002-1009
44. Borenstein M. Effect sizes for continuous data. In: Cooper H, Hedges LV, Valentine JC, eds. The Handbook of Research Synthesis and Meta Analysis. New York, NY: Russell Sage Foundation; 2009
45. Peterson RA, Brown SP. On the use of beta coefficients in meta-analysis. J Appl Psychol. 2005;90:175-181
46. Terrin N, Schmid CH, Lau J, Olkin I. Adjusting for publication bias in the presence of heterogeneity. Stat Med. 2003;22:2113-2126
47. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in metaanalyses. BMJ. 2003;327:557-560
48. Wohaibi EM, Bush RW, Earle DB, Seymour NE. Surgical resident performance on a virtual reality simulator correlates with operating room performance. J Surg Res. 2010;160:67-72
49. Scerbo MW, Schmidt EA, Bliss JP. Comparison of a virtual reality simulator and simulated limbs for phlebotomy training. J Infus Nurs. 2006;29:214-224
50. Banerjee PP, Edward DP, Liang S, et al. Concurrent and face validity of a capsulorhexis simulation with respect to human patients. Stud Health Technol Inform. 2012;173:35-41
51. Banks EH, Chudnoff S, Karmin I, Wang C, Pardanani S. Does a surgical simulator improve resident operative performance of laparoscopic tubal ligation?. Am J Obstet Gynecol. 2007;197:541.e1-541.e5
52. Beard JD, Jolly BC, Newble DI, Thomas WE, Donnelly J, Southgate LJ. Assessing the technical skills of surgical trainees. Br J Surg. 2005;92:778-782
53. Crabtree NA, Chandra DB, Weiss ID, Joo HS, Naik VN. Fibreoptic airway training: Correlation of simulator performance and clinical skill. Can J Anaesth. 2008;55:100-104
54. Curtis DA, Lind SL, Brear S, Finzen FC. The correlation of student performance in preclinical and clinical prosthodontic assessments. J Dent Educ. 2007;71:365-372
55. Datta V, Bann S, Beard J, Mandalia M, Darzi A. Comparison of bench test evaluations of surgical skill with live operating performance assessments. J Am Coll Surg. 2004;199:603-606
56. Ende A, Zopf Y, Konturek P, et al. Strategies for training in diagnostic upper endoscopy: A prospective, randomized trial. Gastrointest Endosc. 2012;75:254-260
57. Faulkner H, Regehr G, Martin J, Reznick R. Validation of an objective structured assessment of technical skill for surgical residents. Acad Med. 1996;71:1363-1365
58. Feldman LS, Hagarty SE, Ghitulescu G, Stanbridge D, Fried GM. Relationship between objective assessment of technical skills and subjective in-training evaluations in surgical residents. J Am Coll Surg. 2004;198:105-110
59. Fried GM, Feldman LS, Vassiliou MC, et al. Proving the value of simulation in laparoscopic surgery. Ann Surg. 2004;240:518-525
60. Gale TC, Roberts MJ, Sice PJ, et al. Predictive validity of a selection centre testing nontechnical skills for recruitment to training in anaesthesia. Br J Anaesth. 2010;105:603-609
61. Ghaderi I, Vaillancourt M, Sroka G, et al. Performance of simulated laparoscopic incisional hernia repair correlates with operating room performance. Am J Surg. 2011;201:40-45
62. Jones T, Cason CL, Mancini ME. Evaluating nurse competency: Evidence of validity for a skills recredentialing program. J Prof Nurs. 2002;18:22-28
63. Kessler DO, Auerbach M, Pusic M, Tunik MG, Foltin JC. A randomized trial of simulation-based deliberate practice for infant lumbar puncture skills. Simul Healthc. 2011;6:197-203
64. Kundhal PS, Grantcharov TP. Psychomotor performance measured in a virtual environment correlates with technical skills in the operating room. Surg Endosc. 2009;23:645-649
65. Kurashima Y, Feldman LS, Al-Sabah S, Kaneva PA, Fried GM, Vassiliou MC. A tool for training and evaluation of laparoscopic inguinal hernia repair: The Global Operative Assessment Of Laparoscopic Skills-Groin Hernia (GOALS-GH). Am J Surg. 2011;201:54-61
66. Macmillan AI, Cuschieri A. Assessment of innate ability and skills for endoscopic manipulations by the Advanced Dundee Endoscopic Psychomotor Tester: Predictive and concurrent validity. Am J Surg. 1999;177:274-277
67. Matsuda T, McDougall EM, Ono Y, et al. Positive correlation between motion analysis data on the LapMentor virtual reality laparoscopic surgical simulator and the results from videotape assessment of real laparoscopic surgeries. J Endourol. 2012;26:1506-1511
68. McCluney AL, Vassiliou MC, Kaneva PA, et al. FLS simulator performance predicts intraoperative laparoscopic skill. Surg Endosc. 2007;21:1991-1995
69. Morgan PJ, Cleave-Hogg D. Evaluation of medical students' performance using the anaesthesia simulator. Med Educ. 2000;34:42-45
70. Morgan PJ, Cleave-Hogg D, DeSousa S, Tarshis J. High-fidelity patient simulation: Validation of performance checklists. Br J Anaesth. 2004;92:388-392
71. Morgan PJ, Cleave-Hogg DM, Guest CB, Herold J. Validity and reliability of undergraduate performance assessments in an anesthesia simulator. Can J Anaesth. 2001;48:225-233
72. Mudumbai SC, Gaba DM, Boulet JR, Howard SK, Davies MF. External validation of simulation-based assessments with other performance measures of third-year anesthesiology residents. Simul Healthc. 2012;7:73-80
73. Nunez DW, Taleghani M, Wathen WF, Abdellatif HM. Typodont versus live patient: Predicting dental students' clinical performance. J Dent Educ. 2012;76:407-413
74. Paisley AM, Baldwin PJ, Paterson-Brown S. Validity of surgical simulation for the assessment of operative skill. Br J Surg. 2001;88:1525-1532
75. Schwid HA, Rooke GA, Carline J, et al; Anesthesia Simulator Research Consortium. Evaluation of anesthesia residents using mannequin-based simulation: A multiinstitutional study. Anesthesiology. 2002;97:1434-1444
76. Sedlack RE, Baron TH, Downing SM, Schwartz AJ. Validation of a colonoscopy simulation model for skills assessment. Am J Gastroenterol. 2007;102:64-74
77. Stitik TP, Foye PM, Nadler SF, Chen B, Schoenherr L, Von Hagen S. Injections in patients with osteoarthritis and other musculoskeletal disorders: Use of synthetic injection models for teaching physiatry residents. Am J Phys Med Rehabil. 2005;84:550-559
78. Sugiono M, Teber D, Anghel G, et al. Assessing the predictive validity and efficacy of a multimodal training programme for laparoscopic radical prostatectomy (LRP). Eur Urol. 2007;51:1332-1339
79. Wilasrusmee C, Lertsithichai P, Kittur DS. Vascular anastomosis model: Relation between competency in a laboratory-based model and surgical competency. Eur J Vasc Endovasc Surg. 2007;34:405-410
80. Yudkowsky R, Luciano C, Banerjee P, et al. Practice on an augmented reality/haptic simulator and library of virtual brains improves residents' ability to perform a ventriculostomy. Simul Healthc. 2013;8:25-31
81. Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. JAMA. 2009;302: 1316-1326
82. Cook DA, Brydges R, Hamstra SJ, et al. Comparative effectiveness of technologyenhanced simulation versus other instructional methods: A systematic review and meta-analysis. Simul Healthc. 2012;7:308-320
83. Cook DA, Hamstra SJ, Brydges R, et al. Comparative effectiveness of instructional design features in simulation-based education: Systematic review and metaanalysis. Med Teach. 2013;35:e867-e898