Assessing technical competence in surgical trainees: A systematic review

Annals of Surgery

Volumn 261, Issue 6, 2015, Pages 1046-1055

  Szasz, Peter ^a,d   Louridas, Marisa ^a   Harris, Kenneth A ^b   Aggarwal, Rajesh ^c   Grantcharov, Teodor P ^a  

^a UNIVERSITY OF TORONTO   (Canada)

^b Royal College of Physicians Surgeons Canada   (Canada)

^c UNIVERSITY OF PENNSYLVANIA   (United States)

^d ST MICHAEL'S HOSPITAL   (Canada)

Author keywords

Competency based assessment; Competency based education; Milestones; Surgical education; Systematic review; Technical competence

Indexed keywords

ANALYTICAL ERROR; CHOLECYSTECTOMY; CLINICAL COMPETENCE; COLONOSCOPY; CURRICULUM; HUMAN; LIKERT SCALE; OPERATING ROOM PERSONNEL; PRIORITY JOURNAL; PROFESSIONAL COMPETENCE; RESIDENT; REVIEW; SURGICAL TRAINING; SYSTEMATIC REVIEW; VASCULAR SURGERY; EDUCATION; GENERAL SURGERY; MEDICAL EDUCATION; REPRODUCIBILITY; STANDARDS; SURGERY;

CLINICAL COMPETENCE; COMPETENCY-BASED EDUCATION; EDUCATIONAL MEASUREMENT; GENERAL SURGERY; HUMANS; INTERNSHIP AND RESIDENCY; REPRODUCIBILITY OF RESULTS; SURGICAL PROCEDURES, OPERATIVE;

EID: 84946499569     PISSN: 00034932     EISSN: 15281140     Source Type: Journal    
DOI: 10.1097/SLA.0000000000000866     Document Type: Review

References (156)

1. Pellegrini CA. Surgical education in the United States: navigating the white waters. Ann Surg. 2006;244:335-342.
2. Debas HT, Bass BL, Brennan MF, et al. American Surgical Association blue ribbon committee report on surgical education: 2004. Ann Surg. 2005;241: 1-8.
3. Peracchia A. Presidential address: surgical education in the third millennium. Ann Surg. 2001;234:709-712.
4. Warnock GL. Preparing Canadian surgeons to provide care in the 21st century. Can J Surg. 2012;55:219-220.
5. Sachdeva AK, Bell RH, Jr, Britt LD, et al. National efforts to reformresidency education in surgery. Acad Med. 2007;82:1200-1210.
6. Pellegrini CA, Warshaw AL, Debas HT. Residency training in surgery in the 21st century: a new paradigm. Surgery. 2004;136:953-965.
7. Pellegrini CA. Surgical education in the United States 2010: developing intellectual, technical and human values. Updates Surg. 2012;64:1-3.
8. Darzi A, Datta V, Mackay S. The challenge of objective assessment of surgical skill. Am J Surg. 2001;181:484-486.
9. Smith R. All changed, changed utterly. British medicine will be transformed by the Bristol case. BMJ. 1998;316:1917-1918.
10. Darzi A. Quality and the NHS next stage review. Lancet. 2008;371:1563-1564.
11. Kohn LT, Corrigan JM, Donaldson MS. To Err Is Human: Building a Safer Health System. Washington, DC: Institute of Medicine, National Academy of Sciences; 1999.
12. Drolet BC, Sangisetty S, Tracy TF, et al. Surgical residents' perceptions of 2011 Accreditation Council for Graduate Medical Education duty hour regulations. JAMA Surg. 2013;148:427-433.
13. Villaneuva T. European working time directive faces challenges. CMAJ. 2010;182:E39-E40.
14. Barden CB, Specht MC, McCarter MD, et al. Effects of limited work hours on surgical training. J Am Coll Surg. 2002;195:531-538.
15. Bridges M, Diamond DL. The financial impact of teaching surgical residents in the operating room. Am J Surg. 1999;177:28-32.
16. Hosler MR, Scott IU, Kunselman AR, et al. Impact of resident participation in cataract surgery on operative time and cost. Ophthalmology. 2012;119:95-98.
17. Good DW, Khan N, Kiely E, et al. The impact of rolling theatre closures on core urology training. Ir Med J. 2013;106:149-151.
18. Berry W. Surgical malpractice: myths and realities. J Med Pract Manage. 2006;22:50-51.
19. Ziv A, Wolpe PR, Small SD, et al. Simulation-based medical education: an ethical imperative. Acad Med. 2003;78:783-788.
20. Next accreditation system (NAS) milestones. Accreditation Council for Graduate Medical Education (ACGME) Web site. Available at: https://www.acgme.org/acgmeweb/tabid/430/ProgramandInstitutionalAccreditation/Next AccreditationSystem/Milestones.aspx. Published 2012. Accessed October 8, 2013.
21. CanMEDS. Royal College of Physicians and Surgeons of Canada Web site. Available at: http://www.royalcollege.ca/portal/page/portal/rc/canmeds. Published 2013. Accessed August 15, 2013.
22. Intercollegiate surgical curriculum programme. Royal College of Surgeons of England Web site. Available at: https://http://www.iscp.ac.uk/. Published 2013. Accessed October 1, 2013.
23. Surgical education and training policies. Royal Australasian College of Surgeons Web site. Available at: http://www.surgeons.org/policies-publications/policies/surgical-education-and-training/. Published 2013. Accessed September 1, 2013.
24. Competency-based medical education. Royal College of Physicians and Surgeons of Canada Web site. Available at: http://www.royalcollege.ca/portal/page/portal/rc/resources/cbme. Published 2013. Accessed September 1, 2013.
25. Schindler N, Corcoran J, DaRosa D. Description and impact of using a standard-setting method for determining pass/fail scores in a surgery clerkship. Am J Surg. 2007;193:252-257.
26. Wilkinson TJ, Newble DI, Frampton CM. Standard setting in an objective structured clinical examination: use of global ratings of borderline performance to determine the passing score. Med Educ. 2001;35:1043-1049.
27. Norcini JJ. Setting standards on educational tests. Med Educ. 2003;37: 464-469.
28. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.
29. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924-926.
30. Guyatt GH, Oxman AD, Kunz R, et al. What is "quality of evidence" and why is it important to clinicians? BMJ. 2008;336:995-998.
31. Swanstrom LL, Fried GM, Hoffman KI, et al. Beta test results of a new system assessing competence in laparoscopic surgery. J Am Coll Surg. 2006;202: 62-69.
32. Fraser SA, Klassen DR, Feldman LS, et al. Evaluating laparoscopic skills: setting the pass/fail score for the MISTELS system. Surg Endosc. 2003;17: 964-967.
33. Sroka G, Feldman LS, Vassiliou MC, et al. Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room-a randomized controlled trial. Am J Surg. 2010;199:115-120.
34. McCluney AL, Vassiliou MC, Kaneva PA, et al. FLS simulator performance predicts intraoperative laparoscopic skill. Surg Endosc. 2007;21:1991-1995.
35. Jelovsek JE, Walters MD, Korn A, et al. Establishing cutoff scores on assessments of surgical skills to determine surgical competence. Am J Obstet Gynecol. 2010;203:81.e1-e6.
36. Beard JD. Setting standards for the assessment of operative competence. Eur J Vasc Endovasc Surg. 2005;30:215-218.
37. Reznick R, Regehr G, MacRae H, et al. Testing technical skill via an innovative "bench station" examination. Am J Surg. 1997;173:226-230.
38. Hogle NJ, Widmann WD, Ude AO, et al. Does training novices to criteria and does rapid acquisition of skills on laparoscopic simulators have predictive validity or are we just playing video games? J Surg Educ. 2008;65:431-435.
39. Palter VN, Grantcharov T, Harvey A, et al. Ex vivo technical skills training transfers to the operating room and enhances cognitive learning: a randomized controlled trial. Ann Surg. 2011;253:886-889.
40. Hernandez JD, Bann SD, Munz Y, et al. Qualitative and quantitative analysis of the learning curve of a simulated surgical task on the da Vinci system. Surg Endosc. 2004;18:372-378.
41. Chou B, Bowen CW, Handa VL. Evaluating the competency of gynecology residents in the operating room: validation of a new assessment tool. Am J Obstet Gynecol. 2008;199:571.e1-e5.
42. Syme-Grant J, White PS, McAleer JP. Measuring competence in endoscopic sinus surgery. Surgeon. 2008;6:37-44.
43. Hance J, Aggarwal R, Stanbridge R, et al. Objective assessment of technical skills in cardiac surgery. Eur J Cardiothorac Surg. 2005;28:157-162.
44. Maizels M, Yerkes EB, Macejko A, et al. A new computer enhanced visual learning method to train urology residents in pediatric orchiopexy: a prototype for Accreditation Council for Graduate Medical Education documentation. J Urol. 2008;180:1814-1818; discussion 1818.
45. Francis HW, Masood H, Laeeq K, et al. Defining milestones toward competency in mastoidectomy using a skills assessment paradigm. Laryngoscope. 2010;120:1417-1421.
46. Fargo MV, Edwards JA, Roth BJ, et al. Using a simulated surgical skills station to assess lacerationmanagement by surgical and nonsurgical residents. J Grad Med Educ. 2011;3:326-331.
47. Orzech N, Palter VN, Reznick RK, et al. A comparison of 2 ex vivo training curricula for advanced laparoscopic skills: a randomized controlled trial. Ann Surg. 2012;255:833-839.
48. Vassiliou MC, Feldman LS, Andrew CG, et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg. 2005;190: 107-113.
49. Palter VN, Grantcharov TP. Development and validation of a comprehensive curriculum to teach an advanced minimally invasive procedure: a randomized controlled trial. Ann Surg. 2012;256:25-32.
50. Palter VN, Orzech N, Reznick RK, et al. Validation of a structured training and assessment curriculum for technical skill acquisition in minimally invasive surgery: a randomized controlled trial. Ann Surg. 2013;257:224-230.
51. Taylor JB, Binenbaum G, Tapino P, et al. Microsurgical lab testing is a reliable method for assessing ophthalmology residents' surgical skills. Br J Ophthalmol. 2007;91:1691-1694.
52. Insel A, Carofino B, Leger R, et al. The development of an objective model to assess arthroscopic performance. J Bone Joint Surg Am. 2009;91:2287-2295.
53. Howells NR, Gill HS, Carr AJ, et al. Transferring simulated arthroscopic skills to the operating theatre: a randomised blinded study. J Bone Joint Surg Br. 2008;90:494-499.
54. Beard JD, Choksy S, Khan S. Assessment of operative competence during carotid endarterectomy. Br J Surg. 2007;94:726-730.
55. Pandey V, Wolfe JH, Moorthy K, et al. Technical skills continue to improve beyond surgical training. J Vasc Surg. 2006;43:539-545.
56. Ahmed A, Ishman SL, Laeeq K, et al. Assessment of improvement of trainee surgical skills in the operating room for tonsillectomy. Laryngoscope. 2013;123:1639-1644.
57. Varela DAD, Malik MU, Pandian V, Cummings CW, Bhatti NI. Comparing performance between male and female residents in otolaryngology. Paper presented at Triological Society's 2011 annual meeting, Vol. 121: Laryngoscope 2011, Chicago, IL. p. S247.
58. Ishman SL, Brown DJ, Boss EF, et al. Development and pilot testing of an operative competency assessment tool for pediatric direct laryngoscopy and rigid bronchoscopy. Laryngoscope. 2010;120:2294-2300.
59. Lin SY, Laeeq K, Ishii M, et al. Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. Am J Rhinol Allergy. 2009;23:354-359.
60. Laeeq K, Waseem R, Weatherly RA, et al. In-training assessment and predictors of competency in endoscopic sinus surgery. Laryngoscope. 2010;120:2540-2545.
61. Laeeq K, Bhatti NI, Carey JP, et al. Pilot testing of an assessment tool for competency in mastoidectomy. Laryngoscope. 2009;119:2402-2410.
62. Stack BC, Jr, Siegel E, Bodenner D, et al. A study of resident proficiency with thyroid surgery: creation of a thyroid-specific tool. Otolaryngol Head Neck Surg. 2010;142:856-862.
63. Malik MU, Diaz DA, Varela V, et al. Using a virtual reality temporal bone simulator to enhance surgical competency in procedural tasks: a pilot study. Laryngoscope. 2011;121:S186.
64. Fleming J, Kapoor K, Sevdalis N, et al. Validation of an operating room immersive microlaryngoscopy simulator. Laryngoscope. 2012;122:1099-1103.
65. Wade TJ, Webb TP. Tackling technical skills competency: a surgical skills rating tool. J Surg Res. 2013;181:1-5.
66. Rooney DM, Hungness ES, Darosa DA, et al. Can skills coaches be used to assess resident performance in the skills laboratory? Surgery. 2012;151:796-802.
67. Khan MS, Bann SD, Darzi A, et al. Use of suturing as a measure of technical competence. Ann Plast Surg. 2003;50:304-308; discussion 308-309.
68. Chipman JG, Schmitz CC. Using objective structured assessment of technical skills to evaluate a basic skills simulation curriculum for first-year surgical residents. J Am Coll Surg. 2009;209:364-370.e2.
69. Sanfey H, Ketchum J, Bartlett J, et al. Verification of proficiency in basic skills for postgraduate year 1 residents. Surgery. 2010;148:759-766; discussion 766-767.
70. Mitchell EL, Lee DY, Sevdalis N, et al. Evaluation of distributed practice schedules on retention of a newly acquired surgical skill: a randomized trial. Am J Surg. 2011;201:31-39.
71. Gauger PG, Hauge LS, Andreatta PB, et al. Laparoscopic simulation training with proficiency targets improves practice and performance of novice surgeons. Am J Surg. 2010;199:72-80.
72. Brunner WC, Korndorffer JR, Jr, Sierra R, et al. Determining standards for laparoscopic proficiency using virtual reality. Am Surg. 2005;71:29-35.
73. Sutton E, Chase SC, Klein R, et al. Development of simulator guidelines for resident assessment in flexible endoscopy. Am Surg. 2013;79:14-22.
74. Selvan B, Cha A, Morris JB, et al. Endoscopic skills level-do we need to set standards and evaluation methods for a safe training and practices? Surg Endosc. 2011;25:S241-S372.
75. van Dongen KW, Ahlberg G, Bonavina L, et al. European consensus on a competency-based virtual reality training program for basic endoscopic surgical psychomotor skills. Surg Endosc. 2011;25:166-171.
76. Zendejas B, Cook DA, Hernandez-Irizarry R, et al. Mastery learning simulation-based curriculum for laparoscopic TEP inguinal hernia repair. J Surg Educ. 2012;69:208-214.
77. Stefanidis D, Acker CE, Greene FL. Performance goals on simulators boost resident motivation and skills laboratory attendance. J Surg Educ. 2010;67:66-70.
78. Jenison EL, Gil KM, Lendvay TS, et al. Robotic surgical skills: acquisition, maintenance, and degradation. JSLS. 2012;16:218-228.
79. Grantcharov TP, Funch-Jensen P. Can everyone achieve proficiency with the laparoscopic technique? Learning curve patterns in technical skills acquisition. Am J Surg. 2009;197:447-449.
80. Korndorffer JR, Jr, Dunne JB, Sierra R, et al. Simulator training for laparoscopic suturing using performance goals translates to the operating room. J Am Coll Surg. 2005;201:23-29.
81. Salgado J, Grantcharov TP, Papasavas PK, et al. Technical skills assessment as part of the selection process for a fellowship in minimally invasive surgery. Surg Endosc. 2009;23:641-644.
82. Mashaud LB, Castellvi AO, Hollett LA, et al. Two-year skill retention and certification exam performance after fundamentals of laparoscopic skills training and proficiency maintenance. Surgery. 2010;148:194-201.
83. Lendvay TS, Brand TC, White L, et al. Virtual reality robotic surgery warmup improves task performance in a dry laboratory environment: a prospective randomized controlled study. J Am Coll Surg. 2013;216:1181-1192.
84. Verdaasdonk EG, Dankelman J, Lange JF, et al. Incorporation of proficiency criteria for basic laparoscopic skills training: how does it work? Surg Endosc. 2008;22:2609-2615.
85. Kolkman W, Van de Put MA, Van den Hout WB, et al. Implementation of the laparoscopic simulator in a gynecological residency curriculum. Surg Endosc. 2007;21:1363-1368.
86. Moore AK, Grow DR, Bush RW, et al. Novices outperform experienced laparoscopists on virtual reality laparoscopy simulator. JSLS. 2008;12:358-362.
87. Fried MP, Kaye RJ, Gibber MJ, et al. Criterion-based (proficiency) training to improve surgical performance. Arch Otolaryngol Head Neck Surg. 2012;138:1024-1029.
88. Scott DJ, Goova MT, Tesfay ST. Acost-effective proficiency-based knot-tying and suturing curriculum for residency programs. J Surg Res. 2007;141:7-15.
89. Debes AJ, Aggarwal R, Balasundaram I, et al. Construction of an evidencebased, graduated training curriculum for D-box, awebcam-based laparoscopic basic skills trainer box. Am J Surg. 2012;203:768-775.
90. Brinkman WM, Buzink SN, Alevizos L, et al. Criterion-based laparoscopic training reduces total training time. Surg Endosc. 2012;26:1095-1101.
91. Korndorffer JR, Jr, Scott DJ, Sierra R, et al. Developing and testing competency levels for laparoscopic skills training. Arch Surg. 2005;140:80-84.
92. Nugent E, Shirilla N, Hafeez A, et al. Development and evaluation of a simulator-based laparoscopic training program for surgical novices. Surg Endosc. 2013;27:214-221.
93. Goova MT, Hollett LA, Tesfay ST, et al. Implementation, construct validity, and benefit of a proficiency-based knot-tying and suturing curriculum. J Surg Educ. 2008;65:309-315.
94. Stefanidis D, Sierra R, Korndorffer JR, Jr, et al. Intensive continuing medical education course training on simulators results in proficiency for laparoscopic suturing. Am J Surg. 2006;191:23-27.
95. Perrenot C, Manuela P, Nguyen T, et al. Results of a proficiency-based curriculum with the virtual reality robotic surgery simulator DV-Trainer. Int J Med Robot Comput Assist Surg. 2011;S2:32-66.
96. Shane MD, Pettitt BJ, Morgenthal CB, et al. Should surgical novices trade their retractors for joysticks? Videogame experience decreases the time needed to acquire surgical skills. Surg Endosc. 2008;22:1294-1297.
97. Stefanidis D, Korndorffer JR, Jr, Sierra R, et al. Skill retention following proficiency-based laparoscopic simulator training. Surgery. 2005;138:165-170.
98. Conway NE, Seymour NE, Bush RW, et al. Surgical resident learning curve for a simulated single port laparoscopic surgical task. Surg Endosc. 2011;25:S235-S240.
99. von Websky MW, Vitz M, Raptis DA, et al. Basic laparoscopic training using the Simbionix LAP Mentor: setting the standards in the novice group. J Surg Educ. 2012;69:459-467.
100. Brydges R, Classen R, Larmer J, et al. Computer-assisted assessment of one-handed knot tying skills performed within various contexts: a construct validity study. Am J Surg. 2006;192:109-113.
101. Ganju A, Kahol K, Lee P, et al. The effect of call on neurosurgery residents' skills: implications for policy regarding resident call periods. J Neurosurg. 2012;116:478-482.
102. Adrales GL, Park AE, Chu UB, et al. A valid method of laparoscopic simulation training and competence assessment. J Surg Res. 2003;114:156-162.
103. Adrales GL, Donnelly MB, Chu UB, et al. Determinants of competency judgments by experienced laparoscopic surgeons. Surg Endosc. 2004;18:323-327.
104. Adrales GL, Chu UB, Witzke DB, et al. Evaluatingminimally invasive surgery training using low-cost mechanical simulations. Surg Endosc. 2003;17:580-585.
105. Goff B, Mandel L, Lentz G, et al. Assessment of resident surgical skills: is testing feasible? Am J Obstet Gynecol. 2005;192:1331-1338; discussion 1338-1340.
106. Bath AP, Wilson T. Objective assessment of surgical competency-ENT trainees. Clin Otolaryngol. 2007;32:475-479.
107. Martin M, Vashisht B, Frezza E, et al. Competency-based instruction in critical invasive skills improves both resident performance and patient safety. Surgery. 1998;124:313-317.
108. Parent RJ, Plerhoples TA, Long EE, et al. Early, intermediate, and late effects of a surgical skills "boot camp" on an objective structured assessment of technical skills: a randomized controlled study. J Am Coll Surg. 2010;210:984-989.
109. Taravella MJ, Davidson R, Erlanger M, et al. Characterizing the learning curve in phacoemulsification. J Cataract Refract Surg. 2011;37:1069-1075.
110. Proctor DD, Price J, Dunn KA, et al. Prospective evaluation of a teaching model to determine competency in performing flexible sigmoidoscopies. Am J Gastroenterol. 1998;93:1217-1221.
111. Wilasrusmee C, Lertsithichai P, Kittur DS. Vascular anastomosis model: relation between competency in a laboratory-based model and surgical competency. Eur J Vasc Endovasc Surg. 2007;34:405-410.
112. Velmahos GC, Toutouzas KG, Sillin LF, et al. Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory. Am J Surg. 2004;187:114-119.
113. Miskovic D, Wyles SM, Carter F, et al. Development, validation and implementation of a monitoring tool for training in laparoscopic colorectal surgery in the English National Training Program. Surg Endosc. 2011;25:1136-1142.
114. Gofton WT, Dudek NL, Wood TJ, et al. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Acad Med. 2012;87:1401-1407.
115. Hawes R, Lehman GA, Hast J, et al. Training resident physicians in fiberoptic sigmoidoscopy. How many supervised examinations are required to achieve competence? Am J Med. 1986;80:465-470.
116. Ahlberg G, Enochsson L, Gallagher AG, et al. Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg. 2007;193:797-804.
117. Matsuda K, Yoshida Y, Kawahara Y, et al. How fast does a surgical resident learn colonoscopy?-Analysis of competency in colonoscopy for ten surgical residents supervised by expert endoscopists (gastroenterologists) in the highvolume endoscopy center in Japan. Gastrointest Endosc. 2012;75:489.
118. Zetlitz E, Wearing SC, Nicol A, et al. Objective assessment of surgical training in flexor tendon repair: the utility of a low-cost porcine model as demonstrated by a single-subject research design. J Surg Educ. 2012;69:504-510.
119. Bhatti NI, Cummings CW. Competency in surgical residency training: defining and raising the bar. Acad Med. 2007;82:569-573.
120. Hall JC, Crebbin W, Ellison A. Towards a hybrid philosophy of surgical education. ANZ J Surg. 2004;74:908-911.
121. van der Vleuten CP, Schuwirth LW, Scheele F, et al. The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol. 2010;24:703-719.
122. Grober ED, Jewett MA. The concept and trajectory of "operative competence" in surgical training. Can J Surg. 2006;49:238-240.
123. Thomas WE. Teaching and assessing surgical competence. Ann R Coll Surg Engl. 2006;88:429-432.
124. Mohr J, Batalden P, Barach P. Integrating patient safety into the clinical microsystem. Qual Saf Health Care. 2004;13(suppl 2):ii34-ii38.
125. Karamichalis JM, Barach PR, Nathan M, et al. Assessment of technical competency in pediatric cardiac surgery. Prog Pediatr Cardiol. 2012;33: 15-20.
126. Dreyfus SE, Dreyfus HL. A Five-Stage Model of the Mental Activities Involved in Directed Skill Acquisition. Berkeley, CA: University of California, Berkeley; 1980.
127. Proffitt M, ed. The Oxford English Dictionary Second Edition. Oxford, United Kingdom: Oxford University Press; 1989.
128. Babcock P, ed. Webster's Third New International Dictionary. Cambridge, MA: Merriam-Webster Inc; 1961.
129. Gallagher AG, Ritter EM, Champion H, et al. Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg. 2005;241:364-372.
130. Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84:273-278.
131. Faulkner H, Regehr G, Martin J, et al. Validation of an objective structured assessment of technical skill for surgical residents. Acad Med. 1996;71:1363-1365.
132. MacRae H, Regehr G, Leadbetter W, et al. A comprehensive examination for senior surgical residents. Am J Surg. 2000;179:190-193.
133. Scott DJ, Valentine RJ, Bergen PC, et al. Evaluating surgical competency with the American Board of Surgery In-Training Examination, skill testing, and intraoperative assessment. Surgery. 2000;128:613-622.
134. Paisley AM, Baldwin PJ, Paterson-Brown S. Validity of surgical simulation for the assessment of operative skill. Br J Surg. 2001;88:1525-1532.
135. van Hove PD, Tuijthof GJ, Verdaasdonk EG, et al. Objective assessment of technical surgical skills. Br J Surg. 2010;97:972-987.
136. Ahmed K, Miskovic D, Darzi A, et al. Observational tools for assessment of procedural skills: a systematic review. Am J Surg. 2011;202:469-480.e6.
137. Faurie C, Khadra M. Technical competence in surgeons. ANZ J Surg. 2012;82:682-690.
138. Camp RC. Learning from the best leads to superior performance. J Bus Strategy. 1992;13:3-6.
139. Kiefe CI, Weissman NW, Allison JJ, et al. Identifying achievable benchmarks of care: concepts and methodology. Int J Qual Health Care. 1998;10:443-447.
140. Weissman NW, Allison JJ, Kiefe CI, et al. Achievable benchmarks of care: the ABCs of benchmarking. J Eval Clin Pract. 1999;5:269-281.
141. Berkey T. Benchmarking in health care: turning challenges into success. Jt Comm J Qual Improv. 1994;20:277-284.
142. Berwick DM. Continuous improvement as an ideal in health care. N Engl J Med. 1989;320:53-56.
143. Hahn U, Krummenauer F, Kolbl B, et al. Determination of valid benchmarks for outcome indicators in cataract surgery: a multicenter, prospective cohort trial. Ophthalmology. 2011;118:2105-2112.
144. Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002;236:458-463; discussion 463-464.
145. Sloan DA, Donnelly MB, Schwartz RW, et al. The objective structured clinical examination. The new gold standard for evaluating postgraduate clinical performance. Ann Surg. 1995;222:735-742.
146. Ansell JS, Boughton R, Cullen T, et al. Lack of agreement between subjective ratings of instructors and objective testing of knowledge acquisition in a urological continuing medical education course. J Urol. 1979;122:721-723.
147. Sloan D, Donnelly M, Drake D, et al. Faculty sensitivity in detecting medical students' clinical competence. Med Teach. 1993;17:335-342.
148. Maxim BR, Dielman TE. Dimensionality, internal consistency and interrater reliability of clinical performance ratings. Med Educ. 1987;21:130-137.
149. Wanzel KR, Ward M, Reznick RK. Teaching the surgical craft: from selection to certification. Curr Probl Surg. 2002;39:573-659.
150. Feldman LS, Hagarty SE, Ghitulescu G, et al. Relationship between objective assessment of technical skills and subjective in-training evaluations in surgical residents. J Am Coll Surg. 2004;198:105-110.
151. Alaraj A, Charbel FT, Birk D, et al. Role of cranial and spinal virtual and augmented reality simulation using immersive touchmodules in neurosurgical training. Neurosurgery. 2013;72:115-123.
152. Asimakopoulos G, Karagounis AP, Valencia O, et al. How safe is it to train residents to perform off-pump coronary artery bypass surgery? Ann Thorac Surg. 2006;81:568-572.
153. Chiu CC, Wei PL, Wang W, et al. Role of appendectomy in laparoscopic training. J Laparoendosc Adv Surg Tech A. 2006;16:113-118.
154. Birkmeyer JD, Finks JF, O'Reilly A, et al. Surgical skill and complication rates after bariatric surgery. N Engl J Med. 2013;369:1434-1442.
155. Dauphinee WD, Blackmore DE, Smee S, et al. Using the Judgments of Physician Examiners in Setting the Standards for a National Multi-center High Stakes OSCE. Adv Health Sci Educ Theory Pract. 1997;2:201-211.
156. Roberson DW, Kentala E, Forbes P. Development and validation of an objective instrument to measure surgical performance at tonsillectomy. Laryngoscope. 2005;115:2127-2137.