Objective Assessment of Surgical Technical Skill and Competency in the Operating Room

Annual Review of Biomedical Engineering

Volumn 19, Issue , 2017, Pages 301-325

  Vedula, S Swaroop ^a   Ishii, Masaru ^b   Hager, Gregory D ^a  

^a Johns Hopkins University   (United States)

^b JOHNS HOPKINS UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

Objective computer aided surgical skill evaluation; Objective skill assessment; OCASE; Surgical data science; Surgical technical competence; Surgical technical skill

Indexed keywords

OPERATING ROOMS;

OCASE; SKILL ASSESSMENT; SKILL EVALUATION; SURGICAL DATA SCIENCE; TECHNICAL COMPETENCE; TECHNICAL SKILLS;

SURGERY;

ACCURACY; ARTICLE; AUTOMATION; CLINICAL COMPETENCE; COMPUTER ASSISTED SURGERY; CONSENSUS; HEALTH CARE ACCESS; HEALTH CARE DISPARITY; HEALTH CARE QUALITY; HEALTH CARE SYSTEM; HUMAN; OPERATING ROOM; PATIENT SAFETY; SIMULATION TRAINING; SURGEON; SURGICAL TECHNOLOGY; SURGICAL TRAINING; ALGORITHM; CLASSIFICATION; JOB PERFORMANCE; ORGANIZATION AND MANAGEMENT;

ALGORITHMS; CLINICAL COMPETENCE; OPERATING ROOMS; SURGEONS; WORK PERFORMANCE;

EID: 85021149685     PISSN: 15239829     EISSN: 15454274     Source Type: Book Series    
DOI: 10.1146/annurev-bioeng-071516-044435     Document Type: Article

References (135)

1. Weiser TG, Haynes AB, Molina G, Lipsitz SR, Esquivel MM, et al. 2016. Size and distribution of the global volume of surgery in 2012. Bull. World Health Organ. 94: F201-9
2. Weiser TG, Regenbogen SE, Thompson KD, Haynes AB, Lipsitz SR, et al. 2008. An estimation of the global volume of surgery: a modelling strategy based on available data. Lancet 372: 139-44
3. Birkmeyer JD, Finks JF, O'Reilly A, Oerline M, Carlin AM, et al. 2013. Surgical skill and complication rates after bariatric surgery. N. Engl. J. Med. 369: 1434-42
4. Nathan M, Karamichalis JM, Liu H, del Nido P, Pigula R, et al. 2011. Intraoperative adverse events can be compensated by technical performance in neonates and infants after cardiac surgery: a prospective study. J. Thorac. Cardiovasc. Surg. 142: 1098-107
5. Nathan M, Karamichalis JM, Liu H, Emani S, Baird C, et al. 2012. Surgical technical performance scores are predictors of late mortality and unplanned reinterventions in infants after cardiac surgery. J. Thorac. Cardiovasc. Surg. 144: 1095-101
6. Shuhaiber J, Gauvreau K, Thiagarajan R, Bacha E, Mayer J, et al. 2012. Congenital heart surgeons technical proficiency affects neonatal hospital survival. J. Thorac. Cardiovasc. Surg. 144: 1119-24
7. Parsa CJ, Organ CH Jr., BarkanH. 2000. Changing patterns of resident operative experience from 1990 to 1997. Arch. Surg. 135: 570-75
8. Smith R. 1998. All changed, changed utterly. BMJ 316: 1917-18
9. Szasz P, Louridas M, Harris KA, Aggarwal R, Grantcharov TP. 2015. Assessing technical competence in surgical trainees: a systematic review. Ann. Surg. 261: 1046-55
10. Bhatti NI, Cummings CW. 2007. Viewpoint: competency in surgical residency training: defining and raising the bar. Acad. Med. 82: 569-73
11. Satava RM, Gallagher AG, Pellegrini CA. 2003. Surgical competence and surgical proficiency: definitions, taxonomy, and metrics. J. Am. Coll. Surg. 196: 933-37
12. Sharma B, Mishra A, Aggarwal R, Grantcharov TP. 2011. Non-technical skills assessment in surgery. Surg. Oncol. 20: 169-77
13. Yule S, Flin R, Paterson-Brown S, Maran N. 2006. Non-technical skills for surgeons in the operating room: a review of the literature. Surgery 139: 140-49
14. Reznick RK. 1993. Teaching and testing technical skills. Am. J. Surg. 165: 358-61
15. Pradarelli JC, Campbell DA Jr., Dimick JB. 2015. Hospital credentialing and privileging of surgeons: a potential safety blind spot. JAMA 313: 1313-14
16. Roman H, Marpeau L, Hulsey TC. 2008. Surgeons experience and interaction effect in randomized controlled trials regarding new surgical procedures. Am. J. Obstet. Gynecol. 199: 108
17. Regenbogen SE, Greenberg CC, Studdert DM, Lipsitz SR, ZinnerMJ, Gawande AA. 2007. Patterns of technical error among surgical malpractice claims: an analysis of strategies to prevent injury to surgical patients. Ann. Surg. 246: 705-11
18. Hull L, Arora S, Aggarwal R, Darzi A, Vincent C, Sevdalis N. 2012. The impact of nontechnical skills on technical performance in surgery: a systematic review. J. Am. Coll. Surg. 214: 214-30
19. Mishra A, Catchpole K, Dale T, McCulloch P. 2007. The influence of non-technical performance on technical outcome in laparoscopic cholecystectomy. Surg. Endosc. 22: 68-73
20. Martin JA, Regehr G, Reznick R, Macrae H, Murnaghan J, et al. 1997. Objective structured assessment of technical skill (OSATS) for surgical residents. Br. J. Surg. 84: 273-78
21. Ghaderi I, Manji F, Park YS, Juul D, Ott M, et al. 2015. Technical skills assessment toolbox: a review using the unitary framework of validity. Ann. Surg. 261: 251-62
22. Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB. 2011. Observational tools for assessment of procedural skills: a systematic review. Am. J. Surg. 202: 469-80. e6
23. Jelovsek JE, KowN, DiwadkarGB. 2013. Tools for the direct observation and assessment of psychomotor skills in medical trainees: a systematic review. Med. Educ. 47: 650-73
24. Middleton RM, Baldwin MJ, Akhtar K, Alvand A, Rees JL. 2016. Which global rating scale J. Bone Joint Surg. Am. 98: 75-81
25. Anderson DD, Long S, ThomasGW, PutnamMD, Bechtold JE, KaramMD. 2015. Objective Structured Assessments of Technical Skills (OSATS) does not assess the quality of the surgical result effectively. Clin. Orthop. Relat. Res. 474: 874-81
26. Faulkner H, Regehr G, Martin J, Reznick R. 1996. Validation of an objective structured assessment of technical skill for surgical residents. Acad. Med. 71: 1363-65
27. Rooney DM, Hungness ES, DaRosa DA, Pugh CM. 2012. Can skills coaches be used to assess resident performance in the skills laboratory Surgery 151: 796-802
28. Lendvay TS, White L, Kowalewski T. 2015. Crowdsourcing to assess surgical skill. JAMA Surg. 150: 1086-87
29. GhaniKR, MillerDC, Linsell S, Brachulis A, Lane B, et al. 2016. Measuring to improve: peer and crowdsourced assessments of technical skill with robot-assisted radical prostatectomy. Eur. Urol. 69: 547-50
30. Powers MK, Boonjindasup A, Pinsky M, Dorsey P, Maddox M, et al. 2015. Crowdsourcing assessment of surgeon dissection of renal artery and vein during robotic partial nephrectomy: a novel approach for quantitative assessment of surgical performance. J. Endourol. 30: 447-52
31. Rutegård M, Lagergren J, Rouvelas I, Lagergren P. 2009. Surgeon volume is a poor proxy for skill in esophageal cancer surgery. Ann. Surg. 249: 256-61
32. Sheikh F, Gray RJ, Ferrara J, Foster K, Chapital A. 2010. Disparity between actual case volume and the perceptions of case volume needed to train competent general surgeons. J. Surg. Ed. 67: 371-75
33. Snyder RA, Tarpley MJ, Tarpley JL, Davidson M, Brophy C, Dattilo JB. Teaching in the operating room: results of a national survey. J. Surg. Ed. 69: 643-49
34. Feldman LS, Hagarty SE, Ghitulescu G, Stanbridge D, FriedGM. 2004. Relationship between objective assessment of technical skills and subjective in-training evaluations in surgical residents. J. Am. Coll. Surg. 198: 105-10
35. Porter ME. 2010. What is value in health care N. Engl. J. Med. 363: 2477-81
36. Bell RH Jr. 2009. Why Johnny cannot operate. Surgery 146: 533-42
37. Chikwe J, de Souza AC, Pepper JR. 2004. No time to train the surgeons. BMJ 328: 418-19
38. Vedula SS, Ishii M, Hager GD. 2016. Perspectives on surgical data science. Presented at Worksh. Surg. Data Sci., June 20, Heidelberg, Ger.
39. Bishop CM. 2009. Pattern Recognition and Machine Learning. New York: Springer
40. Hastie T, Tibshirani R, Friedman J. 2001. The Elements of Statistical Learning. New York: Springer
41. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York: Springer
42. ReileyCE, LinHC, Yuh DD, Hager GD. 2010. Review of methods for objective surgical skill evaluation. Surg. Endosc. 25: 356-66
43. vanHovePD, Tuijthof GJM, VerdaasdonkEGG, StassenLPS, Dankelman J. 2010. Objective assessment of technical surgical skills. Br. J. Surg. 97: 972-87
44. Oropesa P, Sánchez-González P, Lamata P, Chmarra MK, Pagador JB, et al. 2011. Methods and tools for objective assessment of psychomotor skills in laparoscopic surgery. J. Surg. Res. 171: e81-95
45. Dosis A, Bello F, Rockall T, Munz Y, Moorthy K, et al. 2003. ROVIMAS: a software package for assessing surgical skills using the da Vinci telemanipulator system. In Proceedings of the 4th International IEEE EMBS Special Topic Conference on Information Technology Applications in Biomedicine, pp. 326-29. Piscataway, NJ: IEEE
46. AizuddinM, Oshima N, Midorikawa R, Takanishi A. 2006. Development of sensor system for effective evaluation of surgical skill. In Proceedings of the 1st IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, pp. 678-83. Piscataway, NJ: IEEE
47. Rosen J. 1999. Surgeon-tool force/torque signatures-evaluation of surgical skills in minimally invasive surgery. In Medicine Meets Virtual Reality. Convergence of Physical and Informational Technologies: Options for a New Era in Healthcare, ed. JD Westwood, HM Hoffman, R Robb, D Stredney, pp. 290-96. Lansdale, PA: IOS
48. Hattori M, Egi H, Tokunaga M, Suzuki T, OhdanH, KawaharaT. 2012. The integrated deviation in the HUESAD(Hiroshima University endoscopic surgical assessment device) represents the surgeon's visual-spatial ability. In Proceedings of the 2012 ICME International Conference on Complex Medical Engineering, pp. 316-20. Piscataway, NJ: IEEE
49. Wang C, Noh Y, Ishii H, Kikuta G, Ebihara K, et al. 2011. Development of a 3D simulation which can provide better understanding of trainees performance of the task using airway management training system WKA-1RII. In Proceedings of the 2011 IEEE International Conference on Robotics and Biomimetics, pp. 2635-40. Piscataway, NJ: IEEE
50. Ahmidi N, Tao L, Sefati S, Gao Y, Lea C, et al. 2017. A dataset and benchmark for segmentation and recognition of gestures in robotic surgery. IEEE Trans. Biomed. Eng. In press. https: //doi. 10. 1109/ TBME. 2016. 2647680
51. Lea C, Hager GD, Vidal R. 2015. An improved model for segmentation and recognition of fine-grained activities with application to surgical training tasks. IEEE Winter Conf. Appl. Comput. Vis. 2015: 1123-29
52. Gao Y, Vedula SS, Lee GI, LeeMR, Khudanpur S, Hager GD. 2016. Int. J. Comput. Assist. Radiol. Surg. 11: 987-96
53. Lalys F, Jannin P. 2014. Surgical process modelling: a review. Int. J. CARS 9: 495-511
54. Ahmidi N, Poddar P, Jones JD, Vedula SS, Ishii L, et al. 2015. Automated objective surgical skill assessment in the operating room from unstructured tool motion in septoplasty. Int. J. CARS 10: 981-91
55. Richstone L, Schwartz MJ, Seideman C, Cadeddu J, Marshall S, Kavoussi LR. 2010. Eye metrics as an objective assessment of surgical skill. Ann. Surg. 252: 177-82
56. Ahmidi N, Hager GD, Ishii L, Fichtinger G, Gallia GL, Ishii M. 2010. Surgical task and skill classification from eye tracking and tool motion in minimally invasive surgery. Med. Image Comput. Comput. Assist. Interv. 13: 295-302
57. Ahmidi N, Hager GD, Ishii L, Gallia GL, Ishii M. 2012. Robotic path planning for surgeon skill evaluation in minimally-invasive sinus surgery. Med. Image Comput. Comput. Assist. Interv. 15: 471-78
58. Ahmidi N, Ishii M, Fichtinger G, Gallia GL, Hager GD. 2012. An objective and automated method for assessing surgical skill in endoscopic sinus surgery using eye-tracking and tool-motion data. Int. Forum Allergy Rhinol. 2: 507-15
59. Ahmidi N, Hager GD, Ishii M. 2012. Towards surgical skill assessment based on fractal patterns in minimally-invasive surgeries. Int. J. CARS 7: 185-200
60. Rosen J, Solazzo M, Hannaford B, SinananM. 2001. Objective laparoscopic skills assessments of surgical residents using hidden Markov models based on haptic information and tool/tissue interactions. Stud. Health Technol. Inform. 81: 417-23
61. Rosen J, Hannaford B, Richards CG, SinananMN. 2001. Markov modeling ofminimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. IEEE Trans. Biomed. Eng. 48: 579-91
62. Rosen J, Brown JD, Chang L, Sinanan MN, Hannaford B. 2006. Generalized approach for modeling minimally invasive surgery as a stochastic process using a discrete Markov model. IEEE Trans. Biomed. Eng. 53: 399-413
63. Chen J, Yeasin M, Sharma R. 2003. Visual modelling and evaluation of surgical skill. Pattern Anal. Appl. 6: 1-11
64. Sharma Y, Bettadapura V, Plötz T, Hammerld N, Mellor S, et al. 2014. Video based assessment of OSATS using sequential motion textures. In Proceedings of the 5th Workshop on Modeling and Monitoring of Computer Assisted Interventions. https: //smartech. gatech. edu/handle/1853/53651
65. Sharma Y, Plötz T, Hammerld N, Mellor S, McNaney R, et al. 2014. Automated surgical OSATS prediction from videos. In Proceedings of the 11th IEEE International Symposium on Biomedical Imaging, pp. 461-64. Piscataway, NJ: IEEE
66. Khan A, Mellor S, Berlin E, Thompson R, McNaney R, et al. 2015. Beyond activity recognition: skill assessment from accelerometer data. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 1155-66. New York: ACM
67. Zia A, Sharma Y, BettadapuraV, Sarin EL, ClementsMA, Essa I. 2015. Automated assessment of surgical skills using frequency analysis. Med. Image Comput. Comput. Assist. Interv. 11: 430-38
68. Speidel S, Zentek T, Sudra G, Gehrig T, Müller-Stich BP, et al. 2009. Recognition of surgical skills using hidden Markov models. Proc. SPIE 7261: 25
69. Zhang Q, Li B. 2011. Video-based motion expertise analysis in simulation-based surgical training using hierarchical Dirichlet process hidden Markov model. In Proceedings of the 2011 International ACM Workshop on Medical Multimedia Analysis and Retrieval, pp. 19-24. New York: ACM
70. Horeman T, Rodrigues SP, Jansen FW, Dankelman J, van den Dobbelsteen JJ. 2012. Force parameters for skills assessment in laparoscopy. IEEE Trans. Haptics 5: 312-22
71. Horeman T, Dankelman J, Jansen FW, van den Dobbelsteen JJ. 2014. Assessment of laparoscopic skills based on force and motion parameters. IEEE Trans. Biomed. Eng. 61: 805-13
72. Kowalewski TM. 2012. Real-time quantitative assessment of surgical skill. PhD thesis, Univ. Wash., Seattle
73. Lin Z, Uemura M, Zecca M, Sessa S, Ishii H, et al. 2013. Objective skill evaluation for laparoscopic training based on motion analysis. IEEE Trans. Biomed. Eng. 60: 977-85
74. Zhang Q, Li B. 2015. Relative hidden Markov models for video-based evaluation of motion skills in surgical training. IEEE Trans. Pattern Anal. Mach. Intell. 37: 1206-18
75. Islam G, Kahol K, Li B, Smith M, Patel VL. 2016. Affordable, web-based surgical skill training and evaluation tool. J. Biomed. Inform. 59: 102-14
76. Reiley CE, Hager GD. 2009. Task versus subtask surgical skill evaluation of robotic minimally invasive surgery. Med. Image Comput. Comput. Assist. Interv. 12: 435-42
77. Ahmidi N, Gao Y, Bjar B, Vedula SS, Khudanpur S, et al. 2013. String motif-based description of tool motion for detecting skill and gestures in robotic surgery. Med. Image Comput. Comput. Assist. Interv. 16: 26-33
78. Kumar R, Jog A, Malpani A, Vagvolgyi B, YuhD, et al. 2012. Assessing system operation skills in robotic surgery trainees. Int. J. Med. Robot. 8: 118-24
79. Kumar R, Jog A, Vagvolgyi B, Nguyen H, Hager G, et al. 2012. Objective measures for longitudinal assessment of robotic surgery training. J. Thorac. Cardiovasc. Surg. 143: 528-34
80. Tao L, Elhamifar E, Khudanpur S, Hager GD, Vidal R. 2012. Sparse hiddenMarkovmodels for surgical gesture classification and skill evaluation. In Lecture Notes in Computer Science, vol. 7330: Information Processing in Computer-Assisted Interventions, pp. 167-77. New York: Springer
81. Malpani A, Vedula SS, Chen CCG, Hager GD. 2014. Pairwise comparison-based objective score for automated skill assessment of segments in a surgical task. In Lecture Notes in Computer Science, vol. 8498: Information Processing in Computer-Assisted Interventions, pp. 138-47. New York: Springer
82. Malpani A, Vedula SS, Chen CCG, Hager GD. 2015. A study of crowdsourced segment-level surgical skill assessment using pairwise rankings. Int. J. CARS 10: 1435-47
83. Gomez ED, Aggarwal R, McMahan W, Bark K, Kuchenbecker KJ. 2015. Objective assessment of robotic surgical skill using instrument contact vibrations. Surg. Endosc. 30: 1419-31
84. Fard MJ. 2016. Computational modeling approaches for task analysis in robotic-assisted surgery. PhD thesis, pap. 1449, Wayne State Univ., Detroit, Mich.
85. Vedula SS, Malpani A, AhmidiN, Khudanpur S, HagerG, Chen CCG. 2016. Task-level versus segmentlevel quantitative metrics for surgical skill assessment. J. Surg. Ed. 73: 482-89
86. Gomez ED, Aggarwal R, McMahan W, Bark K, Kuchenbecker KJ. 2016. Objective assessment of robotic surgical skill using instrument contact vibrations. Surg. Endosc. 30: 1419-31
87. Rafii-Tari H, Payne CJ, Liu J, Riga C, Bicknell C, Yang GZ. 2015. Towards automated surgical skill evaluation of endovascular catheterization tasks based on force and motion signatures. In Proceedings of the IEEE International Conference on Robotics and Automation, pp. 1789-94. Piscataway, NJ: IEEE
88. Kramer BD, Losey DP, O'Malley MK. 2016. SOM and LVQ classification of endovascular surgeons using motion-based metrics. In Advances in Intelligent Systems and Computing, vol. 428: Advances in Self-Organizing Maps and Learning Vector Quantification, pp. 227-37. New York: Springer
89. Sewell C. 2007. Automatic performance evaluation in surgical simulation. PhD thesis, Stanford Univ., Stanford, CA
90. Rhienmora P, Haddawy P, Suebnukarn S, Dailey MN. 2009. Providing objective feedback on skill assessment in a dental surgical training simulator. In Lecture Notes in Computer Science, vol. 5651: Artificial Intelligence in Medicine, ed. C Combi, Y Shahar, A Abu-Hanna, pp. 305-14. New York: Springer
91. Rhienmora P, Haddawy P, Suebnukarn S, Dailey MN. 2011. Intelligent dental training simulator with objective skill assessment and feedback. Artif. Intell. Med. 52: 115-21
92. Liang H, Shi MY. 2010. Surgical skill evaluation model for virtual surgical training. Appl. Mech. Mater. 4041: 812-19
93. Hajshirmohammadi I. 2006. Using fuzzy set theory to objectively evaluate performance on minimally invasive surgical simulators. PhD thesis, Simon Fraser Univ., Burnaby, Can.
94. Leong JJH, NicolaouM, Atallah L, MylonasGP, DarziAW, Yang GZ. 2006. HMMassessment of quality of movement trajectory in laparoscopic surgery. Med. Image Comput. Comput. Assist. Interv. 9: 752-59
95. Megali G, Sinigaglia S, TonetO, Dario P. 2006. Modelling and evaluation of surgical performance using hidden Markov models. IEEE Trans. Biomed. Eng. 53: 1911-19
96. Loukas C, Georgiou E. 2011. Multivariate autoregressive modeling of hand kinematics for laparoscopic skills assessment of surgical trainees. IEEE Trans. Biomed. Eng. 58: 3289-97
97. Jog A, Itkowitz B, Liu M, DiMaio S, Hager G, et al. 2011. Towards integrating task information in skills assessment for dexterous tasks in surgery and simulation. In Proceedings of the 2011 IEEE International Conference on Robotics and Automation, pp. 5273-78. Piscataway, NJ: IEEE
98. Arora S, Sevdalis N, Nestel D, Woloshynowych M, Darzi A, Kneebone R. 2010. The impact of stress on surgical performance: a systematic review of the literature. Surgery 147: 318-30
99. Lee G, Lee T, Dexter D, Klein R, Park A. 2007. Methodological infrastructure in surgical ergonomics: a review of tasks, models, and measurement systems. Surg. Innov. 14: 153-67
100. Ericsson KA, Charness N, Feltovich PJ, Hoffman RR, ed. 2006. The Cambridge Handbook of Expertise and Expert Performance. New York: Cambridge Univ. Press
101. Crochet P, Aggarwal R, Dubb SS, Ziprin P, Rajaretnam N, et al. 2011. Deliberate practice on a virtual reality laparoscopic simulator enhances the quality of surgical technical skills. Ann. Surg. 253: 1216-22
102. Bonrath EM, Dedy NJ, Gordon LE, GrantcharovTP. 2015. Comprehensive surgical coaching enhances surgical skill in the operating room: a randomized controlled trial. Ann. Surg. 262: 205-12
103. Messick S. 1995. Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. Am. Psychol. 50: 741-49
104. Anderson F, Birch DW, Boulanger P, Bischof WF. 2012. Sensor fusion for laparoscopic surgery skill acquisition. Comput. Aided Surg. 17: 269-83
105. Snaineh STA, Seales B. 2015. Minimally invasive surgery skills assessment using multiple synchronized sensors. In Proceedings of the IEEE International Symposium on Signal Processing and Information Technology, pp. 314-19. Piscataway, NJ: IEEE
106. Sonnadara R, RittenhouseN, Khan A, Mihailidis A, Drozdzal G, et al. 2012. A novel multimodal platform for assessing surgical technical skills. Am. J. Surg. 203: 32-36
107. Ruda K, Beekman D, White LW, Lendvay TS, Kowalewski TM. 2013. SurgTrak-a universal platform for quantitative surgical data capture. J. Med. Devices 7: 030923
108. Ostler D, Kranzfelder M, Stauder R, Wilhelm D, Feussner H, Schneider A. 2015. A centralized data acquisition framework for operating theatres. In Proceedings of the 17th International Conference on E-Health Networking, Application Services, pp. 1-5. Piscataway, NJ: IEEE
109. Zhao X. 2005. Acquisition, storage and reconstruction of multidimensional surgical information in a digital operation room environment. PhD thesis, Va. Commonw. Univ., Richmond
110. Chmarra MK, Grimbergen CA, Dankelman J. 2007. Systems for tracking minimally invasive surgical instruments. Minim. Invasive Ther. Allied Technol. 16: 328-40
111. DiMaio S, Hasser C. 2008. The da Vinci research interface. MIDAS J. http: //hdl. handle. net/10380/ 1464
112. Trejos AL, Patel RV, Naish MD, Lyle AC, Schlachta CM. 2009. A sensorized instrument for skills assessment and training in minimally invasive surgery. J. Med. Devices 3: 041002
113. Rutherford DN, D'Angelo ALD, Law KE, Pugh CM. 2015. Advanced engineering technology for measuring performance. Surg. Clin. N. Am. 95: 813-26
114. Kirby GSJ, Guyver P, Strickland L, Alvand A, Yang GZ, et al. 2015. Assessing arthroscopic skills using wireless elbow-worn motion sensors. J. Bone Joint Surg. Am. 97: 1119-27
115. King RC, Atallah L, Lo BPL, Yang GZ. 2009. Development of a wireless sensor glove for surgical skills assessment. IEEE Trans. Inf. Technol. Biomed. 13: 673-79
116. Saggio G, Santosuosso GL, Cavallo P, Pinto CA, Petrella M, et al. 2011. Gesture recognition and classification for surgical skill assessment. In Proceedings of the IEEE International Workshop on Medical Measurements, pp. 662-66. Piscataway, NJ: IEEE
117. Kodama H, Tercero C, Ooe K, Shi C, Ikeda S, et al. 2012. 2-D optical encoding of catheter motion and cyber-physical system for technical skills measurement and quantitative evaluation in endovascular surgery. In Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3565-70. Piscataway, NJ: IEEE
118. Conditt MA, Noble PC, Thompson MT, Ismaily SK, Moy GJ, Mathis KB. 2007. A computerized bioskills system for surgical skills training in total knee replacement. Proc. Inst. Mech. Eng. H 221: 61-69
119. Lu J, Kowalewski TM. 2015. Flexible, stretchable skin sensors for two-dimensional position tracking in medical simulators. J. Med. Devices 9: 020927
120. Atkins MS, Tien G, Khan RSA, Meneghetti A, Zheng B. 2012. What do surgeons see: capturing and synchronizing eye gaze for surgery applications. Surg. Innov. 20: 241-48
121. Tien T, Pucher PH, Sodergren MH, Sriskandarajah K, Yang GZ, Darzi A. 2015. Differences in gaze behaviour of expert and junior surgeons performing open inguinal hernia repair. Surg. Endosc. 29: 405-13
122. Pandya A, Reisner LA, King B, Lucas N, Composto A, et al. 2014. A review of camera viewpoint automation in robotic and laparoscopic surgery. Robotics 3: 310-29
123. Morris MC, Frodl T, D'Souza A, Fagan AJ, Ridgway PF. 2015. Assessment of competence in surgical skills using functional magnetic resonance imaging: a feasibility study. J. Surg. Ed. 72: 198-204
124. Evans AW, Leeson RMA, Petrie A. 2005. Correlation between a patient-centred outcome score and surgical skill in oral surgery. Br. J. Oral Maxillofac. Surg. 43: 505-10
125. Ahmidi N. 2015. Activity detection and skill assessment for dexterous motions in robotic and minimally-invasive surgery. PhD thesis, Johns Hopkins Univ., Baltimore, Md.
126. Bergland GD. 1969. A guided tour of the fast Fourier transform. IEEE Spectrum 6: 41-52
127. Zhang D, Lu G. 2004. Review of shape representation and description techniques. Pattern Recognit. 37: 1-19
128. Laptev I, Marszalek M, Schmid C, Rozenfeld B. 2008. Learning realistic human actions from movies. In Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. Piscataway, NJ: IEEE
129. Ershad M, Koesters Z, Rege R, Majewicz A. 2016. Meaningful assessment of surgical expertise: semantic labeling with data and crowds. In Medical Image Computing and Computer-Assisted Intervention, ed. S Ourselin, L Joskowicz, MR Sabuncu, G Unal, WWells, pp. 508-15. Berlin: Springer
130. Sant'Anna A, Wickström N. 2011. Symbolization of time-series: an evaluation of SAX, Persist, and ACA. In Proceedings of the 4th International Congress on Image and Signal Processing, pp. 2223-28. Piscataway, NJ: IEEE
131. Shafiei SB, Guru KA, Esfahani ET. 2015. Using two-third power law for segmentation of hand movement in robotic assisted surgery. In Proceedings of the 2015 ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference: 39th Mechanisms and Robotics Conference, pap. V05CT08A014. Boston: ASME
132. Bettadapura V, Schindler G, Ploetz T, Essa I. 2013. Augmenting bag-of-words: data-driven discovery of temporal and structural information for activity recognition. In Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2619-26. Piscataway, NJ: IEEE
133. Jelinek F. 2001. Statistical Methods for Speech Recognition. Cambridge, MA: MIT Press
134. Porte MC, Xeroulis G, Reznick RK, Dubrowski A. 2007. Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills. Am. J. Surg. 193: 105-10
135. Bjerrum F, MaagaardM, Sorensen JL, Larsen CR, Ringsted C, et al. 2015. Effect of instructor feedback on skills retention after laparoscopic simulator training: follow-up of a randomized trial. J. Surg. Ed. 72: 53-60