A Dataset and Benchmarks for Segmentation and Recognition of Gestures in Robotic Surgery

IEEE Transactions on Biomedical Engineering

Volumn 64, Issue 9, 2017, Pages 2025-2041

  Ahmidi, Narges ^a   Tao, Lingling ^b   Sefati, Shahin ^b   Gao, Yixin ^b   Lea, Colin ^b   Haro, Benjamin Bejar ^b   Zappella, Luca ^b   Khudanpur, Sanjeev ^b   Vidal, Rene ^b   Hager, Gregory D ^b  

^a Johns Hopkins University   (United States)

^b JOHNS HOPKINS UNIVERSITY   (United States)

Author keywords

Activity recognition; benchmark robotic dataset; kinematics and video; surgical motion

Indexed keywords

CLASSIFICATION (OF INFORMATION); DATA HANDLING; DYNAMICAL SYSTEMS; GESTURE RECOGNITION; HIDDEN MARKOV MODELS; IMAGE RECOGNITION; INFORMATION ANALYSIS; KINEMATICS; LINEAR CONTROL SYSTEMS; MARKOV PROCESSES; PATTERN RECOGNITION; RANDOM PROCESSES; ROBOTICS; SURGERY;

ACTIVITY RECOGNITION; AUTOMATED SEGMENTATION; CONDITIONAL RANDOM FIELD; EVALUATION METHODOLOGIES; LINEAR DYNAMICAL SYSTEMS; SPATIO TEMPORAL FEATURES; STATE-OF-THE-ART TECHNIQUES; SURGICAL MOTION;

ROBOTIC SURGERY;

BENCHMARKING; CLASSIFICATION; DATA ANALYSIS; GESTURE; HIDDEN MARKOV MODEL; HUMAN; JOINT; KINEMATICS; MOTION; PREDICTION; ROBOT ASSISTED SURGERY; SKILL; SURGEON; SURGERY; TIME SERIES ANALYSIS; VALIDATION PROCESS; VIDEORECORDING; AUTOMATED PATTERN RECOGNITION; CLINICAL COMPETENCE; FACTUAL DATABASE; PROCEDURES; ROBOTIC SURGICAL PROCEDURE; STANDARDS; STATISTICS AND NUMERICAL DATA; THREE DIMENSIONAL IMAGING; UNITED STATES;

BENCHMARKING; CLINICAL COMPETENCE; DATABASES, FACTUAL; GESTURES; HUMANS; IMAGING, THREE-DIMENSIONAL; PATTERN RECOGNITION, AUTOMATED; ROBOTIC SURGICAL PROCEDURES; UNITED STATES;

EID: 85021087012     PISSN: 00189294     EISSN: 15582531     Source Type: Journal    
DOI: 10.1109/TBME.2016.2647680     Document Type: Article

References (76)

1. B. Bass, "Fundamental changes in general surgery residency training," Amer. J. Surg., vol. 73, pp. 109-113, 2007
2. M. Moller et al., "Mentoring the modern surgeon," Amer. Coll. Surgeons, pp. 19-25, 2008
3. H. Polavarapu et al., "100 years of surgical education: The past, present, and future," Amer. Coll. Surgeons, vol. 98, no. 7, pp. 22-27, 2013
4. J. Martin et al., "Objective structured assessment of technical skill for surgical residents," Brit. J. Surg., vol. 84, pp. 273-278, 1997
5. A. Goh et al., "Global evaluative assessment of robotic skills: Validation of a clinical assessment tool to measure robotic surgical skills," J. Urol., vol. 187, pp. 247-252, 2012
6. R. Hatala et al., "Constructing a validity argument for the objective structured assessment of technical skills (OSATS): A systematic review of validity evidence," Adv. Health Sci. Educ., Theory Pract., vol. 20, pp. 1149-1175, 2015
7. L. Kohn et al., To Err is Human: Building a Safer Health System. Washington, DC, USA: Nat. Acad. Press, 2002
8. C. Pellegrini, "TheACGMEoutcomes project," Amer. Council Grad. Med. Educ. Surg., vol. 131, pp. 214-215, 2002
9. J. Shah and A. Darzi, "Surgical skills assessment: An ongoing debate," Brit. J. Urol. Int., vol. 88, pp. 655-660, 2001
10. S. Hamstra and A. Dubrowski, "Effective training and assessment of surgical skills, and the correlates of performance," Surg. Innov., vol. 12, pp. 71-77, 2005
11. G. Fried and L. Feldman, "Objective assessment of technical performance," World J. Surg., vol. 32, pp. 156-160, 2008
12. A. Darzi and S. Mackay, "Assessment of surgical competence," Qual. Health Care, vol. 10, pp. ii64-ii69, 2001
13. V. Datta et al., "The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in laboratory-based model," J. Amer. Coll. Surg., vol. 193, pp. 479-485, 2001
14. T. Judkins et al., "Objective evaluation of expert and novice performance during robotic surgical training tasks," Surg. Endoscopy, vol. 1, no. 4, 2008
15. C. Richards et al., "Skills evaluation in minimally invasive surgery using force/torque signatures," Surg. Endoscopy, vol. 14, pp. 791-798, 2000
16. Y. Yamauchi et al., "Surgical skill evaluation by force data for endoscopic sinus surgery training system," Med. Image Comput. Comput.-Assisted Intervention, pp. 44-51, 2002
17. J. Rosen et al., "Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills," IEEE Trans. Biomed. Eng., vol. 48, no. 5, pp. 579-591, May 2001
18. C. McKenzie et al., "Hierarchical decomposition of laparoscopic surgery: A human factors approach to investigating the operating room environment," J. Minimally Invasive Therapy Allied Technol., vol. 10, no. 3, pp. 121-127, 2001
19. H. C. Lin et al., "Automatic detection and segmentation of robot-assisted surgical motions," Med. Image Comput. Comput.-Assisted Intervention, vol. 8, pp. 802-810, 2005
20. H. C. Lin et al., "Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions," Comput. Aided Surg., vol. 11, pp. 220-230, 2006
21. C. E. Reiley et al., "Automatic recognition of surgical motions using statistical modeling for capturing variability," Med. Meets Virtual Reality, vol. 132, pp. 396-401, 2008
22. A. Dosis et al., "Laparoscopic task recognition using hidden Markov models," Stud. Health Technol. Inform., vol. 111, pp. 115-122, 2005
23. J. Rosen et al., "Task decomposition of laparoscopic surgery for objective evaluation of surgical residents' learning curve using hidden Markov model," Comput. Aided Surg., vol. 7, no. 1, pp. 49-61, 2002
24. J. Leong et al., "HMM assessment of quality of movement trajectory in laparoscopic surgery," Med. Image Comput. Comput.-Assisted. Intervention, vol. 9, pp. 752-759, 2006
25. C. E. Reiley and G. D. Hager, "Task versus subtask surgical skill evaluation of robotic minimally invasive surgery," Med. Image Comput. Comput.-Assisted Intervention, vol. 12, pp. 435-442, 2009
26. S. S. Vedula et al., "Analysis of the structure of surgical activity for a suturing and knot-tying task," PLoS One, vol. 11, 2016, Art. no. e0149174
27. N. Ahmidi et al., "String motif-based description of tool motion for detecting skill and gestures in robotic surgery," Med. Image Comput. Comput.-Assisted Intervention, vol. 16, pp. 26-33, 2013
28. N. Ahmidi et al., "Automated objective surgical skill assessment in the operating room from unstructured tool motion in septoplasty," Int. J. Comput.-Assisted Radiol. Surg., vol. 10, pp. 981-991, 2015
29. B. Varadarajan, "Learning and inference algorithms for dynamic system models of dextrous motion," Ph.D. dissertation, Johns Hopkins Univ., 2011
30. L. Tao et al., "Sparse hidden Markov models for surgical gesture classification and skill evaluation," Inf. Process. Comput.-Assisted Interventions, vol. 7330, pp. 167-177, 2012
31. L. Tao et al., "Surgical gesture segmentation and recognition,"Med. Image Comput. Comput.-Assisted Intervention, vol. 16, pp. 339-346, 2013
32. C. Lea et al., "Using vision to improve activity recognition in surgical training tasks," in Proc. Role Human Sensorimotor Control Surg. Robot. Workshop, 2014, vol. 61, pp. 55-79
33. C. Lea et al., "Improved modeling of fine grained activities with application to surgical training tasks," Appl. Comput. Vis., vol. 61, pp. 55-79, 2014
34. F. Miyawaki et al., "Scrub nurse robot system-intraoperative motion analysis of a scrub nurse and timed-automata-based model for surgery," IEEE Trans. Ind. Electron., vol. 52, no. 5, pp. 1227-1235, Oct. 2005
35. U. Klank et al., "Automatic feature generation in endoscopic images," Int. J. Comput-Assisted Radiol. Surg., vol. 3, pp. 331-339, 2008
36. N. Padoy et al., "A boosted segmentation method for surgical workflow analysis," Med. Image Comput. Comput.-Assisted Intervention, vol. 10, pp. 102-109, 2007
37. T. Blum et al., "Workflow mining for visualization and analysis of surgeries," Int. J. Comput.-Assisted Radiol. Surg., vol. 3, no. 5, pp. 379-386, 2008
38. N. Padoy et al., "Statistical modeling and recognition of surgical workflow," Med. Image Anal., vol. 16, no. 3, pp. 632-641, 2012
39. F. Lalys et al., "Automatic knowledge-based recognition of low-level tasks in ophthalmological procedures," Int. J. Comput.-Assisted Radiol. Surg., vol. 8, no. 1, pp. 39-49, 2013
40. H. C. Lin, "Structure in surgical motion," Ph.D. dissertation, , Johns Hopkins Univ., 2010
41. F. Lalys et al., "An application-dependent framework for the recognition of high-level surgical tasks in the OR," Med. Image Comput. Comput.-Assisted Intervention, vol. 14, pp. 331-338, 2011
42. B. Béjar et al., "Surgical gesture classification from video data," Med. Image Comput. Comput.-Assisted Intervention, vol. 15, pp. 34-41, 2012
43. L. Zappella et al., "Surgical gesture classification from video and kinematic data," Med. Image Anal., vol. 17, pp. 732-745, 2013
44. Y. Gao et al., "Language of surgery: A surgical gesture dataset for human motion modeling," 2014. [Online]. Available: http://cirl.lcsr.jhu.edu/wpcontent/ uploads/2015/11/JIGSAWS.pdf
45. A. Y. Ng and M. I. Jordan, "On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes," Adv. Neural Inf. Process. Syst., vol. 14, 2002, Art. no. 841
46. B. Varadarajan et al., "Data-derived models for segmentation with application to surgical assessment and training," Med. Image Comput. Comput.-Assisted Intervention, vol. 12, pp. 426-434, 2009
47. P. N. Belhumeur et al., "Eigenfaces vs. fisherfaces: Recognition using class specific linear projection," IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 7, pp. 711-720, Jul. 1997
48. L. R. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proc. IEEE, vol. 77, no. 2, pp. 257-286, Feb. 1989
49. G. Forney-Jr, "The Viterbi algorithm," Proc. IEEE, vol. 61, no. 3, 1973
50. Y. Gao et al., "Query-by-example surgical activity detection," Int. J. Comput. Assisted Radiol. Surg., vol. 11, pp. 987-986, 2016
51. M. Aharon et al., "K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation," IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311-4322, Nov. 2006
52. I. Tsochantaridis et al., "Large margin methods for structured and interdependent output variables," J. Mach. Learn. Res., vol. 6, pp. 1453-1484, 2005
53. T. Joachims et al., "Cutting-plane training of structural SVMs," Mach. Learn., vol. 77, no. 1, pp. 27-59, 2009
54. Q. Shi et al., "Human action segmentation and recognition using discriminative semi-Markovmodels," Int. J. Comput. Vis., vol. 93, no. 1, pp. 22-32, 2011
55. H. Wang et al., "Action recognition by dense trajectories," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 3169-3176
56. C. Sutton and A. Mccallum, Introduction to Conditional Random Fields for Relational Learning. Cambridge, MA, USA: MIT Press, 2006
57. P. Felzenszwalb and D. Huttenlocher, "Pictorial structures for object recognition," Int. J. Comput. Vis., vol. 61, no. 1, pp. 55-79, 2005
58. S. Lacoste-Julien and M. Jaggi, "Block-coordinate Frank-Wolfe optimization for structural SVMs," in Proc. Int. Conf. Mach. Learn., 2013, pp. 53-61
59. C. Lea et al., "Learning convolutional action primitives for finegrained action recognition," in Proc. Int. Conf. Robot. Autom., 2016, pp. 1642-1649
60. C. Rupprecht et al., "Sensor substitution for video-based action recognition," in Proc. Int. Conf. Intell. Robots Syst., 2016, pp. 1-8
61. G. Csurka et al., "Visual categorization with bags of keypoints," in Proc. Eur. Conf. Comput. Vis., 2004, pp. 1-22
62. J. Sivic and A. Zisserman, "Video google: A text retrieval approach to object matching in videos," in Proc. IEEE Int. Conf. Comput. Vis., 2003, pp. 1470-1477
63. I. Laptev, "On space-time interest points," Int. J. Comput. Vis., vol. 64, pp. 107-123, 2005
64. G. Willems et al., "An efficient dense and scale-invariant spatiotemporal interest point detector," in Proc. Eur. Conf. Comput. Vis., 2008, pp. 650-663
65. R. Chaudhry et al., "Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 1932-1939
66. H. Wang et al., "Evaluation of local spatio-temporal features for action recognition," in Proc. Brit. Mach. Vis. Conf., 2009, pp. 1-11
67. J. Zhang et al., "Local features and kernels for classification of texture and object categories: A comprehensive study," Int. J. Comput. Vis., vol. 73, no. 2, pp. 213-238, 2007
68. J. Y. K. Yu et al., "Linear spatial pyramidmatching using sparse coding for image classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 1794-1801
69. J. Mairal et al., "Online dictionary learning for sparse coding," in Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp. 689-696
70. A. Martin, "A metric for ARMA processes," IEEE Trans. Signal Process., vol. 48, no. 4, pp. 1164-1170, Apr. 2000
71. K. Cock and B. Moor, "Subspace angles and distances between ARMA models," Syst. Control Lett., vol. 46, pp. 265-270, 2002
72. S. Vishwanathan et al., "Binet-Cauchy kernels on dynamical systems and its application to the analysis of dynamic scenes," Int. J. Comput. Vis., vol. 73, pp. 95-119, 2007
73. B. Afsari et al., "Group action induced distances for averaging and clustering linear dynamical systems with applications to the analysis of dynamic visual scenes," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 2208-2215
74. G. Doretto et al., "Dynamic textures," Int. J. Comput. Vis., vol. 51, pp. 91-109, 2003
75. B. Scholkopf and A. Smola, Learning With kerznels. Cambridge, MA, USA: MIT Press, 2002
76. C. Lea et al., "Segmental spatiotemporal CNNs for fine-grained action segmentation," in Proc. Comput. Vis. Pattern Recognit., 2016, pp. 36-52.