EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos

IEEE Transactions on Medical Imaging

Volumn 36, Issue 1, 2017, Pages 86-97

  Twinanda, Andru P ^a   Shehata, Sherif ^a   Mutter, Didier ^b   Marescaux, Jacques ^b   De Mathelin, Michel ^a   Padoy, Nicolas ^a  

^a UNIVERSITÉ DE STRASBOURG   (France)

^b IRCAD   (France)

Author keywords

cholecystectomy; convolutional neural network; Laparoscopic videos; phase recognition; tool presence detection

Indexed keywords

AUTOMATIC INDEXING; CONVOLUTION; LAPAROSCOPY; MEDICAL APPLICATIONS; NETWORK ARCHITECTURE; NEURAL NETWORKS; SURGERY;

CHOLECYSTECTOMY; CONVOLUTIONAL NEURAL NETWORK; LAPAROSCOPIC VIDEOS; PHASE RECOGNITION; PRESENCE DETECTIONS;

BIOMEDICAL SIGNAL PROCESSING;

CHOLECYSTECTOMY; EXPERIMENTAL MODEL; INTERMETHOD COMPARISON; NERVOUS SYSTEM; TOOL USE; VIDEORECORDING; VISUAL INFORMATION; ALGORITHM; ARTIFICIAL NEURAL NETWORK; FACTUAL DATABASE; LAPAROSCOPY;

ALGORITHMS; DATABASES, FACTUAL; LAPAROSCOPY; NEURAL NETWORKS (COMPUTER);

EID: 85013103816     PISSN: 02780062     EISSN: 1558254X     Source Type: Journal    
DOI: 10.1109/TMI.2016.2593957     Document Type: Article

References (35)

1. K. Cleary, H. Y. Chung, and S. K. Mun, "Or2020 workshop overview: Operating room of the future," Int. Congr. Ser., vol. 1268, pp. 847-852, Jun. 2004.
2. L. Bouarfa, P. P. Jonker, and J. Dankelman, "Discovery of high-level tasks in the operating room," J. Biomed. Informat., vol. 44, no. 3, pp. 455-462, 2011.
3. N. Padoy et al., "Statistical modeling and recognition of surgical workflow," Med. Image Anal., vol. 16, no. 3, pp. 632-641, 2012.
4. D. Katíc et al., "Knowledge-driven formalization of laparoscopic surgeries for rule-based intraoperative context-Aware assistance," in Information Processing in Computer-Assisted Interventions (Lecture Notes in Computer Science), vol. 8498. 2014, pp. 158-167.
5. G. Forestier et al., "Multi-site study of surgical practice in neurosurgery based on surgical process models," J. Biomed. Informat., vol. 46, no. 5, pp. 822-829, 2013.
6. F. Lalys, D. Bouget, L. Riffaud, and P. Jannin, "Automatic knowledgebased recognition of low-level tasks in ophthalmological procedures," Int. J. Comput. Assist. Radiol. Surg., vol. 8, no. 1, pp. 39-49, 2012.
7. T. Blum, H. Feuner, and N. Navab, "Modeling and segmentation of surgical workflow from laparoscopic video," in Medical Image Computing and Computer-Assisted Intervention (Lecture Notes in Computer Science), vol. 6363. 2010, pp. 400-407.
8. L. Zappella, B. Béjar, G. Hager, and R. Vidal, "Surgical gesture classification from video and kinematic data," Med. Image Anal., vol. 17, no. 7, pp. 732-745, 2013.
9. F. Lalys, L. Riffaud, D. Bouget, and P. Jannin, "A framework for the recognition of high-level surgical tasks from video images for cataract surgeries," IEEE Trans. Biomed. Eng., vol. 59, no. 4, pp. 966-976, Apr. 2012.
10. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1097-1105.
11. R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," in Proc. CVPR, 2014, pp. 580-587.
12. R. Stauder et al., "Random forests for phase detection in surgical workflow analysis," in Information Processing in Computer-Assisted Interventions (Lecture Notes in Computer Science), vol. 8498. 2014, pp. 148-157.
13. R. Sznitman, C. Becker, and P. Fua, "Fast part-based classification for instrument detection in minimally invasive surgery," in Medical Image Computing and Computer-Assisted Intervention (Lecture Notes in Computer Science), 2014, pp. 692-699.
14. D. Bouget et al., "Detecting surgical tools by modelling local appearance and global shape," IEEE Trans. Med. Imag., vol. 34, no. 12, pp. 2603-2617, Dec. 2015.
15. M. Allan et al., "Image based surgical instrument pose estimation with multi-class labelling and optical flow," in Medical Image Computing and Computer-Assisted Intervention (Lecture Notes in Computer Science), vol. 9349. 2015, pp. 331-338.
16. N. Rieke et al., "Surgical tool tracking and pose estimation in retinal microsurgery," in Medical Image Computing and Computer-Assisted Intervention (Lecture Notes in Computer Science), vol. 9349. 2015, pp. 266-273.
17. A. Reiter, P. K. Allen, and T. Zhao, "Feature classification for tracking articulated surgical tools," in Medical Image Computing and Computer-Assisted Intervention (Lecture Notes in Computer Science), vol. 7511. 2012, pp. 592-600.
18. M. Kranzfelder et al., "Real-time instrument detection in minimally invasive surgery using radiofrequency identification technology," J. Surg. Res., vol. 185, no. 2, pp. 704-710, 2013.
19. T. Neumuth and C. Meiner, "Online recognition of surgical instruments by information fusion," Int. J. Comput. Assist. Radiol. Surg., vol. 7, no. 2, pp. 297-304, 2012.
20. S. Speidel et al., "Automatic classification of minimally invasive instruments based on endoscopic image sequences," Proc. SPIE, vol. 7261, p. 72610A, Mar. 2009.
21. G. Quellec, M. Lamard, B. Cochener, and G. Cazuguel, "Real-time task recognition in cataract surgery videos using adaptive spatiotemporal polynomials," IEEE Trans. Med. Imag., vol. 34, no. 4, pp. 877-887, Apr. 2015.
22. B. P. L. Lo, A. Darzi, and G.-Z. Yang, "Episode classification for the analysis of tissue/instrument interaction with multiple visual cues," in Medical Image Computing and Computer-Assisted Intervention (Lecture Notes in Computer Science), vol. 2878. 2003, pp. 230-237.
23. G. Forestier, L. Riffaud, and P. Jannin, "Automatic phase prediction from low-level surgical activities," Int. J. Comput. Assist. Radiol. Surg., vol. 10, no. 6, pp. 833-841, 2015.
24. C. Lea, J. Facker, G. Hager, R. Taylor, and S. Saria, "3D sensing algorithms towards building an intelligent intensive care unit," in Proc. AMIA Summits Transl. Sci., 2013, p. 136.
25. N. Padoy, T. Blum, H. Feussner, M.-O. Berger, and N. Navab, "On-line recognition of surgical activity for monitoring in the operating room," in Proc. 20th IAAI, 2008, pp. 1718-1724.
26. C. Lea, G. D. Hager, and R. Vidal, "An improved model for segmentation and recognition of fine-grained activities with application to surgical training tasks," in Proc. WACV, Jan. 2015, pp. 1123-1129.
27. U. Klank, N. Padoy, H. Feussner, and N. Navab, "Automatic feature generation in endoscopic images," Int. J. Comput. Assist. Radiol. Surg., vol. 3, no. 3, pp. 331-339, 2008.
28. O. Russakovsky et al., "ImageNet large scale visual recognition challenge," Int. J. Comput. Vis., vol. 115, no. 3, pp. 211-252, Dec. 2015.
29. J. Sanchez and F. Perronnin, "High-dimensional signature compression for large-scale image classification," in Proc. CVPR, Washington, DC, USA, 2011, pp. 1665-1672.
30. J. Donahue et al., "Decaf: A deep convolutional activation feature for generic visual recognition," in Proc. ICML, 2013, pp. 647-655.
31. N. Padoy, D. Mateus, D. Weinland, M.-O. Berger, and N. Navab, "Workflow monitoring based on 3D motion features," in Proc. ICCV Workshops, 2009, pp. 585-592.
32. A. J. Viterbi, "Error bounds for convolutional codes and an asymptotically optimum decoding algorithm," IEEE Trans. Inf. Theory, vol. 13, no. 2, pp. 260-269, Apr. 1967.
33. P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, "Object detection with discriminatively trained part-based models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 9, pp. 1627-1645, Sep. 2010.
34. Y. Jia et al. (2014). "Caffe: Convolutional architecture for fast feature embedding." [Online]. Available: http://arxiv.org/abs/1408.5093
35. A. P. Twinanda, J. Marescaux, M. De Mathelin, and N. Padoy, "Towards better laparoscopic video database organization by automatic surgery classification," in Proc. IPCAI, 2014, pp. 186-195.