An integrated approach to endoscopic instrument tracking for augmented reality applications in surgical simulation training

International Journal of Medical Robotics and Computer Assisted Surgery

Volumn 9, Issue 4, 2013, Pages

  Loukas, Constantinos ^a   Lahanas, Vasileios ^a   Georgiou, Evangelos ^a  

^a UNIVERSITY OF ATHENS MEDICAL SCHOOL   (Greece)

Author keywords

Augmented reality; Instrument tracking; Laparoscopic training; Occlusion handling; Surgical simulation

Indexed keywords

CAMERAS; LAPAROSCOPY; SURGICAL EQUIPMENT; THREE DIMENSIONAL COMPUTER GRAPHICS; TRANSPLANTATION (SURGICAL); VIRTUAL REALITY;

AUGMENTED REALITY APPLICATIONS; AVERAGE ERRORS; ENDOSCOPIC CAMERA; ENDOSCOPIC INSTRUMENTS; INSTRUMENT TRACKING; INTEGRATED APPROACH; LAPAROSCOPIC TRAINING; OCCLUSION HANDLING; SIMULATION TRAINING; SURGICAL SIMULATION;

AUGMENTED REALITY;

ALGORITHM; ARTICLE; CALIBRATION; CAMERA; DEPTH PERCEPTION; ENDOSCOPE; ENDOSCOPY; EYE HAND COORDINATION; HUMAN; IMAGING SYSTEM; LIGHT; OCCLUSION; SIMULATION; SURGICAL EQUIPMENT; SURGICAL TRAINING; VISUAL INFORMATION;

AUGMENTED REALITY; INSTRUMENT TRACKING; LAPAROSCOPIC TRAINING; OCCLUSION HANDLING; SURGICAL SIMULATION;

ALGORITHMS; COMPUTER-ASSISTED INSTRUCTION; ENDOSCOPES; ENDOSCOPY; EQUIPMENT DESIGN; EQUIPMENT FAILURE ANALYSIS; HUMANS; PHANTOMS, IMAGING; REPRODUCIBILITY OF RESULTS; ROBOTICS; SENSITIVITY AND SPECIFICITY; SURGERY, COMPUTER-ASSISTED; SYSTEMS INTEGRATION; USER-COMPUTER INTERFACE;

EID: 84890425907     PISSN: 14785951     EISSN: 1478596X     Source Type: Journal    
DOI: 10.1002/rcs.1485     Document Type: Article

References (50)

1. Bashankaev B, Baido S, Wexner SD. Review of available methods of simulation training to facilitate surgical education. Surg Endosc 2011; 25(1): 28-35.
2. Semm K. Endoscopic appendectomy. Endoscopy 1983; 15(2): 59-64.
3. Liu A, Tendick F, Cleary K, et al. A survey of surgical simulation: applications, technology, and education. Presence 2003; 12(6): 599-614.
4. Fried GM, Feldman LS, Vassiliou MC, et al. Proving the value of simulation in laparoscopic surgery. Ann Surg 2004; 240(3): 518-525; discussion, 25-28.
5. Alaraj A, Lemole MG, Finkle JH, et al. Virtual reality training in neurosurgery: review of current status and future applications. Surg Neurol Int 2011; 2: 52.
6. Vidal FP, Bello F, Brodlie KW, et al. Principles and applications of computer graphics in medicine. Computer Graphics Forum 2006; 25(1): 113-137.
7. Satava RM. Surgical education and surgical simulation. World J Surg 2001; 25(11): 1484-1489.
8. Palter VN, Grantcharov TP. Simulation in surgical education. Can Med Assoc J 2010; 182(11): 1191-1196.
9. Shuhaiber JH. Augmented reality in surgery. Arch Surg 2004; 139(2): 170-174.
10. Maithel S, Sierra R, Korndorffer J, et al. Construct and face validity of MIST-VR, Endotower, and CELTS: are we ready for skills assessment using simulators? Surg Endosc 2006; 20(1): 104-112.
11. Woodrum DT, Andreatta PB, Yellamanchilli RK, et al. Construct validity of the LapSim laparoscopic surgical simulator. Am J Surg 2006; 191(1): 28-32.
12. Zhou M, Tse S, Derevianko A, et al. Effect of haptic feedback in laparoscopic surgery skill acquisition. Surg Endosc 2012; 26(4): 1128-1134.
13. Panait L, Akkary E, Bell RL, et al. The role of haptic feedback in laparoscopic simulation training. J Surg Res 2009; 156(2): 312-316.
14. Samset E, Schmalstieg D, Vander Sloten J, et al. Augmented reality in surgical procedures. In Proceedings of SPIE: Human Vision and Electronic Imaging XIII, 2008.
15. Mendez E, Kalkofen D, Schmalstieg D. Interactive context-driven visualization tools for augmented reality. In Third IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2004), Santa Barbara, CA, USA, 2004; 209-218.
16. Wang A, Mirsattari SM, Parrent AG, et al. Fusion and visualization of intraoperative cortical images with preoperative models for epilepsy surgical planning and guidance. Comput Aided Surg 2011; 16(4): 149-160.
17. Volonte F, Pugin F, Bucher P, et al. Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion. J Hepatobiliary Pancreat Sci 2011; 18(4): 506-509.
18. Nicolau S, Goffin L, Soler L. A low cost and accurate guidance system for laparoscopic surgery: validation on an abdominal phantom. In ACM Symposium on Virtual Reality Software and Technology (VRST 2005), Monterey, CA, USA, 2005; 124-133.
19. Soler L, Nicolau S, Schmid J, et al. Virtual reality and augmented reality in digestive surgery. In Third IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2004), Washington, DC, USA, 2004; 278-279.
20. Pellen MG, Horgan LF, Barton JR, et al. Construct validity of the ProMIS laparoscopic simulator. Surg Endosc 2009; 23(1): 130-139.
21. Botden SM, Jakimowicz JJ. What is going on in augmented reality simulation in laparoscopic surgery? Surg Endosc 2009; 23(8): 1693-1700.
22. Chmarra MK, Grimbergen CA, Dankelman J. Systems for tracking minimally invasive surgical instruments. Minim Invas Therapy Allied Technol 2007; 16(6): 328-340.
23. Martinez H, Mucha D, Kosmecki B, et al. Calibration method for electromagnetic tracked instruments in clinical applications. Int J Comput Assist Radiol Surg 2008; 3(1): 111-113.
24. Pagador JB, Sánchez LF, Sánchez JA, et al. Augmented reality haptic (ARH): an approach of electromagnetic tracking in minimally invasive surgery. Int J Comput Assist Radiol Surg 2011; 6(2): 257-263.
25. Loukas C, Georgiou E. Multivariate autoregressive modeling of hand kinematics for laparoscopic skills assessment of surgical trainees. IEEE Trans Biomed Eng 2011; 58(11): 3289-3297.
26. Dosis A, Aggarwal R, Bello F, et al. Synchronized video and motion analysis for the assessment of procedures in the operating theater. Arch Surg 2005; 140(3): 293-299.
27. Rosen J, Brown JD, Barreca M, et al. The Blue Dragon - a system for monitoring the kinematics and the dynamics of endoscopic tools in minimally invasive surgery for objective laparoscopic skill assessment. Stud Health Technol Inform 2002; 85: 412-418.
28. Wei GQ, Arbter K, Hirzinger G. Real-time visual servoing for laparoscopic surgery. Controlling robot motion with color image segmentation. IEEE Eng Med Biol Mag 1997; 16(1): 40-45.
29. Tonet O, Thoranaghatte RU, Megali G, et al. Tracking endoscopic instruments without a localizer: a shape-analysis-based approach. Comput Aided Surg 2007; 12(1): 35-42.
30. Doignon C, Nageotte F, Maurin B, et al. Pose Estimation and Feature Tracking for Robot Assisted Surgery with Medical Imaging. Unifying Perspectives in Computational and Robot Vision'. Series: Lecture Notes in Electrical Engineering, Vol. 8, Springer: New York, USA, 2008.
31. Ji Q, Costa MS, Haralick RM, Sharipo LG. An integrated technique for pose estimation from different geometric features. Int J Pattern Recogn, 1999;13(5): 705-733.
32. Shin S, Kim Y, Kwak H, et al. 3D tracking of surgical instruments using a single camera for laparoscopic surgery simulation. Stud Health Technol Inform 2011; 163: 581-587.
33. Cano AM, Gayá F, Lamata P, et al. Laparoscopic tool tracking method for augmented reality surgical applications. In Lecture Notes in Computer Science, Bello F, Edwards E (eds). Springer-Verlag: Berlin, Heidelberg, 2008; 191-196.
34. Cano AM, Lamata P, Gayá F, et al. New methods for video-based tracking of laparoscopic tools. In Lecture Notes in Computer Science, Harders M, Székely G (eds). Springer-Verlag: Berlin, Heidelberg, 2006; 142-149.
35. Allen BF, Kasper F, Nataneli G, et al. Visual tracking of laparoscopic instruments in standard training environments. Stud Health Technol Inform 2011; 163: 11-17.
36. Voros S, Long JA, Cinquin P. Automatic detection of instruments in laparoscopic images. Int J Robotics Res 2007; 26(11-12): 1173-1190.
37. Lahanas V, Loukas C, Nikiteas N, et al. Psychomotor skills assessment in laparoscopic surgery using augmented reality scenarios. In 17th International IEEE Conference on Digital Signal Processing (DSP 2011), Corfu, Greece, 2011; 1-6.
38. Bradski G, Kaehler A. Learning OpenCV: Computer Vision with the OpenCV Library. O'Reilly: Sebastopol, CA, USA, 2008.
39. Rasmussen C, Hager G. Tracking Objects by Color Alone. Yale University: New Haven, CT, USA, 1996.
40. Yang CC, Rodriguez JJ. Efficient luminance and saturation processing techniques for color images. J Vis Commun Image Represent 1997; 3: 263-277.
41. Doignon C, Graebling P, de Mathelin M. Real-time segmentation of surgical instruments inside the abdominal cavityusing a joint hue saturation color feature. Real-Time Imaging 2005; 11: 429-442.
42. Gonzalez RC, Woods RE. Digital Image Processing (2nd edn). Prentice Hall: Upper Saddle River, NJ, USA, 2002.
43. Kalman RE. A new approach to linear filtering and prediction problems. J Basic Eng 1960; 82(1): 35-45.
44. Mills S, Pridmore T, Hills M. Tracking in a Hough space with the extended Kalman filter. In British Machine Vision Conference, Norwich, UK, 2003; 173-182.
45. Szeliski R. Computer Vision: Algorithms and Applications. Springer: London, 2011.
46. Welsh G, Bishop G. An Introduction to the Kalman Filter. University of North Carolina: Chapel Hill, NC, USA, 1995.
47. Derossis AM, Fried GM, Abrahamowicz M, et al. Development of a model for training and evaluation of laparoscopic skills. Am J Surg 1998; 175(6): 482-487.
48. Kato H, Billinghurst M. Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR '99), San Francisco, CA, USA, 1999; 85-94.
49. Bell AK, Cao GL. Effects of artificial force feedback in laparoscopic surgery training simulators. In IEEE International Conference on Systems, Man and Cybernetics, 2007; 2239-2243.
50. Groeger M, Arbter K, Hirzinger G. Motion Tracking for Minimally Invasive Robotic Surgery. InTech Open-Access Publisher: Rijeka, Croatia, 2008.