A review of camera viewpoint automation in robotic and laparoscopic surgery

Robotics

Volumn 3, Issue 3, 2014, Pages 310-329

  Pandya, Abhilash ^a   Reisner, Luke A ^a   King, Brady ^b   Lucas, Nathan ^a   Composto, Anthony ^a   Klein, Michael ^b   Ellis, Richard Darin ^c  

^a Wayne State University   (United States)

^b CHILDREN'S HOSPITAL OF MICHIGAN   (United States)

^c WAYNE STATE UNIVERSITY   (United States)

Author keywords

Autonomous camera control; Laparoscopic surgery; Path planning; Robotic surgery; Teleoperation

Indexed keywords

EID: 84973275385     PISSN: None     EISSN: 22186581     Source Type: Journal    
DOI: 10.3390/robotics3030310     Document Type: Review

References (70)

1. Berguer, R.; Forkey, D.; Smith, W. The effect of laparoscopic instrument working angle on surgeons' upper extremity workload. Surg. Endosc. 2001, 15, 1027-1029.
2. Keehner, M.M.; Tendick, F.; Meng, M.V.; Anwar, H.P.; Hegarty, M.; Stoller, M.L.; Duh, Q.-Y. Spatial ability, experience, and skill in laparoscopic surgery. Am. J. Surg. 2004, 188, 71-75.
3. Zheng, B.; Cassera, M.A.; Martinec, D.V.; Spaun, G.O.; Swanström, L.L. Measuring mental workload during the performance of advanced laparoscopic tasks. Surg. Endosc. 2010, 24, 45-50.
4. Christensen, H.I.; Goldberg, K.; Kumar, V.; Messina, E. A Roadmap for U.S. Robotics: From Internet to Robotics; Robotics-Virtual Organization: Atlanta, GA, USA, 2013.
5. Kranzfelder, M.; Staub, C.; Fiolka, A.; Schneider, A.; Gillen, S.; Wilhelm, D.; Friess, H.; Knoll, A.; Feussner, H. Toward increased autonomy in the surgical OR: Needs, requests, and expectations. Surg. Endosc. 2013, 27, 1681-1688.
6. Aiono, S.; Gilbert, J.; Soin, B.; Finlay, P.; Gordan, A. Controlled trial of the introduction of a robotic camera assistant (endoassist) for laparoscopic cholecystectomy. Surg. Endosc. Interv. Tech. 2002, 16, 1267-1270.
7. Schäfer, M.; Krähenbühl, L.; Büchler, M. Comparison of adhesion formation in open and laparoscopic surgery. Dig. Surg. 1998, 15, 148-152.
8. Shah, R.D.; Cao, A.; Golenberg, L.; Ellis, R.D.; Auner, G.W.; Pandya, A.K.; Klein, M.D. Performance of basic manipulation and intracorporeal suturing tasks in a robotic surgical system: Single-versus dual-monitor views. Surg. Endosc. 2009, 23, 727-733.
9. Burelli, P.; di Gaspero, L.; Ermetici, A.; Ranon, R. Virtual Camera Composition with Particle Swarm Optimization. In Proceedings of the 8th International Symposium on Smart Graphics, Rennes, France, 27-29 August 2008; pp. 130-141.
10. Christie, M.; Olivier, P.; Normand, J.-M. Camera control in computer graphics. Comput. Graph. Forum 2008, 27, 2197-2218.
11. Drucker, S.M.; Zeltzer, D. Intelligent Camera Control in a Virtual Environment. In Proceedings of the Graphics Interface, Banff, AB, Canada, 18-20 May 1994; p. 190.
12. Gleicher, M.; Witkin, A. Through-the-Lens Camera Control. In Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques, Chicago, IL, USA, 26-31 July 1992; pp. 331-340.
13. Halper, N.; Olivier, P. Camplan: A Camera Planning Agent. In Proceedings of the AAAI Spring Symposium on Smart Graphics, Stanford, CA, USA, 20-22 March 2000; pp. 92-100.
14. Dragan, A.; Lee, K.C.T.; Srinivasa, S. Teleoperation with intelligent and customizable interfaces. J. Hum.-Robot Interact. 2013, 2, 33-79.
15. Duchowski, A.T. A breadth-first survey of eye-tracking applications. Behav. Res. Methods Instrum. Comput. 2002, 34, 455-470.
16. Morimoto, C.H.; Mimica, M.R. Eye gaze tracking techniques for interactive applications. Comput. Vis. Image Underst. 2005, 98, 4-24.
17. Hansen, D.W.; Ji, Q. In the eye of the beholder: A survey of models for eyes and gaze. IEEE Trans. Pattern Anal. Mach. Intell. 2010, 32, 478-500.
18. Salvucci, D.D.; Goldberg, J.H. Identifying Fixations and Saccades in Eye-Tracking Protocols. In Proceedings of the Symposium on Eye Tracking Research & Applications, Palm Beach Gardens, FL, USA, 6-8 November 2000; pp. 71-78.
19. Jacob, R.J.; Karn, K.S. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. In The Mind's Eyes: Cognitive and Applied Aspects of Eye Movements; Hyona, J., Radach, R., Deubel, H., Eds.; Elsevier Science: Amsterdam, The Netherlands, 2003; p. 573.
20. Goldberg, J.H.; Schryver, J.C. Eye-gaze-contingent control of the computer interface: Methodology and example for zoom detection. Behav. Res. Methods Instrum. Comput. 1995, 27, 338-350.
21. Hansen, D.W.; Skovsgaard, H.H.; Hansen, J.P.; Møllenbach, E. Noise Tolerant Selection by Gaze-Controlled Pan and Zoom in 3D. In Proceedings of the Symposium on Eye Tracking Research & Applications, Savannah, GA, USA, 26-28 March 2008; pp. 205-212.
22. Latif, H.O.; Sherkat, N.; Lotfi, A. Telegaze: Teleoperation through Eye Gaze. In Proceedings of the 7th IEEE International Conference on Cybernetic Intelligent Systems, London, UK, 9-10 September 2008; pp. 1-6.
23. Latif, H.O.; Sherkat, N.; Lotfi, A. Teleoperation through Eye Gaze (Telegaze): A Multimodal Approach. In Proceedings of the IEEE International Conference on Robotics and Biomimetics, Guilin, China, 19-23 December 2009; pp. 711-716.
24. Zhu, D.; Gedeon, T.; Taylor, K. "Moving to the centre": A gaze-driven remote camera control for teleoperation. Interact. Comput. 2011, 23, 85-95.
25. Kwok, K.-W.; Sun, L.-W.; Mylonas, G.P.; James, D.R.C.; Orihuela-Espina, F.; Yang, G.-Z. Collaborative gaze channelling for improved cooperation during robotic assisted surgery. Ann. Biomed. Eng. 2012, 40, 2156-2167.
26. Ali, S.M.; Reisner, L.A.; King, B.; Cao, A.; Auner, G.; Klein, M.; Pandya, A.K. Eye gaze tracking for endoscopic camera positioning: An application of a hardware/software interface developed to automate aesop. Stud. Health Technol. Inform. 2007, 132, 4-7.
27. Mylonas, G.P.; Darzi, A.; Zhong Yang, G. Gaze-contingent control for minimally invasive robotic surgery. Comput. Aided Surg. 2006, 11, 256-266.
28. Mudunuri, A.V. Autonomous Camera Control System for Surgical Robots. M.S. Thesis, Wayne State University, Detroit, MI, USA, 2011.
29. Denavit, J.; Hartenberg, R.S. A kinematic notation for lower-pair mechanisms based on matrices. ASME J. Appl. Mech. 1955, 22, 215-221.
30. Lee, C.; Wang, Y.-F.; Uecker, D.R.; Wang, Y. Image Analysis for Automated Tracking in Robot-Assisted Endoscopic Surgery. In Proceedings of the 12th IAPR International Conference on Pattern Recognition, Jerusalem, Israel, 9-13 October 1994; pp. 88-92.
31. Wei, G.-Q.; Arbter, K.; Hirzinger, G. Real-time visual servoing for laparoscopic surgery: Controlling robot motion with color image segmentation. IEEE Eng. Med. Biol. Mag. 1997, 16, 40-45.
32. Omote, K.; Feussner, H.; Ungeheuer, A.; Arbter, K.; Wei, G.-Q.; Siewert, J.R.; Hirzinger, G. Self-guided robotic camera control for laparoscopic surgery compared with human camera control. Am. J. Surg. 1999, 177, 321-324.
33. Fortney, D.R. Real-Time Color Image Guidance System; University of California at Santa Barbara: Santa Barbara, CA, USA, 2000.
34. Hastenteufel, M.; Vetter, M.; Meinzer, H.-P.; Wolf, I. Effect of 3D ultrasound probes on the accuracy of electromagnetic tracking systems. Ultrasound Med. Biol. 2006, 32, 1359-1368.
35. LaScalza, S.; Arico, J.; Hughes, R. Effect of metal and sampling rate on accuracy of flock of birds electromagnetic tracking system. J. Biomech. 2003, 36, 141-144.
36. King, D.L. Errors as a result of metal in the near environment when using an electromagnetic locator with freehand three-dimensional echocardiography. J. Am. Soc. Echocardiogr. 2002, 15, 731-735.
37. Golenberg, L.P. Task Analysis, Modeling, and Automatic Identification of Elemental Tasks in Robot-Assisted Laparoscopic Surgery. Ph.D. Thesis, Wayne State University, Detroit, MI, USA, 2010.
38. Speidel, S.; Delles, M.; Gutt, C.; Dillmann, R. Tracking of instruments in minimally invasive surgery for surgical skill analysis. In Medical Imaging and Augmented Reality; Springer: Berlin, Germany, 2006; pp. 148-155.
39. Murphy, T.E. Towards Objective Surgical Skill Evaluation with Hidden Markov Model-Based Motion Recognition. M.S. Thesis, Johns Hopkins University, Baltimore, MD, USA, 2005.
40. Lin, H.C.; Shafran, I.; Yuh, D.; Hager, G.D. Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions. Comput. Aided Surg. 2006, 11, 220-230.
41. Reiley, C.E.; Lin, H.C.; Varadarajan, B.; Vagvolgyi, B.; Khudanpur, S.; Yuh, D.; Hager, G. Automatic recognition of surgical motions using statistical modeling for capturing variability. Stud. Health Technol. Inform. 2008, 132, 396.
42. Weede, O.; Bihlmaier, A.; Hutzl, J.; Müller-Stich, B.P.; Wörn, H. Towards Cognitive Medical Robotics in Minimal Invasive Surgery. In Proceedings of the Conference on Advances in Robotics, Pune, India, 4-6 July 2013; pp. 1-8.
43. Reiley, C.E.; Lin, H.C.; Yuh, D.D.; Hager, G.D. Review of methods for objective surgical skill evaluation. Surg. Endosc. 2011, 25, 356-366.
44. Alexopoulos, C.; Griffin, P.M. Path planning for a mobile robot. IEEE Trans. Syst. Man Cybern. 1992, 22, 318-322.
45. Beckhaus, S.; Ritter, F.; Strothotte, T. Cubicalpath-Dynamic Potential Fields for Guided Exploration in Virtual Environments. In Proceedings of the Eighth Pacific Conference on Computer Graphics and Applications, Hong Kong, China, 3-5 October 2000; pp. 387-459.
46. Beckhaus, S.; Ritter, F.; Strothotte, T. Guided exploration with dynamic potential fields: The cubical path system. Comput. Graph. Forum 2001, 20, 201-211.
47. Burelli, P.; Jhala, A. Dynamic Artificial Potential Fields for Autonomous Camera Control. In Proceedings of the Fifth Artificial Intelligence and Interactive Digital Entertainment, Palo Alto, CA, USA, 14-16 October 2009.
48. Khatib, O. Real-time obstacle avoidance for manipulators and mobile robots. Int. J. Robot. Res. 1986, 5, 90-98.
49. Barraquand, J.; Langlois, B.; Latombe, J.-C. Numerical potential field techniques for robot path planning. IEEE Trans. Syst. Man Cybern. 1992, 22, 224-241.
50. Bares, W.H.; Grégoire, J.P.; Lester, J.C. Realtime Constraint-Based Cinematography for Complex Interactive 3D Worlds. In Proceedings of the Tenth Conference on Innovative Applications of Artificial Intelligence, Madison, WI, USA, 26-30 July 1998; pp. 1101-1106.
51. He, L.-W.; Cohen, M.F.; Salesin, D.H. The Virtual Cinematographer: A Paradigm for Automatic Real-Time Camera Control and Directing. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, LA, USA, 4-9 August 1996; pp. 217-224.
52. Burelli, P. Interactive Virtual Cinematography. Ph.D. Thesis, IT University Of Copenhagen, Copenhagen, Denmark, 2012.
53. Burelli, P.; Yannakakis, G.N. Global Search for Occlusion Minimisation in Virtual Camera Control. In Proceedings of the IEEE Congress on Evolutionary Computation, Barcelona, Spain, 18-23 July 2010; pp. 1-8.
54. Hornung, A.; Lakemeyer, G.; Trogemann, G. An autonomous real-time camera agent for interactive narratives and games. In Intelligent Virtual Agents; Rist, T., Aylett, R., Ballin, D., Rickel, J., Eds.; Springer: Berlin/Heidelberg, Germany: 2003; Volume 2792, pp. 236-243.
55. Jardillier, F.; Languénou, E. Screen-space constraints for camera movements: The virtual cameraman. Comput. Graph. Forum 1998, 17, 175-186.
56. Hong, L.; Muraki, S.; Kaufman, A.; Bartz, D.; He, T. Virtual Voyage: Interactive Navigation in the Human Colon. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA, 3-8 August 1997; pp. 27-34.
57. Ko, S.-Y.; Kim, J.; Lee, W.-J.; Kwon, D.-S. Compact laparoscopic assistant robot using a bending mechanism. Adv. Robot. 2007, 21, 689-709.
58. Ko, S.-Y.; Kwon, D.-S. A Surgical Knowledge Based Interaction Method for A Laparoscopic Assistant Robot. In Proceedings of the 13th IEEE International Workshop on Robot and Human Interactive Communication, Kurashiki, Okayama, Japan, 20-22 September 2004; pp. 313-318.
59. Kai-Tai, S.; Chun-Ju, C. Autonomous and Stable Tracking of Endoscope Instrument Tools with Monocular Camera. In Proceedings of the IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Kachsiung, Taiwan, 11-14 July 2012; pp. 39-44.
60. Azizian, M.; Khoshnam, M.; Najmaei, N.; Patel, R.V. Visual servoing in medical robotics: A survey. Part I: Endoscopic and direct vision imaging-Techniques and applications. Int. J. Med. Robot. Comput. Assist. Surg. 2013, doi: 10.1002/rcs.1531.
61. King, B.W.; Reisner, L.A.; Pandya, A.K.; Composto, A.M.; Ellis, R.D.; Klein, M.D. Towards an autonomous robot for camera control during laparoscopic surgery. J. Laparoendosc. Adv. Surg. Tech. 2013, 23, 1027-1030.
62. Weede, O.; Monnich, H.; Muller, B.; Worn, H. An Intelligent and Autonomous Endoscopic Guidance System for Minimally Invasive Surgery. In Proceedings of the IEEE International Conference on Robotics and Automation, Shanghai, China, 9-13 May 2011; pp. 5762-5768.
63. Yu, L.; Wang, Z.; Sun, L.; Wang, W.; Wang, T. A Kinematics Method of Automatic Visual Window for Laparoscopic Minimally Invasive Surgical Robotic System. In Proceedings of the IEEE International Conference Mechatronics and Automation, Takamatsu, Japan, 4-7 August 2013; pp. 997-1002.
64. Crandall, B.; Klein, G.A.; Hoffman, R.R. Working Minds: A Practitioner's Guide to Cognitive Task Analysis; MIT Press: Cambridge, MA, USA, 2006.
65. Van Den Berg, J.; Miller, S.; Duckworth, D.; Hu, H.; Wan, A.; Fu, X.-Y.; Goldberg, K.; Abbeel, P. Superhuman Performance of Surgical Tasks by Robots Using Iterative Learning from Human-Guided Demonstrations. In Proceedings of the IEEE International Conference on Robotics and Automation, Anchorage, AK, USA, 3-7 May 2010; pp. 2074-2081.
66. Steinfeld, A.; Fong, T.; Kaber, D.; Lewis, M.; Scholtz, J.; Schultz, A.; Goodrich, M. Common Metrics for Human-Robot Interaction. In Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction, Salt Lake City, UT, USA, 2-3 March 2006; pp. 33-40.
67. Food and Drug Administration. Draft Guidance for Industry and Food and Drug Administration Staff-Applying Human Factors and Usability Engineering to Optimize Medical Device Design. Available online: http://www.fda.gov/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm259748.htm (accessed on 17 July 2014).
68. Chen, Z.; Deguet, A.; Taylor, R.; DiMaio, S.; Fischer, G.; Kazanzides, P. An Open-Source Hardware and Software Platform for Telesurgical Robotics Research. In Proceedings of the MICCAI Workshop on Systems and Architecture for Computer Assisted Interventions, Nagoya, Japan, 22-26 September 2013.
69. Lum, M.J.; Friedman, D.C.; Sankaranarayanan, G.; King, H.; Fodero, K.; Leuschke, R.; Hannaford, B.; Rosen, J.; Sinanan, M.N. The RAVEN: Design and validation of a telesurgery system. Int. J. Robot. Res. 2009, 28, 1183-1197.
70. Hannaford, B.; Rosen, J.; Friedman, D.W.; King, H.; Roan, P.; Cheng, L.; Glozman, D.; Ma, J.; Kosari, S.N.; White, L. Raven-II: An open platform for surgical robotics research. IEEE Trans. Biomed. Eng. 2013, 60, 954-959.