Robust and accurate algorithm for wearable stereoscopic augmented reality with three indistinguishable markers

Electronics (Switzerland)

Volumn 5, Issue 3, 2016, Pages

  Cutolo, Fabrizio ^a   Freschi, Cinzia ^a   Mascioli, Stefano ^a   Parchi, Paolo D ^a   Ferrari, Mauro ^a,b   Ferrari, Vincenzo ^a,b  

^a UNIVERSITY OF PISA   (Italy)

^b UNIVERSITY HOSPITAL OF PISA   (Italy)

Author keywords

Augmented reality; Camera calibration; Image guided surgery; Machine vision; Wearable displays

Indexed keywords

AIR NAVIGATION; AUGMENTED REALITY; CAMERAS; COMPUTER VISION; NAVIGATION SYSTEMS; OPTIMIZATION; STEREO IMAGE PROCESSING; SURGERY; SURGICAL EQUIPMENT; WEARABLE TECHNOLOGY;

CAMERA CALIBRATION; CAMERA POSE ESTIMATION; IMAGE GUIDED SURGERY; LIGHTING CONDITIONS; SUB-PIXEL REGISTRATIONS; SURGICAL NAVIGATION SYSTEMS; VIDEO-BASED TRACKING; WEARABLE DISPLAYS;

WEARABLE COMPUTERS;

EID: 84988689863     PISSN: None     EISSN: 20799292     Source Type: Journal    
DOI: 10.3390/electronics5030059     Document Type: Article

References (64)

1. Milgram, P.; Kishino, F. A taxonomy of mixed reality visual-displays. IEICE Trans. Inf. Syst. 1994, E77-D (12), 1321–1329.
2. Kersten-Oertel, M.; Jannin, P.; Collins, D.L. The state of the art of visualization in mixed reality image guided surgery. Comput. Med. Imaging Graph. 2013, 37, 98–112.
3. Cuchet, E.; Knoplioch, J.; Dormont, D.; Marsault, C. Registration in neurosurgery and neuroradiotherapy applications. J. Image Guid. Surg. 1995, 1, 198–207.
4. Roberts, D.W.; Strohbehn, J.W.; Hatch, J.F.; Murray, W.; Kettenberger, H. A frameless stereotaxic integration of computerized tomographic imaging and the operating microscope. J. Neurosurg. 1986, 65, 545–549.
5. Edwards, P.J.; Hawkes, D.J.; Hill, D.L.G.; Jewell, D.; Spink, R.; Strong, A.; Gleeson, M. Augmentation of reality using an operating microscope for otolaryngology and neurosurgical guidance. J. Image Guid. Surg. 1995, 1, 172–178.
6. Edwards, P.J.; Hill, D.L.G.; Hawkes, D.J.; Spink, R.; Colchester, A.C.F.; Strong, A.; Gleeson, M. Neurosurgical guidance using the stereo microscope. In Computer Vision, Virtual Reality Robotics in Medicine; Ayache, N., Ed.; Springer: Berlin/Heidelberg, Germany, 1995; Volume 905, pp. 555–564.
7. Kersten-Oertel, M.; Jannin, P.; Collins, D.L. Dvv: Towards a taxonomy for mixed reality visualization in image guided surgery. Med. Imaging Augment. Real. 2010, 6326, 334–343.
8. Freysinger, W.; Gunkel, A.R.; Thumfart, W.F. Image-guided endoscopic ENT surgery. Eur. Arch. Oto-Rhino-Laryngol. 1997, 254, 343–346.
9. Caversaccio, M.; Giraldez, J.G.; Thoranaghatte, R.; Zheng, G.; Eggli, P.; Nolte, L.P.; Ballester, M.A.G. Augmented reality endoscopic system (ARES): Preliminary results. Rhinology 2008, 46, 156–158.
10. Baumhauer, M.; Simpfendorfer, T.; Muller-Stich, B.P.; Teber, D.; Gutt, C.N.; Rassweiler, J.; Meinzer, H.P.; Wolf, I. Soft tissue navigation for laparoscopic partial nephrectomy. Int. J. Comput. Assist. Radiol. Surg. 2008, 3, 307–314.
11. Ieiri, S.; Uemura, M.; Konishi, K.; Souzaki, R.; Nagao, Y.; Tsutsumi, N.; Akahoshi, T.; Ohuchida, K.; Ohdaira, T.; Tomikawa, M.; et al. Augmented reality navigation system for laparoscopic splenectomy in children based on preoperative ct image using optical tracking device. Pediatr. Surg. Int. 2012, 28, 341–346.
12. Haouchine, N.; Dequidt, J.; Berger, M.O.; Cotin, S. Deformation-based augmented reality for hepatic surgery. Stud. Health Technol. Inf. 2013, 184, 182–188.
13. Liao, H.; Nakajima, S.; Iwahara, M.; Kobayashi, E.; Sakuma, I.; Yahagi, N.; Dohi, T. Intra-operative real-time 3-D information display system based on integral videography. In Proceedings of the 4th International Conference on Medical Image Computing and Computer-Assisted Intervention—Miccai 2001, Utrecht, The Netherlands, 14–17 October 2001; Niessen, W., Viergever, M., Eds.; Springer: Berlin/Heidelberg, Germany, 2001; Volume 2208, pp. 392–400.
14. Liao, H.; Hata, N.; Nakajima, S.; Iwahara, M.; Sakuma, I.; Dohi, T. Surgical navigation by autostereoscopic image overlay of integral videography. IEEE Trans. Inf. Technol. Biomed. 2004, 8, 114–121.
15. Iseki, H.; Masutani, Y.; Iwahara, M.; Tanikawa, T.; Muragaki, Y.; Taira, T.; Dohi, T.; Takakura, K. Volumegraph (overlaid three-dimensional image-guided navigation). Clinical application of augmented reality in neurosurgery. Stereotact. Funct. Neurosurg. 1997, 68, 18–24.
16. Narita, Y.; Tsukagoshi, S.; Suzuki, M.; Miyakita, Y.; Ohno, M.; Arita, H.; Saito, Y.; Kokojima, Y.; Watanabe, N.; Moriyama, N.; et al. Usefulness of a glass-free medical three-dimensional autostereoscopic display in neurosurgery. Int. J. Comput. Assist. Radiol. Surg. 2014, 9, 905–911.
17. Liao, H.; Inomata, T.; Sakuma, I.; Dohi, T. Surgical navigation of integral videography image overlay for open MRI-guided glioma surgery. Med. Imaging Augment. Real. 2006, 4091, 187–194.
18. Liao, H.E.; Inomata, T.; Sakuma, I.; Dohi, T. 3-D augmented reality for mri-guided surgery using integral videography autostereoscopic image overlay. IEEE Trans. Biomed. Eng. 2010, 57, 1476–1486.
19. Suenaga, H.; Tran, H.H.; Liao, H.G.; Masamune, K.; Dohi, T.; Hoshi, K.; Mori, Y.; Takato, T. Real-time in situ three-dimensional integral videography and surgical navigation using augmented reality: A pilot study. Int. J. Oral Sci. 2013, 5, 98–102.
20. Ferrari, V.C.E. Wearable augmented reality light field optical see-through display to avoid user dependent calibrations: A feasibility study. In Proceedings of the IEEE Science and Information Conference, SAI 2016, London, UK, 13–15 July 2016; pp. 1211–1216.
21. Sielhorst, T.; Feuerstein, M.; Navab, N. Advanced medical displays: A literature review of augmented reality. J. Disp. Technol. 2008, 4, 451–467.
22. Birkfellner, W.; Figl, M.; Huber, K.; Watzinger, F.; Wanschitz, F.; Hummel, J.; Hanel, R.; Greimel, W.; Homolka, P.; Ewers, R.; et al. A head-mounted operating binocular for augmented reality visualization in medicine-design and initial evaluation. IEEE Trans. Med. Imaging 2002, 21, 991–997.
23. Ferrari, V.; Megali, G.; Troia, E.; Pietrabissa, A.; Mosca, F. A 3-D mixed-reality system for stereoscopic visualization of medical dataset. IEEE Trans. Biomed. Eng. 2009, 56, 2627–2633.
24. Sielhorst, T.; Bichlmeier, C.; Heining, S.M.; Navab, N. Depth perception—A major issue in medical AR: Evaluation study by twenty surgeons. In Proceedings of the 9th International Conference on Medical Image Computing and Computer-Assisted Intervention—MICCAI 2006, Copenhagen, Denmark, 1–6 October 2006; Volume 4190, pp. 364–372.
25. Rolland, J.P.; Fuchs, H. Optical versus video see-through mead-mounted displays in medical visualization. Presence Teleoperators Virtual Environ. 2000, 9, 287–309.
26. Kellner, F.; Bolte, B.; Bruder, G.; Rautenberg, U.; Steinicke, F.; Lappe, M.; Koch, R. Geometric calibration of head-mounted displays and its effects on distance estimation. IEEE Trans. Vis. Comput. Graph. 2012, 18, 589–596.
27. Genc, Y.; Tuceryan, M.; Navab, N. Practical solutions for calibration of optical see-through devices. Int. Symp. Mixed Augment. Real. Proc. 2002, 169–175.
28. Plopski, A.; Itoh, Y.; Nitschke, C.; Kiyokawa, K.; Klinker, G.; Takemura, H. Corneal-imaging calibration for optical see-through head-mounted displays. IEEE Trans. Vis. Comput. Graph. 2015, 21, 481–490.
29. Navab, N.; Heining, S.M.; Traub, J. Camera augmented mobile C-arm (CAMC): Calibration, accuracy study, and clinical applications. IEEE Trans. Med. Imaging 2010, 29, 1412–1423.
30. Ferrari, V.; Carbone, M.; Cappelli, C.; Boni, L.; Melfi, F.; Ferrari, M.; Mosca, F.; Pietrabissa, A. Value of multidetector computed tomography image segmentation for preoperative planning in general surgery. Surg. Endosc. 2012, 26, 616–626.
31. Marmulla, R.; Hoppe, H.; Muhling, J.; Eggers, G. An augmented reality system for image-guided surgery. Int. J. Oral Maxillofac. Surg. 2005, 34, 594–596.
32. Ferrari, V.; Viglialoro, R.M.; Nicoli, P.; Cutolo, F.; Condino, S.; Carbone, M.; Siesto, M.; Ferrari, M. Augmented reality visualization of deformable tubular structures for surgical simulation. Int. J. Med. Robot. 2015, 12, 231–240.
33. Franz, A.M.; Haidegger, T.; Birkfellner, W.; Cleary, K.; Peters, T.M.; Maier-Hein, L. Electromagnetic tracking in medicine—A review of technology, validation, and applications. IEEE Trans. Med. Imaging 2014, 33, 1702–1725.
34. Kanbara, M.; Okuma, T.; Takemura, H.; Yokoya, N. A stereoscopic video see-through augmented reality system based real-time vision-based registration. In Proceedings of the IEEE Virtual Reality Conference, New Brunswick, NJ, USA, 18–22 March 2000; pp. 255–262.
35. Haralick, R.M.; Lee, C.N.; Ottenberg, K.; Nolle, M. Review and analysis of solutions of the 3-point perspective pose estimation problem. Int. J. Comput. Vis. 1994, 13, 331–356.
36. Arun, K.S.; Huang, T.S.; Blostein, S.D. Least-squares fitting of 2 3-D point sets. IEEE Trans. Pattern Anal. Mach. Intell. 1987, 9, 699–700.
37. Kyto, M.; Nuutinen, M.; Oittinen, P. Method for measuring stereo camera depth accuracy based on stereoscopic vision. Three-Dimens. Imaging Interact. Meas. 2011, 7864.
38. Schneider, A.; Baumberger, C.; Griessen, M.; Pezold, S.; Beinemann, J.; Jurgens, P.; Cattin, P.C. Landmark-based surgical navigation. Clin. Image-Based Proc. Transl. Res. Med. Imaging 2014, 8361, 57–64.
39. Cutolo, F.; Parchi, P.D.; Ferrari, V. Video see through ar head-mounted display for medical procedures. In Proceedings of the 17th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Munich, Germany, 10–12 September 2014; pp. 393–396.
40. Ferrari, V.; Cutolo, F.; Calabro, E.M.; Ferrari, M. Hmd video see though AR with unfixed cameras vergence. In Proceedings of the 17th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Munich, Germany, 10–12 September 2014; pp. 265–266.
41. Megali, G.; Ferrari, V.; Freschi, C.; Morabito, B.; Turini, G.; Troia, E.; Cappelli, C.; Pietrabissa, A.; Tonet, O.; Cuschieri, A.; et al. Endocas navigator platform: A common platform for computer and robotic assistance in minimally invasive surgery. Int. J. Med. Robot. Comput. Assist. Surg. 2008, 4, 242–251.
42. Cutolo, F.; Siesto, M.; Mascioli, S.; Freschi, C.; Ferrari, M.; Ferrari, V. Configurable software framework for 2D/3D video see-through displays in medical applications. In Augmented Reality, Virtual Reality, and Computer Graphics: Third International Conference, Avr 2016, Lecce, Italy, 15–18 June 2016. Proceedings, Part II; De Paolis, T.L., Mongelli, A., Eds.; Springer International Publishing: Cham, Switzerland, 2016; pp. 30–42.
43. Zhang, Z.Y. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 2000, 22, 1330–1334.
44. Boker, S. The Representation of Color Metrics and Mappings in Perceptual Color Space; The University of Virginia, 1995. Available online: http://people.virginia.edu/~smb3u/ColorVision2/ColorVision2.html (accessed on 13 September 2016).
45. Kyriakoulis, N.; Gasteratos, A. Light-invariant 3D object’s pose estimation using color distance transform. In Proceedings of the 2010 IEEE International Conference on Imaging Systems and Techniques (IST), Thessaloniki, Greece, 1–2 July 2010; pp. 105–110.
46. Loukas, C.; Lahanas, V.; Georgiou, E. An integrated approach to endoscopic instrument tracking for augmented reality applications in surgical simulation training. Int. J. Med. Robot. Comput. Assist. Surg. 2013, 9, E34–E51.
47. Diotte, B.; Fallavollita, P.; Wang, L.J.; Weidert, S.; Thaller, P.H.; Euler, E.; Navab, N. Radiation-free drill guidance in interlocking of intramedullary nails. Lect. Notes Comput. Sci. 2012, 7510, 18–25.
48. Chang, C.C.; Chatterjee, S. Quantization-error analysis in stereo vision. In Proceedings of the Conference Record of The Twenty-Sixth Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, USA, 26–28 October 1992; pp. 1037–1041.
49. Fischler, M.A.; Bolles, R.C. Random sample consensus—A paradigm for model-fitting with applications to image-analysis and automated cartography. Commun. ACM 1981, 24, 381–395.
50. Wu, Y.H.; Hu, Z.Y. PnP problem revisited. J. Math. Imaging Vis. 2006, 24, 131–141.
51. Zhang, C.X.; Hu, Z.Y. Why is the danger cylinder dangerous in the P3P problem. Zidonghua Xuebao/Acta Autom. Sin. 2006, 32, 504–511.
52. Garro, V.; Crosilla, F.; Fusiello, A. Solving the PnP problem with anisotropic orthogonal procrustes analysis. In Proceedings of the 2012 Second Joint 3DIM/3DPVT Conference: 3D Imaging, Modeling, Processing, Visualization & Transmission 2012, Zurich, Switzerland, 13–15 October 2012; pp. 262–269.
53. Quan, L.; Lan, Z.D. Linear N-point camera pose determination. IEEE Trans. Pattern Anal. Mach. Intell. 1999, 21, 774–780.
54. Fiore, P.D. Efficient linear solution of exterior orientation. IEEE Trans. Pattern Anal. Mach. Intell. 2001, 23, 140–148.
55. Gao, X.S.; Hou, X.R.; Tang, J.L.; Cheng, H.F. Complete solution classification for the perspective-three-point problem. IEEE Trans. Pattern Anal. Mach. Intell. 2003, 25, 930–943.
56. Ansar, A.; Daniilidis, K. Linear pose estimation from points or lines. IEEE T Pattern Anal. 2003, 25, 578–589.
57. Lepetit, V.; Moreno-Noguer, F.; Fua, P. Epnp: An accurate o(N) solution to the pnp problem. Int. J. Comput. Vis. 2009, 81, 155–166.
58. Haralick, R.M.; Joo, H.; Lee, C.N.; Zhuang, X.H.; Vaidya, V.G.; Kim, M.B. Pose estimation from corresponding point data. IEEE Trans. Syst. Man Cyber. 1989, 19, 1426–1446.
59. Lu, C.P.; Hager, G.D.; Mjolsness, E. Fast and globally convergent pose estimation from video images. IEEE Trans. Pattern Anal. Mach. Intell. 2000, 22, 610–622.
60. Muller, M.; Rassweiler, M.C.; Klein, J.; Seitel, A.; Gondan, M.; Baumhauer, M.; Teber, D.; Rassweiler, J.J.; Meinzer, H.P.; Maier-Hein, L. Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. Int. J. Comput. Assist. Radiol. Surg. 2013, 8, 663–675.
61. Taketomi, T.; Okada, K.; Yamamoto, G.; Miyazaki, J.; Kato, H. Camera pose estimation under dynamic intrinsic parameter change for augmented reality. Comput. Graph.-UK 2014, 44, 11–19.
62. Badiali, G.; Ferrari, V.; Cutolo, F.; Freschi, C.; Caramella, D.; Bianchi, A.; Marchetti, C. Augmented reality as an aid in maxillofacial surgery: Validation of a wearable system allowing maxillary repositioning. J. Cranio-Maxillofacial Surg. 2014, 42, 1970–1976.
63. Cutolo, F.; Badiali, G.; Ferrari, V. Human-PnP: Ergonomic ar interaction paradigm for manual placement of rigid bodies. In Augmented Environments for Computer-Assisted Interventions; Linte, C., Yaniv, Z., Fallavollita, P., Eds.; Springer International Publishing: Cham, Switzerland, 2015; Volume 9365, pp. 50–60.
64. Cutolo, F.; Carbone, M.; Parchi, P.D.; Ferrari, V.; Lisanti, M.; Ferrari, M. Application of a new wearable augmented reality video see-through display to aid percutaneous procedures in spine surgery. In Augmented Reality, Virtual Reality, and Computer Graphics: Third International Conference, Avr 2016, Lecce, Italy, 15–18 June 2016. Proceedings, Part II; De Paolis, T.L., Mongelli, A., Eds.; Springer International Publishing: Cham, Switzerland, 2016; pp. 43–54.