Monocular 3D Reconstruction and Augmentation of Elastic Surfaces with Self-Occlusion Handling

IEEE Transactions on Visualization and Computer Graphics

Volumn 21, Issue 12, 2015, Pages 1363-1376

  Haouchine, Nazim ^a   Dequidt, Jeremie ^b   Berger, Marie Odile ^c   Cotin, Stephane ^a  

^a INRIA   (France)

^b UNIV LILLE   (France)

^c UNIVERSITÉ DE LORRAINE   (France)

Author keywords

Augmented reality; Computational modeling; Context modeling; Deformable models; Mathematical model; Shape analysis; Three dimensional displays; Virtual reality

Indexed keywords

AUGMENTED REALITY; DEFORMATION; IMAGE RECONSTRUCTION; MATHEMATICAL MODELS; RECOVERY; SHAPE OPTIMIZATION; VIRTUAL REALITY;

COMPUTATIONAL MODEL; CONTEXT MODELING; DEFORMABLE MODELS; DEFORMATION PROPERTIES; ENERGY MINIMIZATION PROBLEM; MONOCULAR 3D RECONSTRUCTION; MONOCULAR VIDEO SEQUENCES; SHAPE ANALYSIS;

THREE DIMENSIONAL DISPLAYS;

SILICONE DERIVATIVE;

ALGORITHM; ANIMAL; COMPUTER GRAPHICS; HUMAN; LIVER; PIG; PROCEDURES; SURFACE PROPERTY; SURGERY; THEORETICAL MODEL; THREE DIMENSIONAL IMAGING;

ALGORITHMS; ANIMALS; COMPUTER GRAPHICS; HUMANS; IMAGING, THREE-DIMENSIONAL; LIVER; MODELS, THEORETICAL; SILICONES; SURFACE PROPERTIES; SWINE;

EID: 84946550266     PISSN: 10772626     EISSN: None     Source Type: Journal    
DOI: 10.1109/TVCG.2015.2452905     Document Type: Review

References (63)

1. M. Silcowitz-Hansen, S. Niebe, and K. Erleben, "A nonsmooth nonlinear conjugate gradient method for interactive contact force problems," Vis. Comput., vol. 26, no. 6-8, pp. 893-901, Jun. 2010.
2. S. Nicolau, L. Soler, D. Mutter, and J. Marescaux, "Augmented reality in laparoscopic surgical oncology," Surgical Oncol., vol. 20, no. 3, pp. 189-201, 2011.
3. H. Kenngott, M. Wagner, M. Gondan, F. Nickel, M. Nolden, A. Fetzer, J. Weitz, L. Fischer, S. Speidel, H.-P. Meinzer, D. Bckler, M. Bchler, and B. Mller-Stich, "Real-time image guidance in Laparoscopic liver surgery: First clinical experience with a guidance system based on intraoperative ct imaging," Surgical Endoscopy, vol. 28, no. 3, pp. 933-940, 2014.
4. N. Haouchine, J. Dequidt, I. Peterlik, E. Kerrien, M.-O. Berger, and S. Cotin, "Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery," in Proc. IEEE Int. Symp. Mixed Augmented Reality, Oct. 2013, pp. 199-208.
5. P. Pratt, D. Stoyanov, M. Visentini-Scarzanella, and G.-Z. Yang, "Dynamic guidance for robotic surgery using image-constrained biomechanical models," in Proc. 13th Int. Conf. Med. Image Comput. Comput.-Assisted Intervention, 2010, vol. 6361, pp. 77-85.
6. S. Suwelack, S. Röhl, S. Bodenstedt, D. Reichard, R. Dillmann, T. dos Santos, L. Maier-Hein, M. Wagner, J. Wünscher, H. Kenngott B. P. Müller, and S. Speidel, "Physics-based shape matching for intraoperative image guidance," Med. Phys., vol. 41, no. 11, pp. 111901, 2014.
7. N. Haouchine, S. Cotin, I. Peterlik, J. Dequidt, M. Lopez, E. Kerrien, and M.-O. Berger, "Impact of soft tissue heterogeneity on augmented reality for liver surgery," IEEE Trans. Vis. Comput. Graph., vol. 21, no. 5, pp. 584-597, May 2015.
8. M. Sermesant, C. Forest, X. Pennec, H. Delingette, and N. Ayache, "Deformable biomechanical models: Application to 4D cardiac image analysis," Med. Image Anal., vol. 7, no. 4, pp. 475-488, 2003.
9. M. Baumann, P. Mozer, V. Daanen, and J. Troccaz, "Prostate biopsy assistance system with gland deformation estimation for enhanced precision," in Proc. 12th Int. Conf. Med. Image Comput. Comput.-Assisted Intervention: Part I, 2009, pp. 67-74.
10. H. Courtecuisse, J. Allard, P. Kerfriden, S. P. Bordas, S. Cotin, and C. Duriez, "Real-time simulation of contact and cutting of heterogeneous Soft-tissues," Med. Image Anal., vol. 18, no. 2, pp. 394-410, 2014.
11. M. Kass, A. Witkin, and D. Terzopoulos, "Snakes: Active contour models," Int. J. Comput. Vis., vol. 1, no. 4, pp. 321-331, 1988.
12. T. Shen, H. Li, and X. Huang, "Active volume models for medical image segmentation," IEEE Trans. Med. Imaging, vol. 30, no. 3, pp. 774-791, Mar. 2011.
13. J. Schaerer, C. Casta, J. Pousin, and P. Clarysse, "A dynamic elastic model for segmentation and tracking of the heart in mr image sequences," Med. Image Anal., vol. 14, no. 6, pp. 738-749, 2010.
14. S. Speidel, S. Roehl, S. Suwelack, R. Dillmann, H. Kenngott, and B. Mueller-Stich, "Intraoperative surface reconstruction and biomechanical modeling for soft tissue registration," in Proc. Joint Workshop New Technol. Comput. Robot Assisted Surgery, 2011.
15. G. S. P. Miller, "The motion dynamics of snakes and worms," in Proc. ACM 15th Annu. Conf. Comput. Graph. Interactive Techn., 1988, pp. 169-173.
16. L. Ibai, A. Hugo, A. Iker, and B. Diego, "Real-time deformation, registration and tracking of solids based on physical simulation," in Proc. IEEE Int. Symp. Mixed Augmented Reality, 2014, pp. 165-170.
17. T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, "Active shape models - their training and application," Comput. Vis. Image Understanding, vol. 61, no. 1, pp. 38-59, Jan. 1995.
18. I. Matthews and S. Baker, "Active appearance models revisited," Int. J. Comput. Vis., vol. 60, no. 2, pp. 135-164, Nov. 2004.
19. M. Salzmann, J. Pilet, S. Ilic, and P. Fua, "Surface deformation models for nonrigid 3d shape recovery," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 8, pp. 1481-1487, Aug. 2007.
20. F. L. Bookstein, "Principal warps: Thin-plate splines and the decomposition of deformations," IEEE Trans. Pattern Anal. Mach. Intell., vol. 11, no. 6, pp. 567-585, Jun. 1989.
21. T. W. Sederberg and S. R. Parry, "Free-form deformation of solid geometric models," SIGGRAPH Comput. Graph., vol. 20, no. 4, pp. 151-160, Aug. 1986.
22. A. Bartoli, "Maximizing the predictivity of smooth deformable image warps through Cross-validation," J. Math. Imaging Vis., vol. 31, no. 2-3, pp. 133-145, Jul. 2008.
23. J. Lim and M. Yang, "A direct method for modeling non-rigid motion with thin plate spline," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 12005, pp. 1196-1202.
24. R. Richa, A. P. L. Bó, and P. Poignet, "Robust 3d visual tracking for Robotic-assisted cardiac interventions," in Proc. 13th Int. Conf. Med. Image Comput. Comput.-Assisted Intervention, 2010, pp. 267-274.
25. D. Rueckert, L. Sonoda, C. Hayes, D. Hill, M. Leach, and D. Hawkes, "Nonrigid registration using free-form deformations: Application to breast mr images," IEEE Trans. Med. Imaging, vol. 18, no. 8, pp. 712-721, Aug. 1999.
26. A. Bartoli and A. Zisserman, "Direct estimation of Non-rigid registration," in Proc. Brit. Mach. Vis. Conf., 2004, pp. 899-908.
27. D. Pizarro and A. Bartoli, "Feature-based deformable surface detection with Self-occlusion reasoning," Int. J. Comput. Vis., vol. 97, no. 1, pp. 54-70, 2012.
28. J. Pilet, V. Lepetit, and P. Fua, "Fast non-rigid surface detection, registration and realistic augmentation," Int. J. Comput. Vis., vol. 76, no. 2, pp. 109-122, Feb. 2008.
29. J. Zhu and M. R. Lyu, "Progressive finite newton approach to real-time nonrigid surface detection," in Proc. Conf. Comput. Vis. Pattern Recog., 2007, pp. 1-8.
30. V. Gay-Bellile, A. Bartoli, and P. Sayd, "Direct estimation of non-rigid registrations with image-based self-occlusion reasoning," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 1, pp. 87-104, Jan. 2010.
31. A. Hilsmann, D. C. Schneider, and P. Eisert, "Realistic cloth augmentation in single view video under occlusions," Comput. Graph., vol. 34, no. 5, pp. 567-574, Oct. 2010.
32. S. Shen, W. Shi, and Y. Liu, "Monocular 3-d tracking of inextensible deformable surfaces under l2-norm," Trans. Img. Process., vol. 19, no. 2, pp. 512-521, Feb. 2010.
33. M. Perriollat, R. Hartley, and A. Bartoli, "Monocular template-based reconstruction of inextensible surfaces," Int. J. Comput. Vis., vol. 95, no. 2, pp. 124-137, 2011.
34. M. Salzmann and P. Fua, "Linear local models for monocular reconstruction of deformable surfaces," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 5, pp. 931-944, May 2011.
35. M. Salzmann, R. Hartley, and P. Fua, "Convex optimization for deformable surface 3-d tracking," in Proc. IEEE 11th Int. Conf. Comput. Vis., Oct. 2007, pp. 1-8.
36. F. Moreno-Noguer, M. Salzmann, V. Lepetit, and P. Fua, "Capturing 3d stretchable surfaces from single images in closed form," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2009, pp. 1842-1849.
37. A. Agudo, B. Calvo, and J. M. M. Montiel, "Finite element based sequential Bayesian Non-rigid structure from motion," in Proc, IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 1418-1425.
38. S. Wuhrer, J. Lang, and C. Shu, "Tracking complete deformable objects with finite elements," in Proc. 2nd Int. Conf. 3D Imaging, Modeling, Process., Vis. Transmission, 2012, pp. 1-8.
39. A. Malti, R. Hartley, A. Bartoli, and J.-H. Kim, "Monocular template-based 3d reconstruction of extensible surfaces with local linear elasticity," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 1522-1529.
40. N. Haouchine, J. Dequidt, M.-O. Berger, and S. Cotin, "Single view augmentation of elastic objects," in Proc. IEEE Int. Symp. Mixed Augmented Reality, Sep. 2014, pp. 199-208.
41. S. F. Gibson and B. Mirtich, "A survey of deformable modeling in computer graphics," Mitsubishi Electron. Res. Lab., Nov. 1997.
42. A. Nealen, M. Müller, R. Keiser, E. Boxerman, and M. Carlson, "Physically based deformable models in computer graphics," Comput. Graph. Forum, vol. 25, no. 4, pp. 809-836, 2006.
43. E. Sifakis and J. Barbic, "Fem simulation of 3d deformable solids: A practitioner's guide to theory, discretization and model reduction," in Proc. ACM SIGGRAPH 2012 Courses, New York, NY, USA, 2012, pp. 20:1-20:50.
44. J. Barbic and D. L. James, "Real-time subspace integration for st. Venant-Kirchhoff deformable models," ACM Trans. Graph., vol. 24, no. 3, pp. 982-990, 2005.
45. H. Zhong, M. P. Wachowiak, and T. M. Peters, "A real time finite element based tissue simulation method incorporating nonlinear elastic behavior," Comput. Methods Biomechanics Biomed. Eng., vol. 8, no. 3, pp. 177-189, 2005.
46. R. Kikuuwe, H. Tabuchi, and M. Yamamoto, "An edge-based computationally efficient formulation of saint Venant-Kirchhoff tetrahedral finite elements," ACM Trans. Graph., vol. 28, no. 1, p. 8, 2009.
47. Cgal, Computational Geometry Algorithms Library. [Online]. Available: http://www.cgal.org, 2015.
48. C. Guébert, C. Duriez, and L. Grisoni, "Unified processing of constraints for interactive simulation," in Proc. Workshop Virtual Reality Interactions Phys. Simul., 2008, pp. 21-29.
49. A. Shaji, A. Varol, P. Fua, Yashoteja, A. Jain, and S. Chandran, "Resolving occlusion in multiframe reconstruction of deformable surfaces," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog. Workshops, Jun. 2011, pp. 31-36.
50. H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, "Speeded-up robust features (SURF)," Comput. Vis. Image Understanding, vol. 110, no. 3, pp. 346-359, Jun. 2008.
51. M. C. Yip, D. G. Lowe, S. E. Salcudean, R. N. Rohling, and C. Y. Nguan, "Tissue tracking and registration for Image-guided surgery," IEEE Trans. Med. Imaging, vol. 31, no. 11, pp. 2169-2182, Nov. 2012.
52. F. Faure, C. Duriez, H. Delingette, J. Allard, B. Gilles, S. Marchesseau, H. Talbot, H. Courtecuisse, G. Bousquet, I. Peterlik, and S. Cotin, "SOFA: A multi-model framework for interactive physical simulation," in Soft Tissue Biomechanical Modeling for Computer Assisted Surgery, Y. Payan, Ed. New York, NY, USA: Springer, Jun. 2012.
53. J. Bosman, N. Haouchine, J. Dequidt, I. Peterlik, S. Cotin, and C. Duriez, "The role of ligaments: Patient-specific or scenario-specific?" in Biomedical Simulation, F. Bello and S. Cotin, eds. New York, NY, USA: Springer, 2014, vol. 8789, pp. 228-232.
54. K. Miller and J. Lu, "On the prospect of patient-specific biomechanics without patient-specific properties of tissues," J. Mechanical Behavior Biomedical Materials, vol. 27, no. 0, pp. 154-166, 2013.
55. N. Snavely, S. M. Seitz, and R. Szeliski, "Photo tourism: Exploring photo collections in 3d," in ACM SIGGRAPH. New York, NY, USA, 2006, pp. 835-846.
56. A. Bartoli, Y. Gerard, F. Chadebecq, and T. Collins, "On template-based reconstruction from a single view: Analytical solutions and proofs of well-posedness for developable, isometric and conformal surfaces," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2012, pp. 2026-2033.
57. S. Cotin, H. Delingette, and N. Ayache, "Real-time elastic deformations of soft tissues for surgery simulation," IEEE Trans. Vis. Comput. Graph., vol. 5, no. 1, pp. 62-73, Jan. 1999.
58. B. D. Lucas and T. Kanade, "An iterative image registration technique with an application to stereo vision," in Proc. 7th int. joint conf. Artificial intell. - Volume 2, ser. IJCAI '81. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1981, pp. 674-679.
59. P. A. Yushkevich, J. Piven, H. Cody Hazlett, R. Gimpel Smith, S. Ho, J. C. Gee, and G. Gerig, "User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability," Neuroimage, vol. 31, no. 3, pp. 1116-1128, 2006.
60. I. Peterlík, C. Duriez, and S. Cotin, "Modeling and real-time simulation of a vascularized liver tissue," in Proc. Med. Image Comput. Comput.-Assisted Intervention, 2012, pp. 50-57.
61. L. Maier-Hein, P. Mountney, A. Bartoli, H. Elhawary, D. Elson, A. Groch, A. Kolb, M. Rodrigues, J. Sorger, S. Speidel, and D. Stoyanov, "Optical techniques for 3d surface reconstruction in computer-assisted Laparoscopic surgery," Med. Image Anal., vol. 17, pp. 974-996, 2013.
62. R. Plantefeve, I. Peterlik, H. Courtecuisse, R. Trivisonne, J.-P. Radoux, and S. Cotin, "Atlas-based transfer of boundary conditions for biomechanical simulation," in Proc. Int. Conf. Med. Image Comput. Comput.-Assisted Intervention, 2014, pp. 33-40.
63. C. J. Paulus, L. Untereiner, H. Courtecuisse, S. Cotin, and D. Cazier, "Virtual cutting of deformable objects based on efficient topological operations," The Vis. Comput., vol. 31, no. 6-8, pp. 831-841, 2015.