Exemplar-based inpainting from a variational point of view

SIAM Journal on Mathematical Analysis

Volumn 42, Issue 3, 2010, Pages 1246-1285

  Aujol, Jean François ^a,b   Ladjal, Saïd ^c   Masnou, Simon ^d,e  

^a UNIVERSITÉ PARIS SACLAY   (France)

^b AIX MARSEILLE UNIVERSITY   (France)

^c TELECOM PARISTECH   (France)

^d UNIVERSITÉ LYON 1   (France)

^e SORBONNE UNIVERSITÉ   (France)

Author keywords

Functions of bounded variation; Image processing; Image restoration; Inpainting; Nonlocal functional; Texture synthesis

Indexed keywords

COMPUTER GRAPHICS; GEOMETRY; HEURISTIC METHODS; IMAGE PROCESSING; IMAGE TEXTURE;

EXPERIMENTAL EVIDENCE; FUNCTIONS OF BOUNDED VARIATIONS; GEOMETRIC INFORMATION; GEOMETRY RECONSTRUCTION; INPAINTING; NONLOCAL; TEXTURE RECONSTRUCTION; TEXTURE SYNTHESIS;

IMAGE RECONSTRUCTION;

EID: 77953489211     PISSN: 00361410     EISSN: None     Source Type: Journal    
DOI: 10.1137/080743883     Document Type: Article

References (91)

1. S. ALINEY, A property of the minimum vectors of a regularizing functional defined by means of the absolute norm, IEEE Trans. Signal Process., 45 (1997), pp. 913-917.
2. L. AMBROSIO, N. FUSCO, AND D. PALLARA, Functions of Bounded Variation and Free Discontinuity Problems, Oxford University Press, London, 2000.
3. L. AMBROSIO, N. GIGLI, AND G. SAVARÉ, Gradient Flows in Metric Spaces and in the Space of Probability Measures, Lectures Math. ETH Zürich 5681, Birkháuser, Basel, 2005.
4. L. AMBROSIO AND S. MASNOU, A direct variational approach to a problem arising in image reconstruction, Interfaces Free Bound., 5 (2003), pp. 63-81.
5. P. ARIAS, V. CASELLES, AND G. SAPIRO, A variational framework for non-local image inpainting, in Energy Minimization Methods in Computer Vision and Pattern Recognition, Lecture Notes in Comput. Sci., Springer, New York, 2009, pp. 345-358.
6. M. ASHIKHMIN, Synthesizing natural textures, in Proceedings of the ACM'01 Symposium on Interactive 3D Graphics, 2001, pp. 217-226.
7. G. AUBERT AND J.-F. AUJOL, Modeling very oscillating signals. Application to image processing, Appl. Math. Optim., 51 (2005), pp. 163-182.
8. G. AUBERT AND P. KORNPROBST, Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations, Appl. Math. Sci. 147, 2001, Springer, New York.
9. J.-F. AUJOL, G. AUBERT, L. BLANC-FĚRAUD, AND A. CHAMBOLLE, Image decomposition into a bounded variation component and an oscillating component, J. Math. Imaging Vision, 22 (2005), pp. 71-88.
10. J.-F. AUJOL AND A. CHAMBOLLE, Dual norms and image decomposition models, Internat. J. Comput. Vision, 63 (2005), pp. 85-104.
11. J.-F. AUJOL, G. GILBOA, T. CHAN, AND S. OSHER, Structure-texture image decomposition-modeling, algorithms, and parameter selection, Internat. J. Comput. Vision, 67 (2006), pp. 111-136.
12. D. AUROUX AND M. MASMOUDI, A one-shot inpainting algorithm based on the topological asymptotic analysis, Comput. Appl. Math., 25 (2006), pp. 1-17.
13. C. BALLESTER, M. BERTALMIO, V. CASELLES, G. SAPIRO, AND J. VERDERA, Filling-in by joint interpolation of vector fields and gray levels, IEEE Trans. Image Process., 10 (2001), pp. 1200-1211.
14. M. BERTALMIO, A. L. BERTOZZI, AND G. SAPIRO, Navier-Stokes, fluid dynamics and image and video inpainting, in Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR'01), Kauai, HI, Vol. I, IEEE Press, New York, 2001, pp. 355-362.
15. M. BERTALMIO, G. SAPIRO, V. CASELLES, AND C. BALLESTER, Image inpainting, in Proceedings of the ACM Conference Comp. Graphics (SIGGRAPH), New Orleans, 2000, pp. 417-424.
16. M. BERTALMIO, L. VESE, G. SAPIRO, AND S. OSHER, Simultaneous structure and texture image inpainting, IEEE Trans. Image Process., 12 (2003), pp. 882-889.
17. A. L. BERTOZZI, S. ESEDOGLU, AND A. GILLETTE, Inpainting of binary images using the Cahn-Hilliard equation, IEEE Trans. Image Process., 16 (2007), pp. 285-291.
18. J. BESAG, Spatial interaction and the statistical analysis of lattice systems, J. Roy. Statist. Soc. Ser. B, 36 (1974), pp. 192-236.
19. J. S. DE BONET, Multiresolution sampling procedure for analysis and synthesis of texture images, in ACM Trans. On Graphics (SIGGRAPH'97), 1997, pp. 361-368.
20. R. BORNARD, E. LECAN, L. LABORELLI, AND J.-H. CHENOT, Missing data correction in still images and image sequences, in Proceedings of the 10th ACM International Conference on Multimedia, 2002, pp. 355-361.
21. F. BORNEMANN AND T. MÄRZ, Fast image inpainting based on coherence transport, J. Math. Imaging Vision, 28 (2007), pp. 259-278.
22. A. BUADES, B. COLL, AND J. M. MOREL, A review of image denoising algorithms, with a new one, Multiscale Model. Simul., 4 (2005), pp. 490-530.
23. J. CAI, R. CHAN, AND Z. SHEN, A framelet-based image inpainting algorithm, Appl. Comput. Harmon. Anal., (2008), p. 24.
24. F. CAO, Y. GOUSSEAU, S. MASNOU, AND P. PÉREZ, Geometrically guided exemplar-based inpainting, 2009, submitted.
25. A. S. CARASSO, Singular integrals, image smoothness, and the recovery of texture in image deblurring, SIAM J. Appl. Math., 64 (2004), pp. 1749-1774.
26. A. CHAMBOLLE, A. GIACOMINI, AND M. PONSIGLIONE, Piecewise rigidity, J. Funct. Anal., 244 (2007), pp. 134-153.
27. T. CHAN AND J. SHEN, Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods, SIAM, Philadelphia, 2005.
28. 1 function approximation, SIAM J. Appl. Math., 65 (2005), pp. 1817-1837.
29. T. F. CHAN, S. H. KANG, AND J. SHEN, Euler's elastica and curvature-based inpainting, SIAM J. Appl. Math., 63 (2003), pp. 564-592.
30. T. F. CHAN AND J. SHEN, Mathematical models for local nontexture inpaintings, SIAM J. Appl. Math., 62 (2002), pp. 1019-1043.
31. T. F. CHAN AND J. SHEN, Non-texture inpainting by curvature-driven diffusion (CDD), J. Visual Comm. Image Rep., 12 (2001), pp. 436-449.
32. T. F. CHAN, J. SHEN, AND H.-M. ZHOU, Total variation wavelet inpainting, J. Math. Imaging Vision, 25 (2006), pp. 107-125.
33. A. CRIMINISI, P. PÉREZ, AND K. TOYAMA, Object removal by exemplar-based inpainting, in IEEE CVPR, 2 (2003), pp. 721-728.
34. G. R. CROSS AND A. K. JAIN, Markov random fields texture models, IEEE Trans. PAMI, 5 (1983), pp. 25-39.
35. J. DARBON AND M. SIGELLE, Image restoration with discrete constrained total variation part I: Fast and exact optimization, J. Math. Imaging Vision, 26 (2006), pp. 277-291.
36. I. DAUBECHIES AND G. TESCHKE, Variational image restoration by means of wavelets: Simultaneous decomposition, deblurring and denoising, Appl. Comput. Harmon. Anal., 19 (2005), pp. 1-16.
37. L. DEMANET, B. SONG, AND T. CHAN, Image inpainting by correspondence maps: A deterministic approach, in Proceedings of the 2nd IEEE Workshop on Variational, Geometric and Level Set Methods in Computer Vision (VLSM), Nice, 2003. Also available as UCLA CAM report 03-40, 2003.
38. I. DRORI, D. COHEN-OR, AND H. YESHURUN, Fragment-based image completion, in ACM Trans. Graphics (SIGGRAPH'03), 22 (2003), pp. 303-312.
39. 1 model: A geometric point of view, Multiscale Model. Simul., 18 (2009), pp. 154-189.
40. A. EFROS AND T. K. LEUNG, Texture synthesis by non-parametric sampling, in Proceedings of the IEEE Conference on Computer Vision, Vol. 2, Corfu, Greece, 1999, pp. 1033-1038.
41. A. A. EFROS AND W. T. FREEMAN, Image quilting for texture synthesis and transfer, in Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques SIGGRAPH '01, ACM, New York, pp. 341-346.
42. M. ELAD, J.-L STARCK, D. DONOHO, AND P. QUERRE, Simultaneous cartoon and texture image inpainting using Morphological Component Analysis, Appl. Comput. Harmon. Anal., 2005.
43. S. ESEDOGLU, S. RUUTH, AND R. TSAI, Threshold dynamics for shape reconstruction and disocc l u sion, in Proceedings of the IEEE International Conference on Image Processing, 2005.
44. S. ESEDOGLU AND J. SHEN, Digital image inpainting by the Mumford-Shah-Euler image model, European J. Appl. Math., 13 (2002), pp. 353-370.
45. M. D. FAIRCHILD, Color Appearance Models, 2nd ed., Wiley-IS&T, Chichester, UK, 2005.
46. W. T. FREEMAN, E. C. PASZTOR, AND O. T. CARMICHAEL, Learning low-level vision, Internat. J. Comput. Vision, 40 (2000), pp. 25-47.
47. I. GALIĆ, J. WEICKERT, M. WELK, A. BRUHN, A. BELYAEV, AND H.-P. SEIDEL, Image compression with anisotropic diffusion, J. Math. Imaging Vision, 31 (2008), pp. 255-269.
48. G. GILBOA AND S. OSHER, Nonlocal operators with applications to image processing, Multiscale Model. Simul., 7 (2007), pp. 1005-1028.
49. Y. GOUSSEAU AND J.-M. MOREL, Are natural images of bounded variation? SIAM J. Math. Anal., 33 (2001), pp. 634-648.
50. H. GROSSAUER AND O. SCHERZER, Using the complex Ginzburg-Landau equation for digital inpainting in 2D and 3D, in Scale Space Methods in Computer Vision, Lecture Notes in Comput. Sci. 2695, Springer, New York, 2003.
51. O. G. GULERYUZ, Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising: Part I - Theory, IEEE Trans. Image Process., 15 (2006), pp. 539-554.
52. O. G. GULERYUZ, Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising: Part II - Adaptive algorithms, IEEE Trans. Image Process., 15 (2006) pp. 555-571.
53. B. GUO, H. SHUM, AND Y. Q. XU, Chaos Mosaic: Fast and Memory Efficient Texture Synthesis, Research report MSR-TR-2000-32, Microsoft Research, 2000.
54. P. HARRISON, A non-hierarchical procedure for re-synthesis of complex texture, in Proceedings of the International Conference on Central Europe Comp. Graphics, Visual. and Comp. Vision, Plzen, Czech Rep., 2001.
55. D. J. HEEGER AND J. R. BERGEN, Pyramid-based texture analysis/synthesis, in Proceedings of ACM SIGGRAPH 1995, 1995, pp. 229-238.
56. A. N. HIRANI AND T. TOTSUKA, Combining frequency and spatial domain information for fast interactive image noise removal, in Proceedings of ACM SIGGRAPH 1996, 1996, pp. 269-276.
57. H. IGEHY AND L. PEREIRA, Image replacement through texture synthesis, in Proceedings of the 1997 International Conference on Image Processing (ICIP'97), Vol. 3, 1997, p. 186.
58. J. JIA AND C.-K. TANG, Image repairing: Robust image synthesis by adpative ND tensor voting, in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. I, 2003, pp. 643-650.
59. B. JULESZ, Visual pattern discrimination, IRE Trans. Inform. Theory, 8 (1962), pp. 84-92.
60. B. JULESZ, Textons, the elements of texture perception and their interactions, Nature, 290 (1981), pp. 91-97.
61. B. JULESZ, E. N. GILBERT, L. A. SHEPP, AND H. L. FRISH, Inability of humans to discriminate between visual textures that agree in second-order statistics-revisited, Perception, 2 (1973), pp. 391-405.
62. G. KANIZSA, Organization in Vision: Essays on Gestalt Perception, Präger, Westport, CT, 1979.
63. N. KOMODAKIS AND G. TZIRITAS, Image completion using efficient belief propagation via priority scheduling and dynamic pruning, IEEE Trans. Image Process., 16 (2007), pp. 2649-2661.
64. T. LE AND L. VESE, Image decomposition using total variation and div (BMO), SIAM J. Multiscale Model. Simul., 4 (2005), pp. 390-423.
65. G. P. LEONARDI, Personal communication.
66. G. P. LEONARDI AND S. MASNOU, Locality of the mean curvature of rectifiable varifolds, Adv. Calc. Var., 2 (2009), pp. 17-42.
67. E. LEVINA AND P. BICKEL, Texture synthesis and non-parametric resampling of random fields, Ann. Statist., 34 (2006), pp. 1751-1773.
68. L. LIANG, C. LIU, Y.-Q. XU, B. GUO, AND H. SHUM, Real-time texture synthesis by patch-based sampling, ACM Trans. Graphics, 20 (2001), pp. 127-150.
69. J. MALIK AND P. PERONA, Preattentive texture discrimination with early vision mechanism, J. Opt. Soc. Amer. A, 7 (1990), pp. 923-932.
70. S. MASNOU, Disocclusion: A variational approach using level lines, IEEE Trans. Image Process., 11 (2002), pp. 68-76.
71. S. MASNOU AND J.-M. MOREL, Level lines based disocclusion, in Proceedings of the IEEE International Conference on Image Processing, 3 (1998), pp. 259-263.
72. G. DAL MASO, An Introduction to G-Convergence, Birkhäuser, Boston, 1993.
73. P. MAUREL, J.-F. AUJOL, AND G. PEYRÉ, Locally parallel texture modeling, CMLA Preprint, 2009.
74. Y. MEYER, Oscillating patterns in image processing and nonlinear evolution equations, The Fifteenth Dean Jacqueline B. Lewis Memorial Lectures, University Lecture Series, Vol. 22, American Mathematical Society, Providence, RI, 2001.
75. K. Y. NI, Variational PDE-based image segmentation and inpainting with applications in computer graphics, Ph. D. thesis, UCLA, Los Angeles, 2008.
76. M. NIKOLOVA, A variational approach to remove outliers and impulse noise, J. Math. Imaging Vision, 20 (2004), pp. 99-120.
77. M. NITZBERG AND D. MUMFORD, AND T. SHIOTA, Filtering, Segmentation and Depth, Lecture Notes in Comput. Sci. 662, Springer, 1993.
78. M. OLIVEIRA, B. BOWEN, R. MCKENNA, AND Y.-S. CHANG, Fast digital image inpainting, in Proceedings of the International Conference on Visualization, Imaging and Image Processing (VIIP 2001), Marbella, Spain, 2001, pp. 261-266.
79. 1, Multiscale Model. Simul., 1 (2003), pp. 349-370.
80. P. PÉREZ, M. GANGNET, AND A. BLAKE, PatchWorks: Example-based Region Tiling for Image Editing, Research report MSR-TR-2004-04, Microsoft Research, 2004.
81. G. PEYRÉ, Manifold models for signals and images, Comput. Vis. Image Underst., 113 (2009), pp. 249-260.
82. A. RARES, M. J. T. REINDERS, AND J. BIEMOND, Edge-based image restoration, IEEE Trans. Image Process., 14 (2005), pp. 1454-1468.
83. E. P. SIMONCELLI AND J. PORTILLA, Texture characterisation via joint statistics of wavelet coefficient magnitudes, in Proceedings of the IEEE International Conference on Image Processing, 1998.
84. J. SUN, L. YUAN, J. JIA, AND H.-Y. SHUM, Image completion with structure propagation, in ACM Trans. Graphics, SIGGRAPH'05, 24 (2005), pp. 861-868.
85. D. TSCHUMPERLÉ, Fast anisotropic smoothing of multi-valued images using curvaturepreserving PDEs, Internat. J. Comput. Vision, 68 (2006), pp. 65-82.
86. D. TSCHUMPERLÉ AND L. BRUN, Image denoising and registration by PDE's on the space of patches, in Proceedings of the International Workshop on Local and Non-Local Approximation in Image Processing (LNLA'08), Lausanne, Switzerland, 2008.
87. L. A. VESE AND S. J. OSHER, Modeling textures with total variation minimization and oscillating pat t erns in i m age processing, J. Sci. Comput., 19 (2003), pp. 553-572.
88. L. Y. WEI AND M. LEVOY, Fast texture synthesis using tree-structured vector quantization, ACM Trans. Graphics (SIGGRAPH'00), (2000), pp. 479-488.
89. Y. WEXLER, E. SHECHTMAN, AND M. IRANI, Space-time completion of video, IEEE Trans. Pattern Anal. Mach. Intell., 29 (2007), pp. 463-476.
90. W. YIN AND D. GOLDFARB, Second-order cone programming methods for total variation-based image restoration, SIAM J. Sci. Comput., 27 (2005), pp. 622-645.
91. W. YIN, D. GOLDFARB, AND S. OSHER, A comparison of three total variation based texture extraction models, J. Vis. Commun. Image Represent., 18 (2007), pp. 240-252.