Simultaneous structure and texture image inpainting

By Marcelo Bertalmío, Luminita A. Vese, Guillermo Sapiro, Stanley J. Osher. 2003

1,374 citations recorded by the source corpus.

An algorithm for the simultaneous filling-in of texture and structure in regions of missing image information is presented. The basic idea is to first decompose the image into the sum of two functions with different basic characteristics, and then reconstruct each one of these functions separately with structure and texture filling-in algorithms. The first function used in the decomposition is of bounded variation, representing the underlying image structure, while the second function captures the texture and possible noise. The region of missing information in the bounded variation image is reconstructed using image inpainting algorithms, while the same region in the texture image is filled-in with texture synthesis techniques. The original image is then reconstructed adding back these two sub-images. The novel contribution of the paper is then in the combination of these three previously developed components: image decomposition with inpainting and texture synthesis, which permits the simultaneous use of filling-in algorithms that are suited for different image characteristics. Examples on real images show the advantages of this proposed approach.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Pluralistic Image Completion (2019, methodology): “Early image completion works [4, 7, 5, 8, 3, 13] focus only on steps 2 and 3 above, by assuming that gaps should be filled with similar content to that of the background.”
Eye In-painting with Exemplar Generative Adversarial Networks (2017, neutral): “In-painting has been studied extensively, both with and without deep networks [5, 35].”
Object removal by exemplar-based inpainting (2003, methodology): “The work in [5] decomposes the original image into two components; one of which is processed by inpainting and the other by texture synthesis.”
Learning generative models of mid-level structure in natural images (2012, background): “…shape is the representation that is required for many image processing tasks, such as image segmentation, or inpainting on the scale considered by Bertalmio et al. (2003) and Criminisi et al. (2004), who deal with the infilling of relatively large image regions, e.g. after the removal of a…”
Get Out of my Picture! Internet-based Inpainting (2009, background): “[8] being proposed, which try to preserve both large-scale salient structures across the region and fine textures.”
Learning how to inpaint from global image statistics (2003, methodology): “An alternative way to use our method in textured images is to use the decomposition into a “structure” image and a “texture image” proposed in [2] and apply our current method only to the structure image.”
Foreground-Aware Image Inpainting (2019, methodology): “Conventional inpainting methods [8, 6, 5, 26] typically fill missing pixels by matching and pasting patches based on low level features such as mean square difference of RGB values or SIFT descriptors [19].”
A framelet-based image inpainting algorithm (2008, background): “Many useful techniques have been proposed in recent years to address the problem, see, for example, [1, 2, 3, 9, 11, 12, 13, 14, 15, 26, 27, 29, 38].”
Interactive image completion with perspective correction (2006, background): “In the context of image completion there are some approaches working on a per-pixel basis [7, 16, 20], but fragment-based approaches [4, 9, 12, 31] usually produce superior results.”
Faceshop (2018, positive): “Pioneering work by Efros and Leung [2001] leverages this strategy for texture synthesis, and various researchers later extended it for image completion [Bertalmio et al. 2003; Criminisi et al. 2004; Drori et al. 2003].”
Recurrent Feature Reasoning for Image Inpainting (2020, background): “Image Inpainting Traditionally, inpainting algorithms attempt to find patches from the background area to inpaint the hole [1, 2, 3, 5, 12, 23, 7].”
Globally and locally consistent image completion (2017, methodology): “For image completion, several modifications such as optimal patch search have been proposed [Bertalmio et al. 2003; Criminisi et al. 2004; Drori et al. 2003].”
Inpainting and Zooming Using Sparse Representations (2009, methodology): “surrounding pixels/regions [21, 22], best correspondence maps [23]).”
Region Normalization for Image Inpainting (2019, methodology): “Traditional methods use diffusion-based (Bertalmio et al. 2000; Ballester et al. 2001; Esedoglu and Shen 2002; Bertalmio et al. 2003) or patch-based (Drori, Cohen-Or, and Yeshurun 2003; Barnes et al. 2009; Xu and Sun 2010; Darabi et al. 2012) methods to ﬁll the holes.”