Get Out of my Picture! Internet-based Inpainting

By Oliver Whyte, Josef Sivic, Andrew Zisserman. 2009

124 citations recorded by the source corpus.

Often when we review our holiday photos, we notice things we wish we could have avoided, such as vehicles, construction work, or simply other tourists. We cannot go back and retake the photo, so what can we do if we want to remove these things from our photos? We want to replace these sections of the image in a convincing way, preferably with what would really have been seen there without the occlusions. Previous work on this problem, often referred to as “inpainting”, are mainly applicable to small image regions, and rely largely on models of the local behaviour of natural images, e.g. [3, 5]. Recently, replacing large occlusions in photographs has been approached using images from the Internet [2, 7] or by combining several images captured at approximately the same time [1, 9]. In this paper we leverage recent advances in viewpoint invariant image search [8] to find other images of the same scene on the Internet. Beginning with a query image containing a target region to be replaced, we first use an online image search engine to retrieve images of the same scene, and take these to be a set of oracles. Since these images may have significant variations in viewpoint and lighting, we register each oracle to the query image using multiple homographies and a simple global photometric correction. We then use each oracle to propose a solution, by copying image data into the target region using Poisson blending. Finally, we use a Markov random field (MRF) formulation to combine the proposals into a single, occlusion-free result. Figure 1 shows the main stages of our system, and compares our result to those of two other methods.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Free-Form Image Inpainting With Gated Convolution (2018, background): “…2001; Barnes et al. 2009; Drori et al. 2003; Sun et al. 2005], structures [Huang et al. 2014], transformation or distortion information [Huang et al. 2013; Pavi´c et al. 2006] and image exemplars [Criminisi et al. 2004; Hays and Efros 2007; Kwatra et al. 2005; Whyte et al. 2009; Zhao et al. 2018].”
Semantic Image Inpainting with Deep Generative Models (2016, methodology): “Next we compare our method with nearest neighbor (NN) filling from the training dataset, which is a key component in retrieval based methods [10, 37].”
Data-driven visual similarity for cross-domain image matching (2011, methodology): “2008], scene and video completion [Wexler et al. ; Hays and Efros 2007; Whyte et al. 2009], photo restoration [Dale et al.”
Globally and locally consistent image completion (2017, unknown): “This was also extended to the particular case in which images of exactly the same scene are included in the database of images [Whyte et al. 2009].”
Text-Guided Neural Image Inpainting (2020, methodology): “Some methods are also based on patch match algorithms, but use external image sources [5, 43] such as search engine [34].”
What is a good evaluation measure for semantic segmentation? (2013, background): “Semantic segmentation has many potential applications including scene understanding [13], removing undesired objects from photographs [21], copy-pasting objects from one photograph to another, or local class-based image enhancement.”
Semantic Image Inpainting with Perceptual and Contextual Losses (2016, background): “To inpaint a scene, images of the same scenes are retrieved from the Internet [24].”
Guided Image Inpainting: Replacing an Image Region by Pulling Content From Another Image (2018, methodology): “Automated image inpainting is an established computer vision problem with many valuable applications [16, 37, 6, 9, 38, 27, 45].”
Image completion using planar structure guidance (2014, methodology): “Another example of using external database is through instance-level matching methods to fill in missing regions via appropriate geometric and photometric transformation of the retrieved image [Whyte et al. 2009].”