Scene Completion Using Millions of Photographs

By James Hays, Alexei A. Efros. 2007

1,179 citations recorded by the source corpus.

What can you do with a million images? In this paper we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database that are not only seamless but also semantically valid. Our chief insight is that while the space of images is effectively infinite, the space of semantically differentiable scenes is actually not that large. For many image completion tasks we are able to find similar scenes which contain image fragments that will convincingly complete the image. Our algorithm is entirely data-driven, requiring no annotations or labelling by the user. Unlike existing image completion methods, our algorithm can generate a diverse set of results for each input image and we allow users to select among them. We demonstrate the superiority of our algorithm over existing image completion approaches

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Shortcut learning in deep neural networks (2020, background): “Examples include using nearest neighbours for scene completion and estimating geolocation [81, 82], object recognition with local features only [36], reasoning based on single cue words [59, 14] or answering questions about a movie without ever showing the movie to a model [83].”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “Earlier methods compute the output image by stitching pieces from a database of images [3, 14].”
Free-Form Image Inpainting With Gated Convolution (2018, methodology): “In computer graphics, two broad approaches to image inpainting exist: patch-based ones with low-level features and deep generative models with convolutional neural networks. former approach [Barnes et al. 2009; Efros and Freeman 2001; Efros and Leung 1999] can synthesize plausible stationary texture”
Semantic Image Inpainting with Deep Generative Models (2016, methodology): “Next we compare our method with nearest neighbor (NN) filling from the training dataset, which is a key component in retrieval based methods [10, 37].”
A note on the evaluation of generative models (2015, background): “Assigning sufﬁcient probability to all plausible images is important for compression, but it may be enough to generate a single plausible example in certain image reconstruction applications (e.g., Hays & Efros, 2007).”
Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (2015, background): “More ambitiously, entire portions of an image can be in-painted, given a sufficiently large training dataset [13].”
Recognizing indoor scenes (2009, neutral): “[5]).”
IM2GPS: estimating geographic information from a single image (2008, background): “On the other hand, the recent availability of truly gigantic image collections has made data association, such as brute-force scene matching, quite feasible [17, 4].”
Palette: Image-to-Image Diffusion Models (2021, background): “Early inpainting approaches [Barnes et al. 2009; Bertalmio et al. 2000; Hays and Efros 2007; He and Sun 2012] work well on textured regions but often fall short in generating semantically consistent structure.”
SUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels (2013, neutral): “This idea is a natural extension of data-driven approach from other vision tasks, such as scene completion [9].”
Generative Image Inpainting with Contextual Attention (2018, background): “Early works [3, 13] attempted to solve the problem using ideas similar to texture synthesis [9, 10], i.”
Sum-product networks: A new deep architecture (2011, neutral): “) Despite its simplicity, nearest neighbor can give quite good results if an image similar to the test one has been seen in the past [13].”
Image Inpainting for Irregular Holes Using Partial Convolutions (2018, background): “[36] remove the centered hole assumption and can handle irregular shaped holes, but do not perform an extensive quantitative analysis on a large number of images with irregular masks (51 test images in [8]).”
SIFT Flow: Dense Correspondence across Scenes and Its Applications (2011, neutral): “✦”
Scribbler: Controlling Deep Image Synthesis with Sketch and Color (2016, neutral): “Previously, the most successful methods tended to be non-parametric approaches which found clever ways to reuse existing image fragments [30, 9, 15, 22, 6, 3].”
High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis (2016, background): “These approaches assume that regions surrounded by similar context likely possess similar content [19].”
TAC-GAN - Text Conditioned Auxiliary Classifier Generative Adversarial Network (2017, background): “A considerable amount of work on image modelling focused on other tasks such as image inpainting ( e.g ., [8]) or texture generation ( e.g ., [4]).”