Context Encoders: Feature Learning by Inpainting

By Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A. Efros. 2016

5,803 citations recorded by the source corpus.

We present an unsupervised visual feature learning algorithm driven by context-based pixel prediction. By analogy with auto-encoders, we propose Context Encoders - a convolutional neural network trained to generate the contents of an arbitrary image region conditioned on its surroundings. In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s). When training context encoders, we have experimented with both a standard pixel-wise reconstruction loss, as well as a reconstruction plus an adversarial loss. The latter produces much sharper results because it can better handle multiple modes in the output. We found that a context encoder learns a representation that captures not just appearance but also the semantics of visual structures. We quantitatively demonstrate the effectiveness of our learned features for CNN pre-training on classification, detection, and segmentation tasks. Furthermore, context encoders can be used for semantic inpainting tasks, either stand-alone or as initialization for non-parametric methods.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

An efficient framework for learning sentence representations (2018, methodology): “, 2015), inpainting (Pathak et al., 2016) and solving image jigsaw puzzles (Noroozi & Favaro, 2016) have been successfully used for learning visual feature representations.”
Momentum Contrast for Unsupervised Visual Representation Learning (2019, background): “, denoising auto-encoders [58], context autoencoders [48], or cross-channel auto-encoders (colorization) [64, 65].”
UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “…been applied to many Computer Vision tasks, such as image colorization (Zhang et al., 2016), solving jigsaw puzzles (Noroozi & Favaro, 2016; Trinh et al., 2019), inpainting (Pathak et al., 2016), rotation prediction (Gidaris et al., 2018), and relative location prediction (Doersch et al., 2015).”
Rethinking ImageNet Pre-Training (2018, background): “, for selfsupervised learning [5, 47, 33, 32]), as now we learn that even random initialization could produce excellent results.”
What makes ImageNet good for transfer learning? (2016, background): “Attempted self-supervised tasks include predictions of ego-motion [1, 20], spatial context [11, 33, 30], temporal context [45], and even color [51] and sound [32].”
Learning Visual Representations with Caption Annotations (2020, background): “tasks consist in predicting missing pieces in manipulated images, for instance context prediction [15], colorization [13,36,66], inpainting of missing portions [46], prediction of image rotations [19], spotting artifacts [31], or cross-channel prediction [67].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “In particular, relative patch prediction [23, 40], colorizing grayscale images [41, 42], image inpainting [43], image jigsaw puzzle [44], image super-resolution [45], and geometric transformations [46, 47] have been shown to be useful.”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Early works on self-supervised learning proposed hand-crafted pretext tasks, such as context prediction [15], colorization [17, 18], solving jigsaw puzzles [47], predicting rotation [19], inpainting [16], clustering [27], and generative modeling [48].”
Contrastive Multiview Coding (2019, background): “A powerful family of models for unsupervised representations are collected under the umbrella of “self-supervised” learning [64, 35, 85, 84, 78, 60, 83].”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “To learn a representation of images, the tasks could be: predicting the context [2], counting the objects [28], filling in missing parts of an image [31], recovering colors from grayscale images [47], or even solving a jigsaw puzzle [27].”
Unsupervised Representation Learning by Predicting Image Rotations (2018, background): “Speciﬁcally, we ﬁrst train a RotNet model on the training images of the ImageNet dataset and then we evaluate the performance of the self-supervised features on the image Context (Doersch et al., 2015) 19.7 26.7 31.9 32.7 30.9 Context Encoders (Pathak et al., 2016b) 18.2 23.2 23.4 21.9 18.4 Coloriza”
Learning Features by Watching Objects Move (2016, methodology): “Pathak et al .”
Colorful Image Colorization (2016, methodology): “resentations in this way goes back at least to autoencoders [6]. More recent works have explored feature learning via data imputation, where a held-out subset of the complete data is predicted (e.g., [7,8,9,10,11,12,13]). Our method follows in this line, and can be termed a cross-channel encoder. We”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, neutral): “We also compare with the ContextEncoder [36] (CE).”
Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “The subject has been studied in pre-deep learning era [1, 5, 13], and the progress accelerated in recent years through the use of deep and wide neural networks [26, 30, 25] and adversarial learning [34, 18, 56, 44, 57, 32, 54, 61].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “Then, (Köhler et al., 2014; Ren et al., 2015; Xie et al., 2012) adopt deep neural networks for image completion; (Pathak et al., 2016) first exploits conditional GANs.”