Generative Image Inpainting with Contextual Attention
By Jiahui Yu, Zhe L. Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas S. Huang. 2018
2,556 citations recorded by the source corpus.
Recent deep learning based approaches have shown promising results for the challenging task of inpainting large missing regions in an image. These methods can generate visually plausible image structures and textures, but often create distorted structures or blurry textures inconsistent with surrounding areas. This is mainly due to ineffectiveness of convolutional neural networks in explicitly borrowing or copying information from distant spatial locations. On the other hand, traditional texture and patch synthesis approaches are particularly suitable when it needs to borrow textures from the surrounding regions. Motivated by these observations, we propose a new deep generative model-based approach which can not only synthesize novel image structures but also explicitly utilize surrounding image features as references during network training to make better predictions. The model is a feedforward, fully convolutional neural network which can process images with multiple holes at arbitrary locations and with variable sizes during the test time. Experiments on multiple datasets including faces (CelebA, CelebA-HQ), textures (DTD) and natural images (ImageNet, Places2) demonstrate that our proposed approach generates higher-quality inpainting results than existing ones. Code, demo and models are available at: https://github.com/JiahuiYu/generative_inpainting.
Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Graphics and Computer-Aided Design
- Eli Shechtman
- Jun-Yan Zhu
- Jonathan T. Barron
- Richard Zhang
- Alexei A. Efros
- Jia-Bin Huang
- Antonio Torralba
- Peter Wonka
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Chen Change Loy
- Jitendra Malik
- Yi Yang
In Generative Models
- Jun-Yan Zhu
- Eli Shechtman
- Richard Zhang
- Alexei A. Efros
- Ming-Hsuan Yang
- Bolei Zhou
- Daniel Cohen-Or
- Stefano Ermon
Citations in the corpus
- Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, background): “At a more global level, [56] introduced a coarse-to-fine framework that involves two networks.”
- Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “, 2019), a multiplicity of task-specific approaches have been proposed to specialize GAN frameworks, mostly focused on hand-engineered multi-stage architectures, specialized operations, or intermediate structures like edges or contours (Altinel et al., 2018; Ding et al., 2018; Iizuka et al., 2017; J”
- Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, methodology): “[14] introduced contextual attention (CA), which adopted a twostage coarse-to-fine network, where coarse prediction was further refined by computing the similarity between existing patches through a CA layer.”
- EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, methodology): “Note that `1 (%) errors for PConv are lower than those achieved by our method and those reported in CA [53] and GLCIC [22].”
- Free-Form Image Inpainting With Gated Convolution (2018, methodology): “…textures, but usually make critical failures in non-stationary cases like complicated scenes, faces and objects. latter approach [Iizuka et al. 2017; Yu et al. 2018] can exploit semantics learned from large scale datasets to fill contents in non-stationary images in an end-to-end fashion.”
- The Perception-Distortion Tradeoff (2017, background): “GAN-based image restoration Most recently, GAN-based methods have demonstrated unprecedented perceptual quality in super-resolution [3], [6], [9], [37], inpainting [7], [20], [38], compression [21], [39], [40], deblurring [41] and image-to-image translation [11], [22], [42].”
- Palette: Image-to-Image Diffusion Models (2021, methodology): “Tasks like inpainting [Yu et al. 2018b, 2019] and uncropping [Teterwak et al. 2019; Wang et al. 2019b] have often heavily relied on qualitative evaluation.”
- Image Inpainting for Irregular Holes Using Partial Convolutions (2018, methodology): “[36] remove the centered hole assumption and can handle irregular shaped holes, but do not perform an extensive quantitative analysis on a large number of images with irregular masks (51 test images in [8]).”
- MirrorGAN: Learning Text-To-Image Generation by Redescription (2019, neutral): “Similar ideas to our own have recently been used in CycleGAN and DualGAN, which handle the bi-directional translations within two domains together [43, 37, 1, 32], significantly advance image-to-image translation [14, 28, 15, 38, 23].”
- Discriminator Rejection Sampling (2018, methodology): “They have also been applied successfully to semi-supervised and unsupervised learning (Springenberg, 2015; Odena, 2016; Kumar et al., 2017), image editing (Yu et al., 2018; Ledig et al., 2017), and image style transfer (Zhu et al., 2017; Isola et al., 2017; Yi et al., 2017; Azadi et al., 2018).”
- Image2StyleGAN++: How to Edit the Embedded Images? (2019, neutral): “Such embedding algorithms can be used to analyze the limitations of GANs [5], do image inpainting [8, 39, 38, 36], local image editing [40, 17], global image transformations such as image morphing and expression transfer [1], and few-shot video generation [35, 34].”
- SinGAN: Learning a Generative Model From a Single Natural Image (2019, methodology): “Generative models for image manipulation The power of adversarial learning has been demonstrated by recent GAN-based methods, in many different image manipulation tasks [61, 10, 62, 8, 53, 56, 42, 53].”
- Image Inpainting via Generative Multi-column Convolutional Neural Networks (2018, methodology): “In fact, adversarial loss has already been used in recent methods [18, 25, 24, 9, 26] to learn multi-modality.”
- TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up (2021, background): “GANs are widely adopted in image translation [3, 4, 41], image enhancement [7, 42, 43], and image editing [44, 45].”
- Labels4Free: Unsupervised Segmentation using StyleGAN (2021, background): “Examples include various image and video editing tasks, image inpainting [13, 47, 46, 41], local image editing [42, 2], low bit-rate video conferencing [40], image super resolution [28, 12], image colorization [4, 27], and extracting 3D models [30].”
- Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Poorly-Drawn Sketches (2019, methodology): “Since there are many methods [10], [11] for ”well-drawn” sktech2image problem, we argue that such stroke calibration is indispensable for these methods to be well applied in some real applications, such as cultural relics or digital sketch generation for suspects, to produce realistic images.”
- Intermediate Layer Optimization for Inverse Problems using Deep Generative Models (2021, methodology): “Image reconstruction methods can be either supervised (Pathak et al., 2016; Richardson et al., 2020; Yu et al., 2018) or unsupervised (Menon et al., 2020; Bora et al., 2017; Pajot et al., 2019), see the recent survey (Ongie et al., 2020) for a unified presentation.”
- Paint by Example: Exemplar-based Image Editing with Diffusion Models (2022, background): “Deep neural networks can now produce compelling results for various low-level image editing tasks, such as image inpainting [61, 68], composition [42, 67, 72], colorization [54,71,74] and aesthetic enhancement [8,12], by learning from richly available paired data.”