Contextual-Based Image Inpainting: Infer, Match, and Translate

By Chao Yang, Zhe L. Lin, Xiaofeng Liu, Qin Huang, Hao Li, C.-C. Jay Kuo. 2017

227 citations recorded by the source corpus.

We study the task of image inpainting, which is to fill in the missing region of an incomplete image with plausible contents. To this end, we propose a learning-based approach to generate visually coherent completion given a high-resolution image with missing components. In order to overcome the difficulty to directly learn the distribution of high-dimensional image data, we divide the task into inference and translation as two separate steps and model each step with a deep neural network. We also use simple heuristics to guide the propagation of local textures from the boundary to the hole. We show that, by using such techniques, inpainting reduces to the problem of learning two image-feature translation functions in much smaller space and hence easier to train. We evaluate our method on several public datasets and show that we generate results of better visual quality than previous state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “A perceptual loss is also commonly applied, usually with VGG-16 [26, 47, 25, 27] or VGG-19 [51, 43, 32, 52] back-bones pretrained on ImageNet classification [39].”
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, methodology): “ the missing region. Next, a reﬁnement network sharpens the result using an attention mechanism by searching for a collection of background patches with the highest similarity to the coarse estimate. [43] takes a similar approach and introduces a “patch-swap” layer which replaces each patch inside t”
Free-Form Image Inpainting With Gated Convolution (2018, background): “The latter approach [15, 49, 45, 46, 38, 37, 48, 26, 52, 33, 35, 19] can exploit semantics learned from large scale datasets to synthesize contents in nonstationary images in an end-to-end fashion.”
Pluralistic Image Completion (2019, background): “[35] proposed PatchMatch-like ideas on feature domain.”
Image-to-Image Translation: Methods and Applications (2021, background): “e, image inpainting [14], [15], [16], [17], [18] and image outpainting [189], but they treat different occluded images.”
Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness (2021, methodology): “Some other methods divided image inpainting into two stages, and utilized explicitly defined intermediate clues such as edges/textures [37], [38], segmentation maps [12], image structure [13]”
Image Fine-grained Inpainting (2020, unknown): “This problem has been mitigated by GAN [6] or its improved version WGAN-GP [7] that is frequently utilized in image inpainting meth-ods [20, 9, 29, 17, 30, 24, 28, 26, 33, 31].”
Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting (2020, methodology): “Compared to traditional methods, these techniques have strong ability to learn adaptive and high-level features of disparate semantics and thus are more adept in hallucinating visually plausible contents especially when inpainting structured images like faces [15, 20, 24, 35, 36, 37], objects [18, 2”
Diversity-Generated Image Inpainting with Style Extraction (2019, background): “Recently, some works [20, 23] have proposed the use of spatial attention [24, 25] to obtain high-frequency details.”
Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting (2019, methodology): “Contextual attention layer [31] and Patch-swap layer [21] were proposed for filling missing pixels with similar patches from undamaged regions at high-level feature maps.”
Foreground-Aware Image Inpainting (2019, background): “Additionally, Song et al. [25] apply a pretrained image segmentation network to obtain the foreground mask of the corrupted image, then ﬁll the segmentation mask and use it to guide the completion of the image.”
Learning to Incorporate Structure Knowledge for Image Inpainting (2020, methodology): “Song et al.(2018) develop a similar coarse-to-ﬁne method and introduce a Patch-Swap module which can heuristically propagate the textures from surroundings to the holes.”
3D Photography Using Context-Aware Layered Depth Inpainting (2020, background): “Driven by the progress of convolutional neural networks, CNN-based methods have received considerable attention due to their ability to predict semantically meaningful contents that are not available in the known regions [44, 51, 21, 65, 66].”
Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations (2020, methodology): “These two-stage methods [29,24,26,40,27,41,21] typically generate an intermediate image with recovered structures in the first stage (i.”
Text-Guided Neural Image Inpainting (2020, background): “propose a bidirectional similarity synthesis approach [30] to better capture non-stationary information, but lead to high algorithm complexity.”
Learning Oracle Attention for High-Fidelity Face Completion (2020, background): “In order to generate visually realistic images, existing approaches typically adopt the non-local schemes that make use of contextual relations to fill in the missing pixels [32, 21, 39, 24].”