Shift-Net: Image Inpainting via Deep Feature Rearrangement

By Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, Shiguang Shan. 2018

503 citations recorded by the source corpus.

Deep convolutional networks (CNNs) have exhibited their potential in image inpainting for producing plausible results.However, in most existing methods, e.g., context encoder, the missing parts are predicted by propagating the surrounding convolutional features through a fully connected layer, which intends to produce semantically plausible but blurry result. In this paper, we introduce a special shift-connection layer to the U-Net architecture, namely Shift-Net, for filling in missing regions of any shape with sharp structures and fine-detailed textures. To this end, the encoder feature of the known region is shifted to serve as an estimation of the missing parts. A guidance loss is introduced on decoder feature to minimize the distance between the decoder feature after fully connected layer and the ground truth encoder feature of the missing parts. With such constraint, the decoder feature in missing region can be used to guide the shift of encoder feature in known region. An end-to-end learning algorithm is further developed to train the Shift-Net. Experiments on the Paris StreetView and Places datasets demonstrate the efficiency and effectiveness of our Shift-Net in producing sharper, fine-detailed, and visually plausible results.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, background): “Another popular group of choices for the completion network is architectures based on U-Net [37], such as [26, 50, 59, 27].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “, 2019), a multiplicity of task-specific approaches have been proposed to specialize GAN frameworks, mostly focused on hand-engineered multi-stage architectures, specialized operations, or intermediate structures like edges or contours (Altinel et al., 2018; Ding et al., 2018; Iizuka et al., 2017; J”
Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, neutral): “Based on the above assumption, Contextual-based Image Inpainting [7] and Shift-Net [48] were proposed by designing a “patch-swap” layer and a “shift-connection” layer respectively which high-frequency texture details from the existing regions to the missing regions are propagated.”
Pluralistic Image Completion (2019, methodology): “Comparison of our model with PatchMatch(PM) [3], Context Encoder(CE) [29] and Shift-Net [38] on images taken from the Paris [9] test set for center region completion.”
Image Inpainting via Generative Multi-column Convolutional Neural Networks (2018, methodology): “Besides encoder-decoder, U-net-like structure was also used [23].”
Aggregated Contextual Transformations for High-Resolution Image Inpainting (2021, background): “tion modeling, non-local modules tend to repeat patterns in missing regions and distort structures [22], [41], [42], [43].”
3D Photography Using Context-Aware Layered Depth Inpainting (2020, methodology): “Recent efforts include designing CNN architectures to better handle holes with irregular shapes [33, 67, 64] and two-”
Recurrent Feature Reasoning for Image Inpainting (2020, background): “A successfully inpainted image should exhibit coherence in both structure and texture between the estimated pixels and the background [27, 17].”
Flow-edge Guided Video Completion (2020, background): “Learning-based methods have shown promising results in image completion thanks to their ability to synthesize new content that may not exist in the original image [16, 29, 43, 45].”
Learning to Incorporate Structure Knowledge for Image Inpainting (2020, background): “Similar to the ideas of Context Attention and Patch-Swap, Yan et al.(2018) develop a shift-connect module by which the features of the known background at the encoding phase are directly shifted to ﬁll the missing areas at the decoding phase.”
Prior Guided GAN Based Semantic Inpainting (2020, background): “In [44], the authors introduce a shift-connection from encoder to decoder in an UNet architecture.”
Coherent Semantic Attention for Image Inpainting (2019, methodology): “[35] -PC: Partial Conv, proposed by Liu et al.”
Diversity-Generated Image Inpainting with Style Extraction (2019, methodology): “We compared our method with GL [39], CA [20], and SH [17] on images from the CelebA [33], Agricultural Disease, and MauFlex [1] validation sets.”
A Survey of Human-in-the-loop for Machine Learning (2021, methodology): “The image inpainting frameworks proposed by previous studies can be bifurcated as exemplarbased [122, 123, 124] approaches and deep learning-based [125, 126, 127] methods.”
Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness (2021, background): “Reference [34] introduced an U-Net structure [35] combined with special shift-connection layers for filling in holes of arbitrary shape.”