StructureFlow: Image Inpainting via Structure-Aware Appearance Flow

By Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H. Li, Shan Liu, Ge Li. 2019

365 citations recorded by the source corpus.

Image inpainting techniques have shown significant improvements by using deep neural networks recently. However, most of them may either fail to reconstruct reasonable structures or restore fine-grained textures. In order to solve this problem, in this paper, we propose a two-stage model which splits the inpainting task into two parts: structure reconstruction and texture generation. In the first stage, edge-preserved smooth images are employed to train a structure reconstructor which completes the missing structures of the inputs. In the second stage, based on the reconstructed structures, a texture generator using appearance flow is designed to yield image details. Experiments on multiple publicly available datasets show the superior performance of the proposed network.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “Several works suggest two-stage methods that use completion of other structure types as an intermediate step: salient edges in [32], semantic segmentation maps in [44], foreground object contours in [48], gradient maps in [52], and edge-preserved smooth images in [36].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “…al., 2018; Ding et al., 2018; Iizuka et al., 2017; Jiang et al., 2019; Jo & Park, 2019; Lahiri et al., 2017; Liu et al., 2019a; Nazeri et al., 2019; Ren et al., 2019; Sagong et al., 2019; Wang et al., 2018; Xie et al., 2019; Xiong et al., 2019; Yan et al., 2018; Yang et al., 2017; 2019a; Yu et…”
3D Photography Using Context-Aware Layered Depth Inpainting (2020, methodology): “Our inpainting model builds upon the recent two-stage approaches [41, 62, 47] but with two key differences.”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, background): “Recently, deep learning methods [8,18,33,35,48,50,52,53,55,59,28,57,61] dramatically boosted the quality, in terms of both visual quality and semantic coherence.”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, background): “Multi-stage generation has also received a great amount of attention, where intermediate clues like object edges [40], foreground contours [63], structures [44] and semantic segmentation maps [49] are extensively exploited.”
Learning Oracle Attention for High-Fidelity Face Completion (2020, methodology): “The second category of methods [31, 19, 27] typically involve training a deep CNN in an encoder-decoder structure to predict each pixel of the missing area.”
Flow-edge Guided Video Completion (2020, neutral): “, edges) to guide the content [25,33,41].”
Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations (2020, methodology): “These two-stage methods [29,24,26,40,27,41,21] typically generate an intermediate image with recovered structures in the first stage (i.”
Image Fine-grained Inpainting (2020, neutral): “Different from frequently-used two-stage processing [29], Sagong et al.”
UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation (2020, methodology): “We compare with the following baselines: context encoders (CE) [15], contextual attention (CA) [26], coheret semantic attention (CSA) [13], structureflow (SF) [16], shift-net (SN) [23], CVAE [19], BicycleGAN (BG) [31], and PICNet (PN) [29].”
Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness (2021, methodology): “StructureFlow [13] can be regarded as a two-stage solution which firstly reconstructs structure and then generates textures using appearance flow.”
Aggregated Contextual Transformations for High-Resolution Image Inpainting (2021, methodology): “An illustration of the Residual block (a) used in the state-of-the-art deep inpainting models [17], [42], [51] and the proposed AOT block (b).”
Implicit Neural Representations with Periodic Activation Functions (2020, methodology): “These inpainting methods are based on convolutional neural networks (CNNs) [27, 38] and generative adversarial networks (GANs) [22, 44, 28, 32].”
RePaint: Inpainting using Denoising Diffusion Probabilistic Models (2022, background): “Follow-up works have produced impressive results in recent years [15,20,30,34,50].”
Image Inpainting with Learnable Feature Imputation (2020, methodology): “(a) Input (b) GConv [48] (c) PIC [52] (d) SF [34] (e) Ours”