SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting

By Chao Yang, Yeji Shen, Peng Wang, Qin Huang, C.-C. Jay Kuo. 2018

197 citations recorded by the source corpus.

In this paper, we focus on image inpainting task, aiming at recovering the missing area of an incomplete image given the context information. Recent development in deep generative models enables an efficient end-to-end framework for image synthesis and inpainting tasks, but existing methods based on generative models don't exploit the segmentation information to constrain the object shapes, which usually lead to blurry results on the boundary. To tackle this problem, we propose to introduce the semantic segmentation information, which disentangles the inter-class difference and intra-class variation for image inpainting. This leads to much clearer recovered boundary between semantically different regions and better texture within semantically consistent segments. Our model factorizes the image inpainting process into segmentation prediction (SP-Net) and segmentation guidance (SG-Net) as two steps, which predict the segmentation labels in the missing area first, and then generate segmentation guided inpainting results. Experiments on multiple public datasets show that our approach outperforms existing methods in optimizing the image inpainting quality, and the interactive segmentation guidance provides possibilities for multi-modal predictions of image inpainting.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “Inpainting frameworks often incorporate style [26, 30, 30, 47, 32, 25] and feature matching [51, 44, 32, 16] losses.”
Free-Form Image Inpainting With Gated Convolution (2018, background): “The latter approach [15, 49, 45, 46, 38, 37, 48, 26, 52, 33, 35, 19] can exploit semantics learned from large scale datasets to synthesize contents in nonstationary images in an end-to-end fashion.”
SPGNet: Semantic Prediction Guidance for Scene Parsing (2019, background): “Multi-Stage Networks: Multi-stage networks [5,27,32,40, 41, 51, 57, 59, 66] have been widely used and explored in human pose estimation.”
Pluralistic Image Completion (2019, methodology): “This was followed by several CNN-based methods, which included combining global and local discriminators as adversarial loss [14], identifying closest features in the latent space of masked images [40], utilizing semantic labels to guide the completion network [36], introducing additional face parsi”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, background): “Multi-stage generation has also received a great amount of attention, where intermediate clues like object edges [40], foreground contours [63], structures [44] and semantic segmentation maps [49] are extensively exploited.”
Learning Oracle Attention for High-Fidelity Face Completion (2020, background): “Moreover, different types of structural information have been extracted by networks in order to act as prior knowledge for assisting image inpainting; for example, segmentation maps [25], object contours [28], edge [17], facial landmarks [23], and face parsing [13].”
Foreground-Aware Image Inpainting (2019, methodology): “[27] apply a pretrained image segmentation network to obtain the foreground mask of the corrupted image, then fill the segmentation mask and use it to guide the completion of the image.”
StructureFlow: Image Inpainting via Structure-Aware Appearance Flow (2019, background): “Spg-net [26] predicts the semantic segmentation labels of the missing areas as structural information.”
EdgeConnect: Structure Guided Image Inpainting using Edge Prediction (2019, background): “SPG-Net [46] also follows a two-stage model which uses semantic segmentation labels to guide the in-painting process.”
Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations (2020, methodology): “[29] utilize a segmentation prediction network to generate segmentation guidance for detail refinement around the hole region.”
UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation (2020, background): “, visually realistic and semantically correct) according to some rules [1, 21, 2, 15, 20, 25, 13].”
Coherent Semantic Attention for Image Inpainting (2019, unknown): “Learning approaches [38, 22, 31, 37, 41, 7, 42] often use deep learning and GAN strategy to generate pixels of the hole.”
Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness (2021, methodology): “Analogically, SPG-Net [12] factorizes the image inpainting into segmentation repair stage and segmentation guidance stage.”