Pluralistic Image Completion

By Chuanxia Zheng, Tat-Jen Cham, Jianfei Cai. 2019

503 citations recorded by the source corpus.

Most image completion methods produce only one result for each masked input, although there may be many reasonable possibilities. In this paper, we present an approach for pluralistic image completion – the task of generating multiple and diverse plausible solutions for image completion. A major challenge faced by learning-based approaches is that usually only one ground truth training instance per label. As such, sampling from conditional VAEs still leads to minimal diversity. To overcome this, we propose a novel and probabilistically principled framework with two parallel paths. One is a reconstructive path that utilizes the only one given ground truth to get prior distribution of missing parts and rebuild the original image from this distribution. The other is a generative path for which the conditional prior is coupled to the distribution obtained in the reconstructive path. Both are supported by GANs. We also introduce a new short+long term attention layer that exploits distant relations among decoder and encoder features, improving appearance consistency. When tested on datasets with buildings (Paris), faces (CelebA-HQ), and natural images (ImageNet), our method not only generated higherquality completion results, but also with multiple and diverse plausible outputs.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, neutral): “Alternative attention mechanisms were suggested in [28, 47, 65].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “The lack of stochasticity is also observed in image completion (Cai & Wei, 2019; Ma et al., 2019; Zheng et al., 2019).”
Free-Form Image Inpainting With Gated Convolution (2018, background): “The latter approach [15, 49, 45, 46, 38, 37, 48, 26, 52, 33, 35, 19] can exploit semantics learned from large scale datasets to synthesize contents in nonstationary images in an end-to-end fashion.”
Palette: Image-to-Image Diffusion Models (2021, background): “…often require auxiliary objectives on structures, context, edges, contours and hand-engineered features [Iizuka et al. 2017; Kim et al. 2021a; Liu et al. 2020; Nazeri et al. 2019; Yi et al. 2020; Yu et al. 2018b, 2019], and they lack diversity in their outputs [Zhao et al. 2021; Zheng et al. 2019].”
Text-Guided Neural Image Inpainting (2020, methodology): “However, CVAE has been proven to have a disadvantage of grossly underestimating variances in the image completion scenario [44], which can be solved through the dual probabilistic structure.”
RePaint: Inpainting using Denoising Diffusion Probabilistic Models (2022, unknown): “[55] and Zhao et al .”
Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness (2021, background): “multiple and diverse plausible solutions for image completion [41].”
Image Fine-grained Inpainting (2020, unknown): “This problem has been mitigated by GAN [6] or its improved version WGAN-GP [7] that is frequently utilized in image inpainting meth-ods [20, 9, 29, 17, 30, 24, 28, 26, 33, 31].”
Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval (2020, unknown): “Partial Sketch: One of the most popular areas for studying incomplete or partial data is image inpainting [48, 51].”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, background): “[76] propose a probabilistically principled framework with two parallel paths, capable of producing multiple plausible solutions.”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, background): “Dealing with Partial Data One of the prolific areas studying incomplete or partial data is image inpainting [71, 85, 89], where the objective is to generate (or fill) the missing (or masked) region by conditioning on the overall region.”
High-Fidelity Pluralistic Image Completion with Transformers (2021, methodology): “To achieve diverse completion outputs, recent frameworks [41, 39] rely on optimizing the variational lower bound of instance likelihood.”
LaFIn: Generative Landmark Guided Face Inpainting (2019, background): “The competitors involved in the comparison include Context Encoder (CE) [21], Generative Face Completion (GFC) [15], Contextual Attention (CA) [34], Geometry Aware Face Completion (GAFC) [25], Pluralistic Image Completion (PIC) [37], and EdgeConnect (EC) [20].”
UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation (2020, methodology): “We compare with the following baselines: context encoders (CE) [15], contextual attention (CA) [26], coheret semantic attention (CSA) [13], structureflow (SF) [16], shift-net (SN) [23], CVAE [19], BicycleGAN (BG) [31], and PICNet (PN) [29].”
Learning Oracle Attention for High-Fidelity Face Completion (2020, methodology): “Secondly, CA [32], PIC [39], GConv [33] and PEN-Net [34] employ attention modules to learn contextual information for inpainting.”
Describe What to Change: A Text-guided Unsupervised Image-to-image Translation Approach (2020, background): “Most of the previous approaches rely on conditional inpaiting [3, 22, 56], in which the network fills a user-selected area with pixel values coherent with the user preferences and the context.”
Progressive Image Inpainting with Full-Resolution Residual Network (2019, methodology): “trategy and assign several (N) residual blocks in one dilation module to enhance its capability. Here we conduct two groups of experiments to explore the suitable N value. Damaged GLCIC[11] CA[32] PIC[35] Our FRRN Ground Truth Figure 5: Compare with previous work. Due to the long training time on Pl”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, methodology): “Thus [63] proposes a VAE-based [23] method allowing pluralistic image completion.”