UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation

By Lei Zhao, Qihang Mo, Sihuan Lin, Zhizhong Wang, Zhiwen Zuo, Haibo Chen, Wei Xing, Dongming Lu. 2020

200 citations recorded by the source corpus.

Although existing image inpainting approaches have been able to produce visually realistic and semantically correct results, they produce only one result for each masked input. In order to produce multiple and diverse reasonable solutions, we present Unsupervised Cross-space Translation Generative Adversarial Network (called UCTGAN) which mainly consists of three network modules: conditional encoder module, manifold projection module and generation module. The manifold projection module and the generation module are combined to learn one-to-one image mapping between two spaces in an unsupervised way by projecting instance image space and conditional completion image space into common low-dimensional manifold space, which can greatly improve the diversity of the repaired samples. For understanding of global information, we also introduce a new cross semantic attention layer that exploits the long-range dependencies between the known parts and the completed parts, which can improve realism and appearance consistency of repaired samples. Extensive experiments on various datasets such as CelebA-HQ, Places2, Paris Street View and ImageNet clearly demonstrate that our method not only generates diverse inpainting solutions from the same image to be repaired, but also has high image quality.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “…al., 2019; Xiong et al., 2019; Yan et al., 2018; Yang et al., 2017; 2019a; Yu et al., 2018; Yu et al., 2019; Zeng et al., 2019; Lahiri et al., 2020; Zhao et al., 2020a; Li et al., 2020; Zhou et al., 2020), among which (Liu et al., 2018; Yu et al., 2019) introduce partial convolution and gated…”
Palette: Image-to-Image Diffusion Models (2021, background): “Some existing GAN-based models explicitly encourage diversity; [Yang et al. 2019b; Zhu et al. 2017b] propose methods for improving diversity of conditional GANs, and [Han et al. 2019; Zhao et al. 2020] explore diverse sample generation for image inpainting.”
Image-to-Image Translation: Methods and Applications (2021, background): “e, image inpainting [14], [15], [16], [17], [18] and image outpainting [189], but they treat different occluded images.”
High-Fidelity Pluralistic Image Completion with Transformers (2021, methodology): “Similar with [41], UCTGAN [39] project both the masked input and reference image into a common space via optimizing the KL-divergence between encoded features andN (0, I) distribution to achieve diversified sampling.”
RePaint: Inpainting using Denoising Diffusion Probabilistic Models (2022, background): “[53] propose a VAE-based network that trade-offs between diversity and reconstruction.”
Image Inpainting via Conditional Texture and Structure Dual Generation (2021, unknown): “The deep generative methods [35, 36, 8, 34, 38, 40, 37, 26, 12] are currently dominating, which effectively extract meaningful semantics from damaged images and recover reasonable contents with high visual fidelity, owing to their powerful feature learning ability.”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, background): “Recently, deep learning methods [8, 18, 33, 34, 46, 48, 50, 51, 53, 57, 28, 55, 59] dramatically boosted the quality, in terms of both visual quality and semantic coherence.”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, background): “UCTGAN [74] projects the instance image space and masked image space into a common low-dimensional manifold space via optimizing the KL-divergence to allow diverse generations of missing contents.”