Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

By Shengyu Zhao, Jianwei Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I-Chao Chang, Yan Xu. 2021

359 citations recorded by the source corpus.

Numerous task-specific variants of conditional generative adversarial networks have been developed for image completion. Yet, a serious limitation remains that all existing algorithms tend to fail when handling large-scale missing regions. To overcome this challenge, we propose a generic new approach that bridges the gap between image-conditional and recent modulated unconditional generative architectures via co-modulation of both conditional and stochastic style representations. Also, due to the lack of good quantitative metrics for image completion, we propose the new Paired/Unpaired Inception Discriminative Score (P-IDS/U-IDS), which robustly measures the perceptual fidelity of inpainted images compared to real images via linear separability in a feature space. Experiments demonstrate superior performance in terms of both quality and diversity over state-of-the-art methods in free-form image completion and easy generalization to image-to-image translation. Code is available at https://github.com/zsyzzsoft/co-mod-gan.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, background): “CoModGAN [64] 109М MADF [67] 85М Figure 4: The side-by-side comparison of various inpainting systems on 512 × 512 images.”
MaskGIT: Masked Generative Image Transformer (2022, methodology): “Boundless [43] InﬁnityGAN [31] (cid:83) CoModGAN MaskGIT (Ours) Groundtruth”
Palette: Image-to-Image Diffusion Models (2021, background): “2020], Photoshop’s Content-aware Fill, and Co-ModGAN [Zhao et al. 2021].”
RePaint: Inpainting using Denoising Diffusion Probabilistic Models (2022, background): “[54], inspired by the StyleGAN2 [18] modulated convolutions, introduces a co-modulation layer for the inpainting task in order to improve both diversity and reconstruction.”
Aggregated Contextual Transformations for High-Resolution Image Inpainting (2021, background): “Recent SOTA approaches aim at achieving diverse image inpainting by Transformer-based [59] or StyleGAN-based [60] generative models.”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, methodology): “Overall, our model has comparable image quality with the state-of-the-art methods, and it performs favorably against all baselines in terms of known factor diversity, especially compared with CoModGAN [62] and CollageGAN [26].”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, methodology): “Following [75], we train our models for 50M images on Places365-Standard and 25M images on CelebA-HQ.”
One-Step Diffusion with Distribution Matching Distillation (2023, background): “…them, GANs have shown unprecedented quality in realism [4, 26– 28, 30, 67], particularly when the GAN loss can be combined with task-speciﬁc, auxiliary regression losses to mitigate training instability, ranging from paired image translation [24, 54, 79, 89] to unpaired image editing [37, 55, 94].”
Zero-shot Image-to-Image Translation (2023, methodology): “Figure 7 shows the results obtained by distilling using Co-Mod-GAN [67].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, neutral): “image matting [258], image inpainting [259], [260], image fusion [261], image completion [262], [263], and image classification [264].”
Scaling up GANs for Text-to-Image Synthesis (2023, neutral): “There exist skip connections at the same resolution, similar to CoModGAN [78].”