High-Fidelity Pluralistic Image Completion with Transformers

By Ziyu Wan, Jingbo Zhang, Dongdong Chen, Jing Liao. 2021

309 citations recorded by the source corpus.

Image completion has made tremendous progress with convolutional neural networks (CNNs), because of their powerful texture modeling capacity. However, due to some inherent properties (e.g., local inductive prior, spatial-invariant kernels), CNNs do not perform well in understanding global structures or naturally support pluralistic completion. Recently, transformers demonstrate their power in modeling the long-term relationship and generating diverse results, but their computation complexity is quadratic to input length, thus hampering the application in processing high-resolution images. This paper brings the best of both worlds to pluralistic image completion: appearance prior reconstruction with transformer and texture replenishment with CNN. The former transformer recovers pluralistic coherent structures together with some coarse textures, while the latter CNN enhances the local texture details of coarse priors guided by the high-resolution masked images. The proposed method vastly outperforms state-of-the-art methods in terms of three aspects: 1) large performance boost on image fidelity even compared to deterministic completion methods; 2) better diversity and higher fidelity for pluralistic completion; 3) exceptional generalization ability on large masks and generic dataset, like ImageNet. Code and pre-trained models have been publicly released at https://github.com/raywzy/ICT.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

MaskGIT: Masked Generative Image Transformer (2022, background): “MaskGIT beats both DeepFill and HiFill in FID and IS by a significant margin, while achieving scores close to the state-of-the-art inpainting approaches such as CoModGAN [54] and ICT [47].”
Improving Diffusion Models for Inverse Problems using Manifold Constraints (2022, methodology): “We also include comparisons against supervised learning based base-lines: LaMa [43], AOT-GAN [52], ICT [49], and DSI [35].”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, methodology): “Due to quadratically increased computational complexity, existing works [55,68, 77] adopt transformers to synthesize low-resolution results, typically 32 × 32, for subsequent processing.”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, background): “As pointed out by [47], these metrics are more suitable for small mask cases where the synthesized contents are more likely to be similar to the ground truth.”
ShapeFormer: Transformer-based Shape Completion via Sparse Representation (2022, methodology): “Besides using higher resolution quantized features to obtain more accurate generation, another possible improvement to this issue is to include high-resolution features of the input in the decoding procedure as in a recent image inpainting technique [13].”
DiffIR: Efficient Diffusion Model for Image Restoration (2023, methodology): “LQ KDSR S -GAN [17] LDM [11] BSRGAN [21] Real-ESRGAN [15] DiffIR S 2 (Ours) LQ KDSR S -GAN [17] LDM [11] BSRGAN [21] Real-ESRGAN [15] DiffIR S 2 (Ours) LQ KDSR S -GAN [17] LDM [11] BSRGAN [21] Real-ESRGAN [15] DiffIR S 2 (Ours) LQ KDSR S -GAN [17] LDM [11] BSRGAN [21] Real-ESRGAN [15] DiffIR S 2 (Ou”
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows (2021, methodology): “Built upon the success of ViT, many efforts have been devoted to designing better Transformer based architectures for various vision tasks, including low-level image processing [4, 46], image classification [8,8,10,13,14,17,24,42,43,47,49,53–55], object detection [3, 59] and semantic segmentation [3”