Colorization Transformer

By Manoj Kumar, Dirk Weissenborn, Nal Kalchbrenner. 2021

194 citations recorded by the source corpus.

We present the Colorization Transformer, a novel approach for diverse high fidelity image colorization based on self-attention. Given a grayscale image, the colorization proceeds in three steps. We first use a conditional autoregressive transformer to produce a low resolution coarse coloring of the grayscale image. Our architecture adopts conditional transformer layers to effectively condition grayscale input. Two subsequent fully parallel networks upsample the coarse colored low resolution image into a finely colored high resolution image. Sampling from the Colorization Transformer produces diverse colorings whose fidelity outperforms the previous state-of-the-art on colorising ImageNet based on FID results and based on a human evaluation in a Mechanical Turk test. Remarkably, in more than 60% of cases human evaluators prefer the highest rated among three generated colorings over the ground truth. The code and pre-trained checkpoints for Colorization Transformer are publicly available at https://github.com/google-research/google-research/tree/master/coltran

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Transformers in Vision: A Survey (2021, methodology): “2D Image Pretraining without labels and few-shot learning with Class labels Normalized Cross-entropy Image Colorization ColTran [24] Conditional Row/column multi-head attention layers, Progressive multi-scale colorization scheme.”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “Other works have also used Transformers in a variety of tasks such as tracking [61, 62], super-resolution [63], ReID [64], Colorization [65], Retrieval [66] and multi-modal learning [67, 68].”
Palette: Image-to-Image Diffusion Models (2021, methodology): “Consistent with previous inpainting works [Yi et al. 2020; Yu et al. 2018b, 2019], we use random 256×256 crops, and we combine these with the resized random largest square crops used in colorization literature [Kumar et al. 2021].”
Restormer: Efficient Transformer for High-Resolution Image Restoration (2021, methodology): “Due to these characteristics, Transformer models have also been studied for the low-level vision problems such as super-resolution [44,85], image colorization [37], denoising [13, 80], and deraining [80].”
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up (2021, methodology): “This motivates us to evaluate two efficient form of self-attention, (B) Nyström Self-Attention [64] and (C) Axis Self-Attention [67] By replacing all self-attention layers in high-resolution stages (feature map resolution higher than 32× 32) with these efficient variants, both two methods (B)(C) are”
Transformers in Medical Imaging: A Survey (2022, background): “Since their inception, ViT models have been shown to push the state-of-the-art in numerous vision tasks, including image classification [22], object detection [24], semantic segmentation [25], image colorization [26], low-level vision [27], and video understanding [28] to name a few.”
UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes (2022, methodology): “We follow C OL T RAN [28] and report FIDs using the prescribed splits of center-cropped images for metric computation and resize our model predictions to 256 × 256 .”
Finite Scalar Quantization: VQ-VAE Made Simple (2023, background): “(Kumar et al., 2021) 19.37 Table 2: UV I M results for the three tasks.”