Taming Transformers for High-Resolution Image Synthesis
By Patrick Esser, Robin Rombach, Björn Ommer. 2020
4,465 citations recorded by the source corpus.
Designed to learn long-range interactions on sequential data, transformers continue to show state-of-the-art results on a wide variety of tasks. In contrast to CNNs, they contain no inductive bias that prioritizes local interactions. This makes them expressive, but also computationally infeasible for long sequences, such as high-resolution images. We demonstrate how combining the effectiveness of the inductive bias of CNNs with the expressivity of transformers enables them to model and thereby synthesize high-resolution images. We show how to (i) use CNNs to learn a context-rich vocabulary of image constituents, and in turn (ii) utilize transformers to efficiently model their composition within high-resolution images. Our approach is readily applied to conditional synthesis tasks, where both non-spatial information, such as object classes, and spatial information, such as segmentations, can control the generated image. In particular, we present the first results on semantically-guided synthesis of megapixel images with transformers. Project page at https://git.io/JLlvY.
Subfields: Computer Vision and Pattern Recognition, Generative Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Generative Models
- Jun-Yan Zhu
- Richard Zhang
- Eli Shechtman
- Arash Vahdat
- Stefano Ermon
- Alexei A. Efros
- Daniel Cohen-Or
- Karsten Kreis
Citations in the corpus
- High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “Our perceptual compression model is based on previous work [23] and consists of an autoencoder trained by combination of a perceptual loss [102] and a patch-based [32] adversarial objective [20, 23, 99].”
- Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “[42] use diffusion models on the latents obtained from a VQGAN [14] like autoencoder.”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Other likelihood-based models include autoregressive models [58, 43, 13, 8] and diffusion models [11, 21, 20].”
- LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs (2021, methodology): “The VQGAN [12] pretrained on ImageNet is used to encode image tokens.”
- Projected GANs Converge Faster (2021, background): “While combining pretrained perceptual networks [58] with GANs for image-to-image translation has led to impressive results [14, 49, 59, 64], this idea has not yet materialized for unconditional noise-to-image synthesis.”
- Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “As a result, VQGAN (Esser et al., 2021) relies on topk and top-p (nucleus) sampling heuristics (Holtzman et al., 2020) with a default codebook size of 1024 to obtain best results for image synthesis.”
- Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “This also aligns with a recent trend on using Transformers in computer vision [6, 8] and treating Fourier transform as a lightweight replacement to the selfattention [24, 35].”
- CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders (2021, background): “• VQGAN Optimization, in which images are created through sampling a VQGAN codebook (Esser et al., 2021).”
- Score-based Generative Modeling in Latent Space (2021, methodology): “In two-stage training, a separate generative model is trained in latent space as a new prior after training the VAE itself [60, 61, 62, 63, 64, 10].”
- CogView: Mastering Text-to-Image Generation via Transformers (2021, methodology): “In these scenarios, we can (1) train a VQGAN [15] instead of VQVAE for the latent variable for more realistic textures, (2) decrease the number of parameters and increase the length of sequences for a higher resolution.”
- High-Resolution Complex Scene Synthesis with Transformers (2021, methodology): “…generic mechanism for conditional image synthesis via training an autoregressive (AR) transformer model in the latent space of a pre-trained discrete autoencoder model [31, 12], where [7] achieve efficient high-resolution synthesis by extending the reconstruction task with an adversarial objec-tive.”
- Diffusion Models Beat GANs on Image Synthesis (2021, neutral): “57 VQ-VAE-2 (RS)†‡§ [57] ⇠ 10 VQ-GAN‡ [21] 15.”
- CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, methodology): “First Stage We first train a VQ-GAN model with the image dataset in a self-supervised manner.”
- CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “(2021), extended to include images through VQVAE-GAN tokens (Esser et al., 2021) and additional hypertext link structure.”
- Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, methodology): “More advanced autoencoders combine perceptual loss and adversarial loss, e.g., VQGAN [7], or rely on a hierarchy of latent variables, e.g., NVAE [40] and VQ-VAE2 [29].”
- Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, methodology): “VQ-VAE [41, 49], VQGAN [16] and ImageBART [15] train an encoder to compress the image into a low-dimensional discrete latent space and fit the density of the hidden variables.”
- Scalable Diffusion Models with Transformers (2022, background): “They have also been trained on discrete codebooks [56] as both autoregressive models [11,44] and masked generative models [4, 14]; the former has shown excellent scaling behavior up to 20B parameters [59].”
- MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, background): “…on a wide variety of vision tasks as well as impressive generalization abilities that can provide powerful semantic guidance to image generation [14], but since they were only trained to match image-text pairs, cannot inherently be used for tasks that require text generation as an output [44].”