Generative Adversarial Transformers

By Drew A. Hudson, C. Lawrence Zitnick. 2021

214 citations recorded by the source corpus.

We introduce the GANformer, a novel and efficient type of transformer, and explore it for the task of visual generative modeling. The network employs a bipartite structure that enables long-range interactions across the image, while maintaining computation of linear efficiency, that can readily scale to high-resolution synthesis. It iteratively propagates information from a set of latent variables to the evolving visual features and vice versa, to support the refinement of each in light of the other and encourage the emergence of compositional representations of objects and scenes. In contrast to the classic transformer architecture, it utilizes multiplicative integration that allows flexible region-based modulation, and can thus be seen as a generalization of the successful StyleGAN network. We demonstrate the model's strength and robustness through a careful evaluation over a range of datasets, from simulated multi-object environments to rich real-world indoor and outdoor scenes, showing it achieves state-of-the-art results in terms of image quality and diversity, while enjoying fast learning and better data-efficiency. Further qualitative and quantitative experiments offer us an insight into the model's inner workings, revealing improved interpretability and stronger disentanglement, and illustrating the benefits and efficacy of our approach. An implementation of the model is available at https://github.com/dorarad/gansformer.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Projected GANs Converge Faster (2021, methodology): “Besides StyleGAN2-ADA and FastGAN, we compare against SAGAN [69] and GANsformers [20].”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “Cross-attention itself is in widespread use as a mechanism for allowing information from a source to influence processing of a target domain of a different size or structure [18, 20, 53, 88, 64, 68, 34, 51].”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “We believe that with better architectures, such as designs from Style-GAN2 or Transformers [42], D2C can achieve even better performance.”
Aggregating Nested Transformers (2021, background): “ViT is given high expectation to improve vision tasks beyond image classiﬁcation, with existing studies on generative modeling [11, 27, 28, 39], video understanding [2, 38], segmentation and detection [30, 34, 53], interpretability [1, 6, 7].”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, background): “, natural language processing (NLP) [27, 10, 82, 88], visual recognition [31, 79, 90, 63], dense prediction [83, 11, 98, 96, 86], Generative Adversarial Network (GAN) [52, 48], reinforcement learning (RL) [18, 16, 87], robotics [50, 25], and etc.”
Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement (2023, background): “…in many computer vision tasks and achieved impressive results in high-level vision ( e.g. , image classification [2, 4, 14], semantic segmentation [7, 55, 67], object detection [3, 13, 62], etc. ) and low-level vision ( e.g. , image restoration [6, 11, 60], image synthesis [20, 21, 64], etc. ).”
Efficient Training of Visual Transformers with Small Datasets (2021, background): “For this reason, very recently, a second-generation of VTs has focused on hybrid architectures, in which convolutions are used jointly with long-range attention layers [61, 35, 58, 60, 59, 33, 28].”
Scaling up GANs for Text-to-Image Synthesis (2023, unknown): “While recent diffusion-based models [13,22,58] have commonly adopted attention mechanisms, StyleGAN architectures are predominantly convolutional with the notable exceptions such as BigGAN [5], GANformer [24], and ViTGAN [38].”
Improved Transformer for High-Resolution GANs (2021, background): “GANformer [21] leverages the Transformer as a plugin component to build the bipartite structure to allow long-range interactions during the generative process, but its main backbone is still a ConvNet based on StyleGAN [28].”
Pros and Cons of GAN Evaluation Measures: New Developments (2021, background): “…(Goodfellow et al., 2014), autoregressive models such as PixelCNNs (Oord et al., 2016), variational autoencoders (VAEs) (Kingma and Welling, 2013), and recently transformer GANs (Hudson and Zitnick, 2021; Jiang et al., 2021) have been constantly improving the state of the art in image generation.”
A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, background): “Recently, vision transformers have gained an increasing popularity in the vision community and they have been successfully applied to a broad range of vision applications, such as image classification [11, 16, 32, 43, 48, 55], object detection [3, 7, 39], image generation [20,21], and semantic segme”
MaxViT: Multi-Axis Vision Transformer (2022, background): “GANformer [36] explored efficient global attention mechanisms to improve on StyleGAN [40].”
Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding (2021, background): “Since ViT has shown to improve vision tasks beyond image classification, with prior work studying its applicability to generative modeling (Parmar et al. 2018; Child et al. 2019; Jiang, Chang, and Wang 2021; Hudson and Zitnick 2021), video understanding (Neimark et al.”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, background): “Besides, transformer also shows strong power in other vision tasks, including semantic segmentation [67], object detection [5, 68], image processing [10], and image generation [28, 27].”
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up (2021, methodology): “Another concurrent work [35] employs a bipartite self-attention on StyleGAN and thus it can propagate latent variables to the evolving visual features, yet its main structure is still convolutional, including both the”
CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation (2021, background): “Recently, several more advanced deepfake datasets (DFDC [17], CelebDF [45], CelebFOM [64], and WildDeepfake [80]) as well as GAN image datasets (CelebBlend [64], GANsformer [25], and StytleGAN2 [34]) are released.”