TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up

By Shiyu Chang, Zhangyang Wang. Neural Information Processing Systems, 2021

500 citations recorded by the source corpus.

The recent explosive interest on transformers has suggested their potential to become powerful"universal"models for computer vision tasks, such as classification, detection, and segmentation. While those attempts mainly study the discriminative models, we explore transformers on some more notoriously difficult vision tasks, e.g., generative adversarial networks (GANs). Our goal is to conduct the first pilot study in building a GAN completely free of convolutions, using only pure transformer-based architectures. Our vanilla GAN architecture, dubbed TransGAN, consists of a memory-friendly transformer-based generator that progressively increases feature resolution, and correspondingly a multi-scale discriminator to capture simultaneously semantic contexts and low-level textures. On top of them, we introduce the new module of grid self-attention for alleviating the memory bottleneck further, in order to scale up TransGAN to high-resolution generation. We also develop a unique training recipe including a series of techniques that can mitigate the training instability issues of TransGAN, such as data augmentation, modified normalization, and relative position encoding. Our best architecture achieves highly competitive performance compared to current state-of-the-art GANs using convolutional backbones. Specifically, TransGAN sets new state-of-the-art inception score of 10.43 and FID of 18.28 on STL-10, outperforming StyleGAN-V2. When it comes to higher-resolution (e.g. 256 x 256) generation tasks, such as on CelebA-HQ and LSUN-Church, TransGAN continues to produce diverse visual examples with high fidelity and impressive texture details. In addition, we dive deep into the transformer-based generation models to understand how their behaviors differ from convolutional ones, by visualizing training dynamics. The code is available at https://github.com/VITA-Group/TransGAN.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6e8f35c6d54acb14109c9b792a62609eac8a7b5e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Uformer: A General U-Shaped Transformer for Image Restoration (2021, background): “Besides high-level discriminative tasks, there are also some Transformer-based works [24, 65, 83] for generative tasks.”
Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement (2023, background): “…in many computer vision tasks and achieved impressive results in high-level vision ( e.g. , image classification [2, 4, 14], semantic segmentation [7, 55, 67], object detection [3, 13, 62], etc. ) and low-level vision ( e.g. , image restoration [6, 11, 60], image synthesis [20, 21, 64], etc. ).”
Photorealistic Video Generation with Diffusion Models (2023, methodology): “Multiple classes of generative models have utilized Transformers [73] as back-bone, such as, Generative adversarial networks [42, 47, 85], autoregressive [10, 11, 20, 21, 27, 59, 74, 77, 78, 80, 81] and diffusion [2, 22, 41, 50, 53, 87] models.”
End-to-End Instance Edge Detection (2022, background): “Recently, the transformer has begun to be explored for visual recognition tasks including image classification [16], detection [6], image generation [30], etc.”
Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks (2021, methodology): “Subsequently, transformer methods have been successfully applied tomanyvisual tasks, including image classification [12], [31], [32], [33], object detection [34], lower-level vision [35], semantic segmentation [36], tracking [37], video instance segmentation [38], image generation [39],multimodal le”
Boundary-aware Backward-Compatible Representation via Adversarial Learning in Image Retrieval (2023, background): “Adversarial learning is widely used in various fields such as generative adversarial networks [4, 6, 7], person re-identity [33], and domain adaptation [2, 5].”
MaxViT: Multi-Axis Vision Transformer (2022, methodology): “TransGAN [38] built a pure Transformer GAN with a careful design of local attention and upsampling layers, demonstrating effectiveness on small scale datasets [14,45].”
StyTr2: Image Style Transfer with Transformers (2021, methodology): “However, the increment of network depth could cause the loss of feature resolution and fine details [24].”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, background): “, natural language processing (NLP) [27, 10, 82, 88], visual recognition [31, 79, 90, 63], dense prediction [83, 11, 98, 96, 86], Generative Adversarial Network (GAN) [52, 48], reinforcement learning (RL) [18, 16, 87], robotics [50, 25], and etc.”
A Survey of Visual Transformers (2021, methodology): “In addition, the visual Transformers have also obtained great performances for other CV tasks, such as detection [30], segmentation [31], [32], tracking [33], and generation [34].”
Auto-scaling Vision Transformers without Training (2022, background): “…representation learning, recently works try to train deeper ViTs (Touvron et al., 2021; Zhou et al., 2021), incorporate convolutions (Wu et al., 2021; d’Ascoli et al., 2021; Yuan et al., 2021a), and design multi-scale feature extractions (Chen et al., 2021b; Zhang et al., 2021; Wang et al., 2021).”