Self-Attention Generative Adversarial Networks

By Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, Augustus Odena. 2018

4,155 citations recorded by the source corpus.

In this paper, we propose the Self-Attention Generative Adversarial Network (SAGAN) which allows attention-driven, long-range dependency modeling for image generation tasks. Traditional convolutional GANs generate high-resolution details as a function of only spatially local points in lower-resolution feature maps. In SAGAN, details can be generated using cues from all feature locations. Moreover, the discriminator can check that highly detailed features in distant portions of the image are consistent with each other. Furthermore, recent work has shown that generator conditioning affects GAN performance. Leveraging this insight, we apply spectral normalization to the GAN generator and find that this improves training dynamics. The proposed SAGAN achieves the state-of-the-art results, boosting the best published Inception score from 36.8 to 52.52 and reducing Frechet Inception distance from 27.62 to 18.65 on the challenging ImageNet dataset. Visualization of the attention layers shows that the generator leverages neighborhoods that correspond to object shapes rather than local regions of fixed shape.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

In Generative Models

Citations in the corpus

Exploring Self-Attention for Image Recognition (2020, background): “atural language processing more broadly [33, 37, 7, 6, 40]. This has inspired applications of self-attention and related ideas to image recognition [5, 34, 15, 14, 45, 46, 13, 1, 27], image synthesis [43, 26, 2], image captioning [39,41,4], and video prediction [17,35]. Until very recently, applicat”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “Self-Attention: Self-attention mechanism is usually applied to vision models as an add-on to augment CNNs outputs [82,89,38].”
Attention Augmented Convolutional Networks (2019, methodology): “Additionally, multiple attention mechanisms have been proposed for visual tasks to address the weaknesses of convolutions [17, 16, 7, 46, 45, 53].”
Jukebox: A Generative Model for Music (2020, methodology): “The rate of progress in this ﬁeld has been rapid, where only a few years ago we had algorithms producing blurry faces (Kingma & Welling, 2014; Goodfellow et al., 2014) but now we now can generate high-resolution faces indistinguishable from real ones (Zhang et al., 2019b).”
Natural Adversarial Examples (2019, methodology): “ networks with self-attention [25] are designed to better capture long-range dependencies and interactions across an image. Self-attention helps GANs learn how to generate images with plausible shape [46], and in classiﬁcation, self-attention is utilized in state-of-the-art ImageNet-1K models. We co”
Training Generative Adversarial Networks with Limited Data (2020, background): “[31] propose to regularize the discriminator by explicitly enforcing an upper bound for its Lipschitz constant, and several follow-up works [49, 5, 53, 38] have found it to be beneficial.”
CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, background): “BigGAN has a monolithic, classconditional structure, is trained on very large batch sizes, and uses self-attention layers [42, 39].”
Analyzing and Improving the Image Quality of StyleGAN (2019, background): “It should be noted that spectral normalization [26] of the generator [40] only constrains the largest singular value, posing no constraints on the others and hence not necessarily leading to better conditioning.”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, methodology): “, using multiple discriminators [17, 43, 10], multiresolution discrimination [55, 51], or self-attention [57].”
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, methodology): “Our basic setup follows SA-GAN (Zhang et al., 2018), and is implemented in TensorFlow (Abadi et al.”
Generative Adversarial Networks (2021, neutral): “GANs are typically trained using a 2-player minimax game, where a Generator Network learns to produce Synthetic Data samples, and a Discriminator Network learns to differentiate between Real Data and Synthetic Data samples.”
Projected GANs Converge Faster (2021, methodology): “Besides StyleGAN2-ADA and FastGAN, we compare against SAGAN [69] and GANsformers [20].”
Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “…has received much attention with the progress of deep generative models, including Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Zhang et al., 2019b), Variational Autoencoders (VAEs) (Kingma & Welling, 2014; Vahdat & Kautz, 2020), Diffusion Models (Song & Ermon, 2019; Dhariwal…”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “…made possible thanks to fundamental advances in deep learning architectures (He et al., 2015; van den Oord et al., 2016b;c; Vaswani et al., 2017; Zhang et al., 2019; Menick & Kalch-brenner, 2018) as well as the availability of compute resources (Jouppi et al., 2017; Amodei & Hernandez, 2018)…”
Object-Centric Image Generation from Layouts (2020, background): “Apart from unsupervised image generation, GAN-based image generation approaches have done well at conditional image generation from labels (Radford, Metz, and Chintala 2016; Zhang et al. 2019; Brock, Donahue, and Simonyan 2019), captions (Reed et al.”
A U-Net Based Discriminator for Generative Adversarial Networks (2020, methodology): “In SAGAN [50] the self-attention block was added to improve the network ability to model global structure.”
Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, methodology): “Following prior works [48], [49], to address this problem and further guide the regionwise convolutions to infer meaningful semantic contents from the existing regions, a nonlocal”
Improved Precision and Recall Metric for Assessing Generative Models (2019, background): “Due to the large amount of variation within and between classes, generative modeling of ImageNet has proven to be a challenging problem [21, 30, 4].”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “Conditional normalization layers include the Conditional Batch Normalization (Conditional BN) [12] and Adaptive Instance Normalization (AdaIN) [19], Both were first used in the style transfer task and later adopted in various vision tasks [5,9,11,20,25,34,36,39,46,51].”