Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models

By Giannis Daras, Augustus Odena, A. Dimakis. 2019

62 citations recorded by the source corpus.

We introduce a new local sparse attention layer that preserves two-dimensional geometry and locality. We show that by just replacing the dense attention layer of SAGAN with our construction, we obtain very significant FID, Inception score and pure visual improvements. FID score is improved from 18.65 to 15.94 on ImageNet, keeping all other parameters the same. The sparse attention patterns that we propose for our new layer are designed using a novel information theoretic criterion that uses information flow graphs. We also present a novel way to invert Generative Adversarial Networks with attention. Our method uses the attention layer of the discriminator to create an innovative loss function. This allows us to visualize the newly introduced attention heads and show that they indeed capture interesting aspects of two-dimensional geometry of real images.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

TransGAN: Two Transformers Can Make One Strong GAN (2021, methodology): “Different from previous methods (Daras et al., 2020; Parmar et al., 2018; Child et al., 2019; Beltagy et al., 2020) during training we gradually reduce the mask until diminishing it, and eventually the self-attention is fully global 1 .”
Improved Consistency Regularization for GANs (2020, background): “For instance, Miyato et al. (2018a) propose spectral normalization to stabilize the training, which is widely used (Zhang et al., 2019; Brock et al., 2019) and has become the de-facto weight normalization technique for GANs.”
DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation (2021, background): “Especially, due to the diversity of the images in ImageNet, GAN-based inversion and its manipulation in the latent space of ImageNet show limited performance [5, 12].”
Deep Learning Techniques for Inverse Problems in Imaging (2020, background): “2Empirically, gradient descent inversion works well for medium-sized generative models like DCGAN but has not been very effective in inverting bigger generators like BigGAN [134], see [135] for a discussion.”
SMYRF: Efficient Attention using Asymmetric Clustering (2020, background): “attention to pre-defined positions [23, 3, 22, 12].”