Large Scale Adversarial Representation Learning

By Jeff Donahue, Karen Simonyan. 2019

580 citations recorded by the source corpus.

Adversarially trained generative models (GANs) have recently achieved compelling image synthesis results. But despite early successes in using GANs for unsupervised representation learning, they have since been superseded by approaches based on self-supervision. In this work we show that progress in image generation quality translates to substantially improved representation learning performance. Our approach, BigBiGAN, builds upon the state-of-the-art BigGAN model, extending it to representation learning by adding an encoder and modifying the discriminator. We extensively evaluate the representation learning and generation capabilities of these BigBiGAN models, demonstrating that these generation-based models achieve the state of the art in unsupervised representation learning on ImageNet, as well as in unconditional image generation. Pretrained BigBiGAN models -- including image generators and encoders -- are available on TensorFlow Hub (https://tfhub.dev/s?publisher=deepmind&q=bigbigan).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “Adversarial methods for representation learning are explored in [15, 16].”
On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “This pre-training is often referred to as representation learning; representations can be trained using supervised [32, 36], weakly-supervised [44], or unsupervised data [13, 14, 66, 70].”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, methodology): “We also evaluate the encoder from B IG B I GAN (Donahue & Simonyan, 2019).”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, background): “These contrastive learning based approaches learn representations in a discriminative fashion instead of a generative one as in [3, 49, 50, 41, 51].”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “Many of these approaches rely either on auto-encoding of images [24, 25, 26] or on adversarial learning [27], jointly modelling data and representation [28, 29, 30, 31].”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Early works on self-supervised learning proposed hand-crafted pretext tasks, such as context prediction [15], colorization [17, 18], solving jigsaw puzzles [47], predicting rotation [19], inpainting [16], clustering [27], and generative modeling [48].”
Contrastive Multiview Coding (2019, neutral): “On top of it, strengthening data augmentation with RandAugment [14] yields better or comparable results to other state-of-the-art methods [41, 79, 87, 27, 49, 19, 29, 4].”
Vector-quantized Image Modeling with Improved VQGAN (2021, positive): “BigBiGAN (Donahue & Simonyan, 2019a) first demonstrated that a generation-based model can match other self-supervised methods in representation learning on ImageNet. iGPT (Chen et al., 2020a) uses the autoregressive objective to learn a giant transformer that directly predicts pixel values,…”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “These models can be used in a variety of ways, such as generating images from text prompts [78, 56] or learning useful feature representations [18, 10].”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, methodology): “Several methods propose evaluating generative models from a self-supervised feature learning perspective, by repurposing the learned discriminators [59] or accompanying encoders [19] for a downstream classification task.”
GAN Inversion: A Survey (2021, methodology): “[187] propose BigBiGAN (BiGAN with BigGAN generator) for unconditional image generation by adopting the generator and discriminator architectures from the BigGAN model, which shows generative models and inference model benefit from each other.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, methodology): “, using sparse coding [58], adversarial training [12, 13, 50], autoencoders [49, 63, 76], or probabilistic versions thereof [67].”
Learning Robust and Multilingual Speech Representations (2020, neutral): “After a period of focusing on supervised techniques, unsupervised representation learning has recently seen a resurgence in a variety of modalities (Doersch and Zisserman, 2017; van den Oord et al., 2018; Donahue and Simonyan, 2019; Bachman et al., 2019) and has led to improved results, especially i”
What makes for good views for contrastive learning (2020, background): “Models based on contrastive losses have significantly outperformed other approaches [64, 30, 43, 53, 16, 41, 15, 19, 62].”
Prototypical Contrastive Learning of Unsupervised Representations (2020, neutral): “9 BigBiGAN [42] R50 (24M) – 56.”
Continual learning with hypernetworks (2019, methodology): “…adversarial networks (Goodfellow et al., 2014) have become an established method for generative modelling and tend to produce higher quality images compared to VAEs, even at the scale of datasets as complex as ImageNet (Brock et al., 2019; Lu ˇ ci ´ c et al., 2019; Donahue & Simonyan, 2019).”