On the "steerability" of generative adversarial networks

By Ali Jahanian, Lucy Chai, Phillip Isola. 2019

426 citations recorded by the source corpus.

An open secret in contemporary machine learning is that many models work beautifully on standard benchmarks but fail to generalize outside the lab. This has been attributed to biased training data, which provide poor coverage over real world events. Generative models are no exception, but recent advances in generative adversarial networks (GANs) suggest otherwise - these models can now synthesize strikingly realistic and diverse images. Is generative modeling of photos a solved problem? We show that although current GANs can fit standard datasets very well, they still fall short of being comprehensive models of the visual manifold. In particular, we study their ability to fit simple transformations such as camera movements and color changes. We find that the models reflect the biases of the datasets on which they are trained (e.g., centered objects), but that they also exhibit some capacity for generalization: by "steering" in latent space, we can shift the distribution while still creating realistic images. We hypothesize that the degree of distributional shift is related to the breadth of the training data distribution. Thus, we conduct experiments to quantify the limits of GAN transformations and introduce techniques to mitigate the problem. Code is released on our project page: this https URL

Subfields: Biophysics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Rewriting a Deep Generative Model (2020, background): “But why is rewriting a deep generative model useful? A generative model enforces many rules and relationships within the generated images [7,36].”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “Prior work [Jahanian et al. 2020] also investigates in-plane translation.”
Alias-Free Generative Adversarial Networks (2021, background): “While several ways of controlling the generative process have been found [8, 28, 10, 41, 24, 2, 7, 46, 6], the foundations of the synthesis process remain only partially understood.”
Interpreting the Latent Space of GANs for Semantic Face Editing (2019, background): “[18] studies the steerability of GANs concerning camera motion and image color tone.”
Designing an encoder for StyleGAN image manipulation (2021, background): “Others have proposed self-supervised approaches that find latent space directions corresponding to a specific image transformation, such as zoom or rotation [17, 29, 36].”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, background): “[18] find latent space paths that correspond to a specific image transformation, such as zoom or rotation in a self-supervised manner.”
GANSpace: Discovering Interpretable GAN Controls (2020, methodology): “In Figure 5, we compare some of our BigGAN zoom and translation edits to comparable edits found by supervised methods [9], and our StyleGAN face attribute edits to a supervised method [19].”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, methodology): “However, current methods require either a pretrained classifier [10, 28, 29, 34], a large set of paired examples [15], or manual examination of many candidate control directions [12], which limits the versatility of these approaches.”
ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement (2021, background): “, StyleGAN [24, 25, 23], effectively encode semantic information in their latent spaces [16, 36, 21].”
Closed-Form Factorization of Latent Semantics in GANs (2020, background): “Recent work has found that the native GANs, without any constraints or regularizers, are able to automatically encode various semantics in the intermediate feature space [3] and the initial latent space [7, 15, 24, 27].”
Image Augmentations for GAN Training (2020, methodology): “Jahanian et al. [18] found data augmentation improves steerability of GAN models, but they failed to generate realistic samples on CIFAR-10 when jointly optimizing the model and linear walk parameters.”
GAN Steerability without optimization (2020, methodology): “Figure 4: Quantitative comparison with (Jahanian et al., 2020).”
StyleAlign: Analysis and Applications of Aligned StyleGAN Models (2021, methodology): “Many recent works have proposed methods to interpret the semantics encoded in that space and its extensions and apply them to image editing (Jahanian et al., 2019; Shen et al., 2020a; H¨ark¨onen et al., 2020; Tewari et al., 2020; Abdal et al., 2020; Wu et al., 2020; Patashnik et al., 2021).”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, background): “Recent works have attempted to overcome these issues by first training an unconditional generator, and then converting it to a conditional model with a small cost [26, 20, 1, 42].”
Pivotal Tuning for Latent-based Editing of Real Images (2021, methodology): “Some using fullsupervision in the form of semantic labels [10, 11, 34], others [15, 30, 35] find meaningful directions in a selfsupervised fashion, and finally recent works present unsupervised methods to achieve the same goal [14, 39, 40], requiring no manual annotations.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, neutral): “There are many papers related to GANs [114], [115], [116], [117], [118], [119], [120], [121], [122], [123], [124], [125], [126],”
From Continuity to Editability: Inverting GANs with Consecutive Images (2021, background): “Generative Adversarial Networks (GANs) [5, 9, 10] has demonstrated versatile image editing capability, especially by discovering the spontaneously learned interpretable directions that can manipulate corresponding image attributes [17, 8, 19, 4, 22].”