Unsupervised Discovery of Interpretable Directions in the GAN Latent Space

By Andrey Voynov, Artem Babenko. 2020

463 citations recorded by the source corpus.

The latent spaces of GAN models often have semantically meaningful directions. Moving in these directions corresponds to human-interpretable image transformations, such as zooming or recoloring, enabling a more controllable generation process. However, the discovery of such directions is currently performed in a supervised manner, requiring human labels, pretrained models, or some form of self-supervision. These requirements severely restrict a range of directions existing approaches can discover. In this paper, we introduce an unsupervised method to identify interpretable directions in the latent space of a pretrained GAN model. By a simple model-agnostic procedure, we find directions corresponding to sensible semantic manipulations without any form of (self-)supervision. Furthermore, we reveal several non-trivial findings, which would be difficult to obtain by existing methods, e.g., a direction corresponding to background removal. As an immediate practical benefit of our work, we show how to exploit this finding to achieve competitive performance for weakly-supervised saliency detection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “Such methods can be categorized into: (i) methods that use image annotations to find meaningful latent paths [40, 1], and (ii) methods that find meaningful directions without supervision, and require manual annotation for each direction [13, 42, 48, 49].”
GAN Inversion: A Survey (2021, methodology): “3 7 7 7 7 [15], [16], [31] [16], [35], [36] GANLatentDiscovery [55] 2020, ICML O.”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “In the supplementary, we show additional interpolations and qualitative comparisons to BigGAN and ADM.”
Designing an encoder for StyleGAN image manipulation (2021, methodology): “Finally, several methods [14, 38, 39] find latent directions in an unsupervised manner and require manual annotations to determine the semantic meaning of each direction post hoc.”
GANSpace: Discovering Interpretable GAN Controls (2020, neutral): “Several methods developed concurrently to our own explore similar or related ideas [23, 15, 24, 20, 1], and comparing or combining approaches may prove useful as well.”
ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement (2021, background): “On the other end of the supervision spectrum, several methods [16, 41, 42] find directions in a completely unsupervised manner.”
Closed-Form Factorization of Latent Semantics in GANs (2020, methodology): “which is commonly used in the existing approaches [7, 24, 27, 26, 10].”
Ensembling with Deep Generative Views (2021, background): “One set of approaches aims to uncover primary directions of variation in an intermediate latent space [21, 53, 12], while another enforces distinctness of optimized directions during training [58].”
Disentangled Face Attribute Editing via Instance-Aware Latent Space Search (2021, methodology): “This motivates several works [Shen et al. , 2020; Voynov and Babenko, 2020; Shen and Zhou, 2020; H¨ark¨onen et al. , 2020] to edit face attribute by reusing the knowledge learned by GAN models.”
Counterfactual Generative Networks (2021, background): “A recent research direction explores the discovery of interpretable directions in GANs trained on ImageNet (Plumerault et al., 2020; Voynov & Babenko, 2020; Peebles et al., 2020).”
Label-Efficient Semantic Segmentation with Diffusion Models (2021, background): “The ﬁrst line of works (Voynov & Babenko, 2020; Voynov et al., 2021; Melas-Kyriazi et al., 2021) is based on the evidence that the latent spaces of the state-of-the-art GANs have directions corresponding to effects that inﬂuence the foreground/background pixels differently, which allows producing…”
From Continuity to Editability: Inverting GANs with Consecutive Images (2021, background): “[22] discover the semantic directions hidden in the latent space by an unsupervised modelagnostic procedure.”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, methodology): “This method is getting popular recently and both unsupervised methods [45,17,38] and supervised methods [19,36,54] are heavily explored.”
When, Why, and Which Pretrained GANs Are Useful? (2022, background): “…et al., 2019) makes them a key ingredient in applications, including semantic editing (Isola et al., 2017; Zhu et al., 2018; Shen et al., 2020; Voynov & Babenko, 2020), image processing (Pan et al., 2020; Ledig et al., 2017; Menon et al., 2020), video generation (Wang et al., 2018a),…”
Pivotal Tuning for Latent-based Editing of Real Images (2021, methodology): “Some using fullsupervision in the form of semantic labels [10, 11, 34], others [15, 30, 35] find meaningful directions in a selfsupervised fashion, and finally recent works present unsupervised methods to achieve the same goal [14, 39, 40], requiring no manual annotations.”
Generative Adversarial Networks for Face Generation: A Survey (2022, background): “[29, 53, 110, 120, 131] go for generating mixed styles.”