Closed-Form Factorization of Latent Semantics in GANs

By Yujun Shen, Bolei Zhou. 2020

655 citations recorded by the source corpus.

A rich set of interpretable dimensions has been shown to emerge in the latent space of the Generative Adversarial Networks (GANs) trained for synthesizing images. In order to identify such latent dimensions for image editing, previous methods typically annotate a collection of synthesized samples and train linear classifiers in the latent space. However, they require a clear definition of the target attribute as well as the corresponding manual annotations, limiting their applications in practice. In this work, we examine the internal representation learned by GANs to reveal the underlying variation factors in an unsupervised manner. In particular, we take a closer look into the generation mechanism of GANs and further propose a closedform factorization algorithm for latent semantic discovery by directly decomposing the pre-trained weights. With a lightning-fast implementation, our approach is capable of not only finding semantically meaningful dimensions comparably to the state-of-the-art supervised methods, but also resulting in far more versatile concepts across multiple GAN models trained on a wide range of datasets.1

Subfields: Biophysics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generative Adversarial Networks (2021, neutral): “Style transfer using GANs has shown promising results in various domains, s.a. fashion design, art, and photography [19].”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “Such methods can be categorized into: (i) methods that use image annotations to find meaningful latent paths [40, 1], and (ii) methods that find meaningful directions without supervision, and require manual annotation for each direction [13, 42, 48, 49].”
Alias-Free Generative Adversarial Networks (2021, methodology): “The geometric head motion was generated as a random latent space walk along hand-picked directions from GANSpace [25] and SeFa [52].”
Designing an encoder for StyleGAN image manipulation (2021, methodology): “For cars, we use directions obtained by GANSpace [14]; for faces we use StyleFlow [3]; and for horses, cats, and churches we use SeFa [35].”
GANSpace: Discovering Interpretable GAN Controls (2020, neutral): “Several methods developed concurrently to our own explore similar or related ideas [23, 15, 24, 20, 1], and comparing or combining approaches may prove useful as well.”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, background): “[30] do eigenvector decomposition in the affine transformation”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, background): “To overcome this limitation, some recent works [15], [16], [27] focus on the manipulation of the underlying learned latent space, using the vector”
Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs (2020, methodology): “In contrast, SeFa (Shen & Zhou, 2020) and GANSpace (H¨ark¨onen et al., 2020) discover meaningful latent directions without supervision, where some of them is coupled with content pose.”
Positional Encoding as Spatial Inductive Bias in GANs (2020, methodology): “As for image manipulation [1], we customize a convenient pipeline by improving the closed-form factorization [43].”
E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion (2021, neutral): “To control the GAN synthesis process, some works explore the semantic editing of latent codes by finding semantic directions on their latent space in a supervised [3] or unsupervised [39], [40] manner.”
Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation (2021, methodology): “major challenge in artificial intelligence, and can be beneficial to many computer vision tasks, such as domain adaptation [33, 45], controllable image generation [32, 38, 41, 48], and image manipulation [37].”
BlobGAN: Spatially Disentangled Scene Representations (2022, background): “Directions of variation naturally emerge in the latent space and can be discovered when guided by geometry/color changes [31], language or attributes [59, 63, 75, 2, 88], cognitive signals [17], or in an unsupervised manner [21, 76, 61].”
Controllable and Compositional Generation with Latent-Space Energy-Based Models (2021, background): “Some works explore linear manipulations of latent code [26, 20, 46], while others consider more complicated nonlinear manipulations [37, 16, 1, 42].”
Only a matter of style: age transformation using a style-based regression model (2021, background): “…either by directly learning a disentangled mapping of an attribute of interest [Nitzan et al. 2020] or by a latent space traversal [Abdal et al. 2020b; Denton et al. 2019; Goetschalckx et al. 2019; Härkönen et al. 2020; Shen et al. 2020; Shen and Zhou 2020; Voynov and Babenko 2020; Wu et al. 2020].”