Interpreting the Latent Space of GANs for Semantic Face Editing

By Yujun Shen, Jinjin Gu, Xiaoou Tang, Bolei Zhou. 2019

1,258 citations recorded by the source corpus.

Despite the recent advance of Generative Adversarial Networks (GANs) in high-fidelity image synthesis, there lacks enough understanding of how GANs are able to map a latent code sampled from a random distribution to a photo-realistic image. Previous work assumes the latent space learned by GANs follows a distributed representation but observes the vector arithmetic phenomenon. In this work, we propose a novel framework, called InterFaceGAN, for semantic face editing by interpreting the latent semantics learned by GANs. In this framework, we conduct a detailed study on how different semantics are encoded in the latent space of GANs for face synthesis. We find that the latent code of well-trained generative models actually learns a disentangled representation after linear transformations. We explore the disentanglement between various semantics and manage to decouple some entangled semantics with subspace projection, leading to more precise control of facial attributes. Besides manipulating gender, age, expression, and the presence of eyeglasses, we can even vary the face pose as well as fix the artifacts accidentally generated by GAN models. The proposed method is further applied to achieve real image manipulation when combined with GAN inversion methods or some encoder-involved models. Extensive results suggest that learning to synthesize faces spontaneously brings a disentangled and controllable facial attribute representation.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

Analyzing and Improving the Image Quality of StyleGAN (2019, background): “It has been demonstrated [21, 33] that this design allows the intermediate latent space W to be much less entangled than the input latent space Z .”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “The disciminative model can operate on either the image space [66, 70, 27] or the latent space [80, 98].”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, methodology): “Another way to assess the relationship between image semantics and linear motion or separability in the latent space is by moving the latent zsem of an image in a particular direction and observing changes in the image [44].”
StyleGAN-NADA (2021, methodology): “By traversing this intermediate latent space, W , or by mixing different w codes across different network layers, prior work demonstrated fine-grained control over semantic properties in generated images [2, 13, 30, 36].”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “, [13, 40, 50]), a large amount of annotated data, or pretrained classifiers [41, 1].”
GAN Inversion: A Survey (2021, methodology): “Some studies [18], [21] analyze the separability and semantics of both W and Z spaces.”
GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, background): “Over the past couple of years, GANs [13] were the state-of-the-art, with their latent space and conditional inputs being well-studied for controllable manipulation [42, 54] and generation [25, 27, 41, 75].”
StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, methodology): “(ii) InterfaceGAN (Shen et al. 2019): We retrained InterfaceGAN on the same data as StyleFlow.”
Designing an encoder for StyleGAN image manipulation (2021, methodology): “To this end, we follow our inversion method with several existing editing techniques: StyleFlow [3], InterFaceGAN [34], GANSpace [14], and SeFa [35].”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, background): “Recently, numerous methods have shown competence in controlling StyleGAN’s latent space and performing meaningful manipulations in W [17, 35, 36, 13].”
Paint by Word (2021, background): “Early work on GANs observed the presence of some latent vector directions corresponding to semantic concepts [32], and further explorations have developed methods to identify interesting vector directions that steer latent space by using reconstruction losses [19, 37] or by learning from classificat”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, methodology): “However, current methods require either a pretrained classifier [10, 28, 29, 34], a large set of paired examples [15], or manual examination of many candidate control directions [12], which limits the versatility of these approaches.”
A Disentangling Invertible Interpretation Network for Explaining Latent Representations (2020, background): “[46] interprets the latent space of a GAN by ﬁnding attribute vectors as the normal direction of the decision boundary of an attribute classiﬁer.”
On the "steerability" of generative adversarial networks (2019, background): “Shen et al. (2019) also assumes linear latent space trajectories and learns paths for face attribute editing according to semantic concepts such as age and expression, thus demonstrating disentanglement of the latent space.”
Deep Generative Learning via Schrödinger Bridge (2021, background): “Choi et al., 2020), semantic image editing (Zhu et al., 2016; Shen et al., 2020) and audio synthesis (Van Den Oord et al., 2016; Prenger et al., 2019).”
Pivotal Tuning for Latent-based Editing of Real Images (2021, methodology): “Once the generator is tuned, we can edit the input image using any choice of latent-space editing techniques, such as those proposed by Shen et al. [2020] or Harkonen et al.”