InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs

By Yujun Shen, Ceyuan Yang, Xiaoou Tang, Bolei Zhou. 2020

709 citations recorded by the source corpus.

Although generative adversarial networks (GANs) have made significant progress in face synthesis, there lacks enough understanding of what GANs have learned in the latent representation to map a random code to a photo-realistic image. In this work, we propose a framework called InterFaceGAN to interpret the disentangled face representation learned by the state-of-the-art GAN models and study the properties of the facial semantics encoded in the latent space. We first find that GANs learn various semantics in some linear subspaces of the latent space. After identifying these subspaces, we can realistically manipulate the corresponding facial attributes without retraining the model. We then conduct a detailed study on the correlation between different semantics and manage to better disentangle them via subspace projection, resulting in more precise control of the attribute manipulation. Besides manipulating the gender, age, expression, and presence of eyeglasses, we can even alter the face pose and fix the artifacts accidentally made by GANs. Furthermore, we perform an in-depth face identity analysis and a layer-wise analysis to evaluate the editing results quantitatively. Finally, we apply our approach to real face editing by employing GAN inversion approaches and explicitly training feed-forward models based on the synthetic data established by InterFaceGAN. Extensive experimental results suggest that learning to synthesize faces spontaneously brings a disentangled and controllable face representation.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generative Adversarial Networks (2021, neutral): “Style transfer using GANs has shown promising results in various domains, s.a. fashion design, art, and photography [19].”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “It may be seen that in GANSpace [4] manipulation is entangled with skin color and lighting, while in InterFaceGAN [8] the identity may change significantly (when manipulating Lipstick).”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, methodology): “In this section we compare the ability of our approach to achieve disentangled manipulation of visual attributes to that of two state-of-the-art methods, specifically GANSpace [12] and InterFaceGAN [29].”
SofGAN: A Portrait Image Generator with Dynamic Styling (2020, methodology): “The seminal attribute-based approaches [Abdal et al. 2020; Chen et al. 2016; Shen et al. 2020; Tewari et al. 2020] explore to find attribute-specific paths in the latent spaces, e.g., to use direction codes to drag random style code towards each attribute direction.”
Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation (2021, background): “In the recent few years, unsupervised disentanglement learning has attracted intensive attention, owing to its importance in understanding generative models [32, 38] and extensive applications in various vision tasks [37, 45].”
Labels4Free: Unsupervised Segmentation using StyleGAN (2021, background): “Some recent works in this domain [5, 6, 16, 18, 49, 3, 36, 2, 37] study the structure of the activation and latent space.”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, methodology): “Similarly, following the supervised method [37], we train a linear regressor to group all latent codes.”
StyleMC: Multi-Channel Based Fast Text-Guided Image Generation and Manipulation (2021, background): “Supervised approaches typically benefit from pre-trained attribute classifiers that guide the optimization process to discover meaningful directions in the latent space, or use labeled data to train new classifiers that directly aim to learn directions of interest [8, 25].”
One-Shot Generative Domain Adaptation (2021, methodology): “It is also worth noting that when the domain gap increases ( i.e. , from realistic facial images to sketches), the performances of Cross-domain [10] and FreezeD [8] become almost identical (45.67 v.s 46.54) while substantial improvements of synthesis could remain observed by our method (31.97).”
Diffusion Models already have a Semantic Latent Space (2022, background): “…generative adversarial network Goodfellow et al. (2020) address their latent space for image editing (Ling et al. (2021), H¨ark¨onen et al. (2020), Chefer et al. (2021), Shen et al. (2020), Y¨uksel et al. (2021), Patashnik et al. (2021), Gal et al. (2021), Dai et al. (2019), Xu et al. (2022)).”
Explaining in Style: Training a GAN to explain a classifier in StyleSpace (2021, methodology): “This approach has been used to create powerful interfaces for image editing and data visualization [3, 26, 7, 28, 27, 37].”
Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks (2021, methodology): “In the StyleGAN domain, recent works (H ¨ ark ¨ onen et al., 2020; Shen et al., 2020; Tewari et al., 2020a; Abdal et al., 2021b) extract meaningful linear and non-linear paths in the latent space.”
BlobGAN: Spatially Disentangled Scene Representations (2022, background): “Directions of variation naturally emerge in the latent space and can be discovered when guided by geometry/color changes [31], language or attributes [59, 63, 75, 2, 88], cognitive signals [17], or in an unsupervised manner [21, 76, 61].”
DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation (2021, methodology): “Comparision with other GAN inversion-based manipulation: StyleSpace [61] and InterfaceGAN [50].”