Controlling generative models with continuous factors of variations

By Antoine Plumerault, Hervé Le Borgne, Céline Hudelot. 2020

133 citations recorded by the source corpus.

Recent deep generative models are able to provide photo-realistic images as well as visual or textual content embeddings useful to address various tasks of computer vision and natural language processing. Their usefulness is nevertheless often limited by the lack of control over the generative process or the poor understanding of the learned representation. To overcome these major issues, very recent work has shown the interest of studying the semantics of the latent space of generative models. In this paper, we propose to advance on the interpretability of the latent space of generative models by introducing a new method to find meaningful directions in the latent space of any generative model along which we can move to control precisely specific properties of the generated image like the position or scale of the object in the image. Our method does not require human annotations and is particularly well suited for the search of directions encoding simple transformations of the generated image, such as translation, zoom or color variations. We demonstrate the effectiveness of our method qualitatively and quantitatively, both for GANs and variational auto-encoders.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Designing an encoder for StyleGAN image manipulation (2021, background): “Others have proposed self-supervised approaches that find latent space directions corresponding to a specific image transformation, such as zoom or rotation [17, 29, 36].”
GANSpace: Discovering Interpretable GAN Controls (2020, background): “Current attempts to add user control over the output focus on supervised learning of latent directions [9, 7, 25, 19, 16], GAN training with labeled images [12, 22, 21].”
Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation (2021, unknown): “, 2017a) and discovering semantically meaningful directions in a GAN latent space (Härkönen et al., 2020; Jahanian et al., 2019; Plumerault et al., 2020; Shen et al., 2019; Voynov & Babenko, 2020).”
Low-Rank Subspaces in GANs (2021, background): “To identify these semantically meaningful subspaces, existing approaches fall into two folds, i.e. , supervised [13, 30, 18, 24] and unsupervised [33, 15, 28, 31].”
Disentangled Face Attribute Editing via Instance-Aware Latent Space Search (2021, background): “Another set of works [Shen et al. , 2020; Voynov and Babenko, 2020; Shen and Zhou, 2020; Plumerault et al. , 2020; H¨ark¨onen et al. , 2020; Goetschalckx et al. , 2019] propose to edit face attribute by moving the latent code along a speciﬁc semantic direction in the latent space of well-trained…”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, methodology): “Note that there are other methods for explaining the generative models [90, 123, 132].”
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization (2021, methodology): “Deep learning is now powering the majority of computer vision applications ranging from autonomous driving [93, 73] and medical imaging [78, 38] to image editing [69, 15, 98, 88, 70, 74].”
Pivotal Tuning for Latent-based Editing of Real Images (2021, methodology): “2020], others [Jahanian et al. 2019; Plumerault et al. 2020; Spingarn-Eliezer et al. 2020] find meaningful directions in a self-supervised fashion, and, finally, recent works present unsupervised methods to achieve the same goal [Härkönen et al.”
StyleMC: Multi-Channel Based Fast Text-Guided Image Generation and Manipulation (2021, background): “Most of this work discovers domain-independent and interpretable directions such as zoom-in, rotation, and translation [21, 10, 30], while other frameworks propose to find a set of domain-specific directions such as hair color or gender on face images [25] or cognitive features [8].”
EditGAN: High-Precision Semantic Image Editing (2021, methodology): “Previous papers studied encoder-based methods [58–62], used primarily optimization-based techniques [26, 63–69], and developed hybrid approaches [24, 25, 63, 70, 71].”
Counterfactual Generative Networks (2021, background): “A recent research direction explores the discovery of interpretable directions in GANs trained on ImageNet (Plumerault et al., 2020; Voynov & Babenko, 2020; Peebles et al., 2020).”
Fixing Data Augmentation to Improve Adversarial Robustness (2021, background): “Similarly, generative models can be used to create novel views of images (Plumerault et al., 2020; Jahanian et al., 2019; Härkönen et al., 2020) by manipulating them in latent space.”
Unsupervised Discovery of Interpretable Directions in the GAN Latent Space (2020, methodology): “Some of them induce domain-agnostic transformations, like zooming or translation (Ja-hanian et al., 2020; Plumerault et al., 2020), while others correspond to domain-speciﬁc transformations, e.g., adding smile or glasses on face images (Radford et al., 2015).”
High-Fidelity GAN Inversion for Image Attribute Editing (2021, background): “Others [16,24] explored simple geometric transformation via self-supervised learning.”
GAN Steerability without optimization (2020, methodology): “To detect such latent-space directions, the methods of Jahanian et al. (2020) and Plumerault et al. (2020) require a training procedure that limits them to transformations for which synthetic images can be produced for supervision (e.g., shift or zoom).”
Do Not Escape From the Manifold: Discovering the Local Coordinates on the Latent Space of GANs (2021, background): “However, despite several recent efforts (Goetschalckx et al., 2019; Jahanian et al., 2019; Plumerault et al., 2020; Shen et al., 2020) to investigate the disentanglement properties (Bengio et al., 2013) of the latent space in GANs, it is still challenging to ﬁnd meaningful traversal directions in…”