Invertible Conditional GANs for image editing

By Guim Perarnau, Joost van de Weijer, B. Raducanu, José Manuel Álvarez. 2016

672 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) have recently demonstrated to successfully approximate complex data distributions. A relevant extension of this model is conditional GANs (cGANs), where the introduction of external information allows to determine specific representations of the generated images. In this work, we evaluate encoders to inverse the mapping of a cGAN, i.e., mapping a real image into a latent space and a conditional representation. This allows, for example, to reconstruct and modify real images of faces conditioning on arbitrary attributes. Additionally, we evaluate the design of cGANs. The combination of an encoder with a cGAN, which we call Invertible cGAN (IcGAN), enables to re-generate real images with deterministic complex modifications.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation (2017, methodology): “IcGAN combines an encoder with a cGAN [22] model. cGAN learns the mapping G : {z, c} → x that generates an image x conditioned on both the latent vector z and the conditional vector c.”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “Standard approaches for inverting G use either latent optimization [Abdal et al. 2019; Creswell and Bharath 2019; Karras et al. 2020b] or an encoder [Alaluf et al. 2021; Perarnau et al. 2016; Tov et al. 2021].”
cGANs with Projection Discriminator (2018, background): “To the authors’ knowledge, most frameworks of discriminators in cGANs at the time of writing feeds the pair the conditional information y into the discriminator by naively concatenating (embedded) y to the input or to the feature vector at some middle layer (Mirza & Osindero, 2014; Denton et al., 20”
Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, methodology): “The number of convolution layers needed were computed automatically using the techniques from [66].”
Generative Feature Replay For Class-Incremental Learning (2020, background): “Generative adversarial networks (GANs) [11] are able to generate realistic and sharp images conditioned on object categories [12, 39], text [42, 57], another image (image translation) [18, 58] and style transfer [10].”
DualGAN: Unsupervised Dual Learning for Image-to-Image Translation (2017, background): “Soon after, various conditional GANs (cGAN) have been proposed to condition the image generation on class labels [13], attributes [14, 24], texts [15], and images [7, 8, 21, 12, 4, 18].”
Fader Networks: Manipulating Images by Sliding Attributes (2017, methodology): “This task and the related problem of unsupervised domain transfer recently received a lot of interest [18, 25, 10, 27, 22, 24], as a case study for conditional generative models but also for applications like automatic image edition.”
Interpreting the Latent Space of GANs for Semantic Face Editing (2019, methodology): “To make GANs applicable for real image processing, existing methods proposed to reverse the mapping from the latent space to the image space [27, 39, 24, 5, 16] or learn an additional encoder associated with the GAN training [13, 12, 38].”
Designing an encoder for StyleGAN image manipulation (2021, methodology): “Generally, inversion methods either (i) directly optimize the latent vector to minimize the error for the given image [1, 2, 8, 21, 23], (ii) train an encoder to map the given image to the latent space [13, 28, 31], or (iii) use a hybrid approach combining both [4, 45].”
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation (2020, methodology): “Generally, inversion methods either directly optimize the latent vector to minimize the error for the given image [25, 8, 1, 2], train an encoder to map the given image to the latent space [31, 8, 32, 12, 29], or use a hybrid approach combining both [4, 42].”
Semantic Image Synthesis via Adversarial Learning (2017, background): “[25], where two encoders were trained to encode images back to latent variables and conditional representations, respectively.”
Editing in Style: Uncovering the Local Semantics of GANs (2020, background): “Latent code-based techniques learn a manifold for natural images in the latent code space facilitated by a GAN and perform semantic edits by traversing paths along this manifold [23, 32].”
SinGAN: Learning a Generative Model From a Single Natural Image (2019, methodology): “Generative models for image manipulation The power of adversarial learning has been demonstrated by recent GAN-based methods, in many different image manipulation tasks [61, 10, 62, 8, 53, 56, 42, 53].”
Transferring GANs: generating images from limited data (2018, background): “Examples of conditions are class labels [32, 35, 14], text [37, 46], another image (image translation [23, 50] and style transfer [11]).”
An Introduction to Image Synthesis with Generative Adversarial Nets (2018, methodology): “Among them, DCGAN is one of the most classic ones whose structure is used by many later models such as [11] [30] [43] [19].”
RelGAN: Multi-Domain Image-to-Image Translation via Relative Attributes (2019, methodology): “Recent methods for facial attribute transfer [22, 15, 4, 7, 30, 23] formulate the problem as unpaired multi-domain image-to-image translation.”
Generative OpenMax for Multi-Class Open Set Classification (2017, background): “GAN has a wide range of applications including, sketch/text to image synthesis [4], image to image translation [10], image editing [19], super resolution [15] and video prediction [6], etc.”
InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs (2020, background): “To solve this problem, a common practice is to get the reverse mapping from the image space to the latent space, which is also known as GAN Inversion [8], [42], [43], [44], [45].”