Inverting the Generator of a Generative Adversarial Network

By Antonia Creswell, Anil Anthony Bharath. 2016

369 citations recorded by the source corpus.

Generative adversarial networks (GANs) learn a deep generative model that is able to synthesize novel, high-dimensional data samples. New data samples are synthesized by passing latent samples, drawn from a chosen prior distribution, through the generative model. Once trained, the latent space exhibits interesting properties that may be useful for downstream tasks such as classification or retrieval. Unfortunately, GANs do not offer an “inverse model,” a mapping from data space back to latent space, making it difficult to infer a latent representation for a given data sample. In this paper, we introduce a technique, inversion, to project data samples, specifically images, to the latent space using a pretrained GAN. Using our proposed inversion technique, we are able to identify which attributes of a data set a trained GAN is able to model and quantify GAN performance, based on a reconstruction loss. We demonstrate how our proposed inversion technique may be used to quantitatively compare the performance of various GAN models trained on three image data sets. We provide codes for all of our experiments in the website (https://github.com/ToniCreswell/InvertingGAN).

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, methodology): “We note that the optimization-based methods [21], [22], [43] typically require an expensive iterative process in terms of both memory and runtime, as they have to be applied to each latent code independently.”
StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, methodology): “The efficacy of random features for evaluating generative models has been demonstrated in [Naeem et al. 2020].”
Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space? (2019, methodology): “the Variational Auto-Encoder [15]); ii) select a random initial latent code and optimize it using gradient descent [35, 4].”
Designing an encoder for StyleGAN image manipulation (2021, methodology): “Generally, inversion methods either (i) directly optimize the latent vector to minimize the error for the given image [1, 2, 8, 21, 23], (ii) train an encoder to map the given image to the latent space [13, 28, 31], or (iii) use a hybrid approach combining both [4, 45].”
Seeing What a GAN Cannot Generate (2019, background): “Later work found that DCGAN left-inverses can be computed to high precision [21, 39], and that inversions of a GAN for glyphs can reveal specific strokes that the generator is unable to generate [7].”
ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement (2021, background): “Previous works have explored learning-based inversion approaches and train encoders to map a given real image into its corresponding latent code [10, 32, 50, 15, 35, 40].”
Image2StyleGAN++: How to Edit the Embedded Images? (2019, background): “Although it has an inherent problem to generalize beyond the training dataset, it produces higher quality results than the naive latent code optimization methods [41, 7].”
InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs (2020, background): “To solve this problem, a common practice is to get the reverse mapping from the image space to the latent space, which is also known as GAN Inversion [8], [42], [43], [44], [45].”
Image Processing Using Multi-Code GAN Prior (2019, methodology): “One is to directly optimize the latent code by minimizing the reconstruction error through back-propagation [30, 12, 32].”
Source Generator Attribution via Inversion (2019, methodology): “Creswell and Bharath [5] develop a generator inversion method as a means to explore GAN performance as a function of attributes in the image space.”
HyperInverter: Improving StyleGAN Inversion via Hypernetwork (2021, methodology): “There are two general approaches to determining a latent code of a GAN model given an input image: by an iterative optimization [1, 10, 21, 25, 32] and by inference with an encoder [35, 37, 44, 58].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, neutral): “k and a generation network utilizing an adversarial process. Arora et al. [310] show the theoretical limitations of Encoder-Decoder GAN architectures such as BiGANs [40] and ALI [41]. Creswell et al. [311] invert the generator of GANs. 4.3.4 Mathematical perspective such as optimization Mohamed et a”
From Continuity to Editability: Inverting GANs with Consecutive Images (2021, methodology): “To realize real image editing, GAN inversion methods are proposed to inference a latent code of an input image based on the pre-trained GAN [16, 25, 1, 3, 15].”