Optimizing the Latent Space of Generative Networks

By Piotr Bojanowski, Armand Joulin, David Lopez-Paz, Arthur Szlam. 2017

462 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) have achieved remarkable results in the task of generating realistic natural images. In most successful applications, GAN models share two common aspects: solving a challenging saddle point optimization problem, interpreted as an adversarial game between a generator and a discriminator functions; and parameterizing the generator and the discriminator as deep convolutional neural networks. The goal of this paper is to disentangle the contribution of these two factors to the success of GANs. In particular, we introduce Generative Latent Optimization (GLO), a framework to train deep convolutional generators using simple reconstruction losses. Throughout a variety of experiments, we show that GLO enjoys many of the desirable properties of GANs: synthesizing visually-appealing samples, interpolating meaningfully between samples, and performing linear arithmetic with noise vectors; all of this without the adversarial optimization scheme.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

GAN Inversion: A Survey (2021, neutral): “Unlike other deep generative models (like Variational Autoencoders (VAEs) [179] or Generative Latent Optimization (GLO) [180]), the objective function”
Deep Clustering for Unsupervised Learning of Visual Features (2018, background): “Typically, a parametrized mapping is learned between a predefined random noise and the images, with either an autoencoder [4, 22, 29, 40, 62], a generative adversarial network (GAN) [20] or more directly with a reconstruction loss [6].”
Deep Image Prior (2017, methodology): “2 Oxford University, Oxford, England direct pixel-wise errorminimization (Dosovitskiy et al. 2015; Bojanowski et al. 2017) also use ConvNets.”
Interpreting the Latent Space of GANs for Semantic Face Editing (2019, methodology): “GLO [7] optimized the generator and latent code simultaneously to learn a better latent space.”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, background): “• Generative latent optimization (GLO) [32, 4]: Since GAN loss is given by the discriminator, which can be unreliable for limited data, GLO suggests ﬁne-tuning the generator with supervised learning, where the loss is given by the sum of the L1 loss and the perceptual loss [19].”
Image Generation From Small Datasets via Batch Statistics Adaptation (2019, methodology): “Therefore, in this study, we realize supervised learning by simultaneously estimating the latent variable z for all training data such that the generated data are close to the image in the training dataset, which is similar to Generative Latent Optimization [1].”
HyperNeRF (2021, methodology): “2019], let us define S : (x,y,w) → s as an MLP, where the per-shape ambient coordinates {wi } are learned as an embedding layer [Bojanowski et al. 2018].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “Another form of latent conditioning used by NeRF models is Generative Latent Optimization (GLO) [104].”
Semantic Interpolation in Implicit Models (2017, background): “As a consequence, the evaluation of current GAN models is often done by sampling pair of points and linear interpolating between them in the latent space, or performing other types of noise vector arithmetic Bojanowski et al. (2017).”
InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs (2020, background): “[26] optimized the generator and latent code simultaneously to learn a better representation.”
Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields (2023, methodology): “We therefore use a more challenging evaluation procedure, similar to the multiscale Blender dataset used by mip-NeRF [2]: we turn each image into a set of four images that have been bicubically downsampled by scale factors [1, 2, 4, 8] where the downsampled images act as a proxy for additional train”
Context-Aware Synthesis for Video Frame Interpolation (2018, background): “Another alternative that we adopt measures the difference between Laplacian pyramids [3].”
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections (2020, methodology): “Following the framework of Generative Latent Optimization [3], we optimize an appearance embedding for each input image, thereby granting NeRF-W the ﬂexibility to explain away photometric and environmental variations between images by learning a shared appearance representation for the entire photo ”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, unknown): “To reduce the floaters caused by appearance variations, Generative Latent Optimization (GLO) [5] with appearance embeddings [29] is proposed for NeRF-based meth-ods [41, 61].”
SketchEmbedNet: Learning Novel Concepts by Imitating Drawings (2020, background): “It has also been demonstrated in human face images and vector graphics (Bojanowski et al., 2018; Shen et al., 2020; Carlier et al., 2020).”