Autoencoding beyond pixels using a learned similarity metric

By Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, Ole Winther. 2015

2,248 citations recorded by the source corpus.

We present an autoencoder that leverages learned representations to better measure similarities in data space. By combining a variational autoencoder with a generative adversarial network we can use learned feature representations in the GAN discriminator as basis for the VAE reconstruction objective. Thereby, we replace element-wise errors with feature-wise errors to better capture the data distribution while offering invariance towards e.g. translation. We apply our method to images of faces and show that it outperforms VAEs with element-wise similarity measures in terms of visual fidelity. Moreover, we show that the method learns an embedding in which high-level abstract visual features (e.g. wearing glasses) can be modified using simple arithmetic.

Subfields: Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

An efficient framework for learning sentence representations (2018, background): “functions defined in a feature space as opposed to the raw data space have been found to be more attractive in recent work for similar reasons (Larsen et al., 2015; Pathak et al., 2017).”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “We believe that previous claims stating that this type of model can only generate blurry images should be disregarded [27].”
Density estimation using Real NVP (2016, background): “Moreover, the resulting cost function does not to rely on a fixed form reconstruction cost such as square error [38, 47], and generates sharper samples as a result.”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “Therefore, our argument provides an explanation as to why we observe better sample quality results from GANs and diffusion models than VAEs and NFs.”
Taming Transformers for High-Resolution Image Synthesis (2020, methodology): “To do so, we propose VQGAN, a variant of the original VQVAE, and use a discriminator and perceptual loss [36, 26, 35, 16] to keep good perceptual quality at increased compression rate.”
Dual Contradistinctive Generative Autoencoder (2020, positive): “The VAE/GAN algorithm [41] combines a reconstruction loss with an adversarial loss.”
Diagnosing and Enhancing VAE Models (2019, background): “Experiments from Section 5 reveal that this procedure can generate high-quality crisp samples, avoiding the blurriness often attributed to VAE models in the past (Dosovitskiy and Brox, 2016; Larsen et al., 2015).”
Image-to-Image Translation with Conditional Adversarial Networks (2016, background): “It is well known that the L2 loss – and L1, see Figure 3 – produces blurry results on image generation problems [31].”
Generating Images with Perceptual Similarity Metrics based on Deep Networks (2016, background): “This is similar to (Larsen et al., 2015), but the comparator does not have to be a part of the discriminator.”
Large Scale Adversarial Representation Learning (2019, methodology): “In another proposed VAE-GAN hybrid [21] the pixel-space reconstruction error used in most VAEs is replaced with feature space distance from an intermediate layer of a GAN discriminator.”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, methodology): “We note that a similar loss is used for training VAE-GANs [29].”
Unsupervised Image-to-Image Translation Networks (2017, methodology): “The framework, as illustrated in Figure 1, is motivated by recent deep generative models including variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014; Larsen et al., 2016) and generative adversarial networks (GANs) (Goodfellow et al.”
Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks (2017, background): “Indeed, it was observed that applying standard Variational Autoencoders to natural images often results in blurry images (Larsen et al., 2015).”
Semantic Image Inpainting with Deep Generative Models (2016, background): “Jointly training VAEs with an adveserial loss prevents the smoothness [18], but may lead to artifacts.”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, methodology): “To further enforce the smoothness over the conditioning manifold and avoid overfitting [6, 14], we add the following regularization term to the objective of the generator during training,”
Adversarially Learned Inference (2016, methodology): “Recently, efforts have aimed to bridge the gap between VAEs and GANs, to learn generative models with higher-quality samples while learning an efﬁcient inference network (Larsen et al. , 2015; Lamb et al. , 2016; Dosovitskiy and Brox, 2016).”
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks (2017, background): “avoid overfitting [9], [22], we add the following regularization term to the objective of the generator during training,”
Toward Controlled Generation of Text (2017, background): “Larsen et al. (2016) combine VAEs with GANs for enhanced image generation.”