Wasserstein Auto-Encoders

By Ilya O. Tolstikhin, Ilya O. Tolstikhin, Olivier Bousquet, Sylvain Gelly, Sylvain Gelly, Bernhard Schölkopf. 2017

1,174 citations recorded by the source corpus.

We propose the Wasserstein Auto-Encoder (WAE)---a new algorithm for building a generative model of the data distribution. WAE minimizes a penalized form of the Wasserstein distance between the model distribution and the target distribution, which leads to a different regularizer than the one used by the Variational Auto-Encoder (VAE). This regularizer encourages the encoded training distribution to match the prior. We compare our algorithm with several other techniques and show that it is a generalization of adversarial auto-encoders (AAE). Our experiments show that WAE shares many of the properties of VAEs (stable training, encoder-decoder architecture, nice latent manifold structure) while generating samples of better quality, as measured by the FID score.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Most existing methods address this problem by optimizing certain statistical divergences between q φ ( z ) and p θ ( z ) , such as the KL divergence or Wasserstein distance [85].”
Improved Precision and Recall Metric for Assessing Generative Models (2019, background): “While the quality of results from generative adversarial networks (GAN) [8], variational autoencoders (VAE) [15], autoregressive models [28, 29], and likelihood-based models [6, 14] have seen rapid improvement recently [12, 9, 27, 20, 4, 13], the automatic evaluation of these results continues to be”
Diagnosing and Enhancing VAE Models (2019, methodology): “Note though that the CelebA FID results from Table 3 and Figure 4 below all involve the separate protocol from (Tolstikhin et al., 2018), including a consistent cropping step across all methods.”
Multimodal Unsupervised Image-to-Image Translation (2018, background): “Various improvements to GANs have been proposed, such as multi-stage generation [28–33], better training objectives [34–39], and combination with auto-encoders [40–44].”
Implicit Quantile Networks for Distributional Reinforcement Learning (2018, background): “It has also been a topic of increasing interest in generative modeling (Arjovsky et al., 2017; Bous-quet et al., 2017; Tolstikhin et al., 2017), because unlike the KL divergence, the Wasserstein metric inherently trades off approximate solutions with likelihoods.”
Recent Advances in Autoencoder-Based Representation Learning (2018, neutral): “For more details consult [36, Section 3].”
Generating Natural Adversarial Examples (2017, methodology): “images, while VAEs tend to produce blurrier ones (Goodfellow, 2016). We also plan to apply the fusion and variant of VAEs and GANs such as -GAN in Rosca et al. (2017) and Wasserstein Auto-Encoders in Tolstikhin et al. (2018). Note that as more advanced GANs are introduced to address these issues, th”
Adversarial score matching and improved sampling for image generation (2020, methodology): “We note the similarities between these objective functions and those of an LSGAN adversarial autoencoder (Makhzani et al., 2015; Tolstikhin et al., 2017; Tran et al., 2018), with the distinction of using a denoising autoencoder H as opposed to a standard autoencoder.”
Distribution Matching in Variational Inference (2018, methodology): “|| AAEs are connected to marginal distribution matching not only by the ELBO, but also via Wasserstein distance (Tolstikhin et al., 2018) (see Appendix C).”
Sliced Score Matching: A Scalable Approach to Density and Score Estimation (2019, methodology): “The architectures are very similar to those used in the VAE experiments, and the only difference is that we made decoders implicit, as suggested in Tolstikhin et al. (2018).”
Adversarial Uncertainty Quantification in Physics-Informed Neural Networks (2018, background): “This is a known limitation of adversarial algorithms, and, although several works have led to improvements [54, 55], it still largely remains an open research problem.”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, background): “Other attempts to address posterior collapse include proposing new regularizers (Zhao et al., 2017; Goyal et al., 2017; Tolstikhin et al., 2018; Phuong et al., 2018), deploying less powerful de-coders (Yang et al., 2017; Semeniuta et al., 2017), using lossy input (Chen et al., 2017), utilizing…”
Analyzing Inverse Problems with Invertible Neural Networks (2018, methodology): “While a trainable discriminator loss is often preferred for this task in high-dimensional problems, especially in GAN-based image generation, MMD also works well, is easier to use and much cheaper, and leads to more stable training [43].”
Variational Autoencoder with Implicit Optimal Priors (2018, methodology): “However, the AAE cannot estimate the probability of a data point.”
Gaussian mixture models with Wasserstein distance (2018, methodology): “As in [27], we find that λ = 10 works well.”
From Variational to Deterministic Autoencoders (2019, methodology): “We largely follow the models adopted in [53] with the difference that we consistently apply batch normalization [24] for all models as we found it to improve performance across the range.”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, background): “A vast body of work has tried to alleviate the mode collapse problem in GANs [11, 12, 65, 4, 25, 16, 47] and the posterior collapse problem in VAEs [76, 66, 36, 8, 48, 29].”