From optimal transport to generative modeling: the VEGAN cookbook

By Olivier Bousquet, Sylvain Gelly, Ilya O. Tolstikhin, Carl-Johann Simon-Gabriel, B. Schoelkopf. 2017

154 citations recorded by the source corpus.

We study unsupervised generative modeling in terms of the optimal transport (OT) problem between true (but unknown) data distribution $P_X$ and the latent variable model distribution $P_G$. We show that the OT problem can be equivalently written in terms of probabilistic encoders, which are constrained to match the posterior and prior distributions over the latent space. When relaxed, this constrained optimization problem leads to a penalized optimal transport (POT) objective, which can be efficiently minimized using stochastic gradient descent by sampling from $P_X$ and $P_G$. We show that POT for the 2-Wasserstein distance coincides with the objective heuristically employed in adversarial auto-encoders (AAE) (Makhzani et al., 2016), which provides the first theoretical justification for AAEs known to the authors. We also compare POT to other popular techniques like variational auto-encoders (VAE) (Kingma and Welling, 2014). Our theoretical results include (a) a better understanding of the commonly observed blurriness of images generated by VAEs, and (b) establishing duality between Wasserstein GAN (Arjovsky and Bottou, 2017) and POT for the 1-Wasserstein distance.

Subfields: Computational Mechanics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Generative Models

Citations in the corpus

Autoregressive Quantile Networks for Generative Modeling (2018, methodology): “In this case, due to insufﬁcient training of the critic (Bellemare et al., 2017) or limitations of the function approximator, the gradient direction produced can be arbitrarily bad (Bousquet et al., 2017).”
Diagnosing and Enhancing VAE Models (2019, methodology): “This is equivalent to simply adopting a non-adaptive squared-error loss for the decoder and, at least in part, likely contributes to unrealistic/blurry VAE-generated samples (Bousquet et al., 2017).”
Improving GANs Using Optimal Transport (2018, background): “The connection between GANs and dual form optimal transport is further explored by Bousquet et al. (2017) and Genevay et al. (2017a), who extend the analysis to different optimal transport costs and to a broader model class including latent variables.”
Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems (2019, background): “It has also led to many fundamental studies that aim to further elucidate the capabilities and limitations of such models [31–36], enhance the interpretability of their results [37–39], as well as establish formal connections with well studied topics in mathematics and statistics, including…”
Implicit Quantile Networks for Distributional Reinforcement Learning (2018, background): “It has also been a topic of increasing interest in generative modeling (Arjovsky et al., 2017; Bousquet et al., 2017; Tolstikhin et al., 2017), because unlike the KL divergence, the Wasserstein metric inherently trades off approximate solutions with likelihoods.”
Wasserstein Auto-Encoders (2017, methodology): “ude by pointing out some promising directions for future work. 2 Proposed method Our new method minimizes the optimal transport cost W c(P X;P G) based on the novel autoencoder formulation derived in [11] (see Theorem 1 below). In the resulting optimization problem the decoder tries to accurately re”
Learning Generative Models with Sinkhorn Divergences (2017, methodology): “A very similar conclusion is reached by [5] (see in particular their Proposition 3).”
Computational Optimal Transport (2018, background): “, 2017] and VAE [Bousquet et al., 2017]; see also [Genevay et al.”
Generalized Sliced Wasserstein Distances (2019, background): “…2017; Courty et al., 2017; Peyr´e & Cu-turi, 2018; Schmitz et al., 2018), especially in implicit generative modeling such as OT-based generative adversarial networks (GANs) and variational auto-encoders (Arjovsky et al., 2017; Bousquet et al., 2017; Gulrajani et al., 2017; Tolstikhin et al., 2018).”
Amortized Inference Regularization (2018, background): “Proposition 2 generalizes the analysis in [21] which determined the optimal generative model when P is Gaussian.”
Gaussian mixture models with Wasserstein distance (2018, background): “This level of control built into the prior requires not only a multi-modal prior, but also a training procedure that actually leverages the structure in both the prior and variational distribution, which seems to not be the case for VAEs (see Section 2.1).”
Monge-Ampère Flow for Generative Modeling (2018, background): “In this context, the Brenier theorem Brenier (1991) states that under the quadratic distance measure, the optimal generative map is the gradient of a convex function.”
Continual Variational Autoencoder Learning via Online Cooperative Memorization (2022, methodology): “(4) has an upper bound when Gi is the Gaussian decoder [8,54] :”
Primal-Dual Wasserstein GAN (2018, background): “This work builds up on the theoretical analysis presented in [5, 6].”
GAN and VAE from an Optimal Transport Point of View (2017, background): “Following [4, 7], in the special case of a generative model µ θ = g θ(cid:93) ζ , formula (1) can be conveniently re-written as This is advantageous because now π is deﬁned over Z × X , which is lower-dimensional than X × X , and also because, as in Equation (2), θ does not appear in the…”
Parametric Adversarial Divergences are Good Task Losses for Generative Modeling (2017, background): “Indeed it is a known problem that VAEs produce blurry samples (Arjovsky et al., 2017), in fact even if the approximate posterior matches exactly the true posterior, which would correspond to the evidence lower-bound being tight, the output of the VAE would still be blurry (Bousquet et al., 2017).”