Taming VAEs

By Danilo Jimenez Rezende, Fabio Viola. 2018

200 citations recorded by the source corpus.

In spite of remarkable progress in deep latent variable generative modeling, training still remains a challenge due to a combination of optimization and generalization issues. In practice, a combination of heuristic algorithms (such as hand-crafted annealing of KL-terms) is often used in order to achieve the desired results, but such solutions are not robust to changes in model architecture or dataset. The best settings can often vary dramatically from one problem to another, which requires doing expensive parameter sweeps for each new case. Here we develop on the idea of training VAEs with additional constraints as a way to control their behaviour. We first present a detailed theoretical analysis of constrained VAEs, expanding our understanding of how these models work. We then introduce and analyze a practical algorithm termed Generalized ELBO with Constrained Optimization, GECO. The main advantage of GECO for the machine learning practitioner is a more intuitive, yet principled, process of tuning the loss. This involves defining of a set of constraints, which typically have an explicit relation to the desired model performance, in contrast to tweaking abstract hyper-parameters which implicitly affect the model behavior. Encouraging experimental results in several standard datasets indicate that GECO is a very robust and effective tool to balance reconstruction and compression constraints.

Subfields: Artificial Intelligence, Control and Systems Engineering, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Generative Models

Citations in the corpus

BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “Contrarily, we attribute it to the lack of overall model expressiveness for accurately modeling complex input distributions, as discussed in [59, 41].”
Equivariant Hamiltonian Flows (2019, methodology): “Parameters were optimised using Adam [15] (learning rate 3e-4) and Lagrange multipliers were optimised using the same method as [12].”
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, methodology): “To prevent this behaviour, we use the generalised ELBO with constrained optimisation ( GECO ) ob-jective from Rezende and Viola [29] instead, which changes the learning problem to minimising the KL -divergence subject to a reconstruction constraint.”
Improving Generalization for Abstract Reasoning Tasks Using Disentangled Feature Representations (2018, background): “The behavior of these models has been widely studied [4, 5, 6, 7] and a clear trade-off between desirable latent space properties (such as disentanglement of generative factors, linearity & sparsity) and reconstruction quality is usually present.”
Preventing Posterior Collapse with delta-VAEs (2019, background): “This phenomenon, also known as the “posterior holes” problem (Jimenez Rezende & Viola, 2018), can be exacerbated in δ -VAEs, where the systematic mismatch between the prior and the posterior might induce a large gap between the prior and aggregate posterior.”
From Variational to Deterministic Autoencoders (2019, background): “Many works have focused on diagnosing the VAE framework, the terms in its objective [1, 22, 60], and ultimately augmenting it to solve optimization issues [13, 39].”
Learning Hierarchical Priors in VAEs (2019, methodology): “thm. However, in general this objective can only be guaranteed to be the ELBO if = 1, or in case of 0 <1, a scaled lower bound on the ELBO. 1The optimisation problem is not explicitly stated in [25]. 2 2.2 Hierarchical Priors for Learning Informative Latent Representations In this section, we pro”
Perceptual Generative Autoencoders (2019, methodology): “Examples include variational autoencoders (VAEs) (Kingma & Welling, 2014), reversible generative models (Dinh et al., 2014; 2017; Kingma & Dhariwal, 2018), and Wasserstein autoencoders (WAEs) (Tolstikhin et al., 2018).”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “…parameterization, in which the covariance matrix is speciﬁed with a single shared (Kingma et al., 2016; Dai & Wipf, 2019; Edwards & Storkey, 2016; Rezende & Viola, 2018) parameter σ as Σ = σI often works better in practice: The parameter σ can be optimized together with parameters of the…”
Variable-rate discrete representation learning (2021, methodology): “Therefore, we adapt λ on the ﬂy during training, to achieve a target AER (Rezende & Viola, 2018).”
Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis (2019, methodology): “…we would like to smoothly optimize for a speciﬁc bound on the embedding capacity, we adapt the Lagrange multiplier-based optimization approach of Rezende & Viola (2018) by applying it to the KL term rather than the reconstruction term. where θ are the model parameters, β serves as an…”
On the Sentence Embeddings from BERT for Semantic Textual Similarity (2020, background): “This is a common problem in the context of representation learining (Rezende and Viola, 2018; Li et al., 2019; Ghosh et al., 2020).”
Automated curricula through setter-solver interactions (2019, background): “Adding the challenge of trying to discover latent variables with which to condition and performing even a simple manipulation of them makes things much more difﬁcult (Rezende & Viola, 2018) (c.f. the difﬁculty of learning hierarchies of latent variables (Sønderby et al., 2016; Maaløe et al., 2019)).”
Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (2019, unknown): “Several other papers have studied alternative formulations of the VAE objective [34, 13, 2, 29, 41].”
VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention (2021, methodology): “We use the β-VAE training objective and a mel-spectrogram loss inspired by (Rezende & Viola, 2018).”
Generative Neurosymbolic Machines (2020, methodology): “(b) instead of optimizing the Constrained Optimisation objective (GECO) [27], we optimize the evidence lower bound with a β value of 15 for arrow room dataset and 10 for MNIST dataset.”