Fixing a Broken ELBO

By Alexander A. Alemi, Ben Poole, Ian S. Fischer, Joshua V. Dillon, Rif A. Saurous, Kevin P. Murphy. 2017

555 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Evaluating Lossy Compression Rates of Deep Generative Models (2020, methodology): “For lossless compression with generative models, there is a practical compression scheme which nearly achieves the theoretical rate (i.e. the negative ELBO): bits-back encoding.”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “Addressing posterior mismatch in VAEs Most methods address this mismatch problem by improving inference models [64, 51, 89], prior models [90, 3, 87], or objective functions [103, 104, 105, 1, 59]; all these approaches optimize the posterior model to be close to the prior.”
Symbolic Music Generation with Diffusion Models (2021, background): “The VAE loss encourages the marginal posterior to match to the prior (Hoffman & Johnson; Alemi et al., 2018), but to encode information, dimensions must have smaller variance per an example.”
On Mutual Information Maximization for Representation Learning (2019, background): “In the context of VAEs, Alemi et al. (2018) similarly observed that models achieving the same evidence lower bound value can lead to vastly different representations depending on the employed encoder architecture, and do not necessarily capture useful information about the data (Tschannen et al., 20”
GAIT: A Geometric Approach to Information Theory (2019, background): “…ubiquitous in modern machine learning: from variable selection via information gain in decision trees (Ben-David and Shalev-Shwartz, 2014), to using entropy as a regularizer in reinforcement learning (Fox et al., 2016), to rate-distortion theory for training generative models (Alemi et al., 2018).”
Flexibly Fair Representation Learning by Disentanglement (2019, background): “The introduction of the β -VAE (Higgins et al., 2017), as discussed in section 2, motivated a number of subsequent studies that examine why adding additional weight on the KL-divergence of the ELBO encourages disentangled representations (Alemi et al., 2018; Burgess et al., 2017).”
Learning Representations by Maximizing Mutual Information in Variational Autoencoders (2019, background): “Although successful for multiple isolated applications, VAEs have not always proven to be reliable for extracting highlevel representations of observations [10], [11].”
Shaping Belief States with Generative Environment Models for RL (2019, background): “It is known that expressive frame generative models are hard to condition [22, 23, 24, 25].”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, background): “, (Alemi et al., 2018) for a discussion of the KL weighting term.”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, neutral): “Several recent works approached this issue [5, 40, 8, 1, 12, 31, 9].”
Generalized Variational Continual Learning (2020, background): “Outside of the Bayesian Neural Network context, values of β > 1 have also been explored (Higgins et al., 2017), and more generally different values of β trace out different points on a rate-distortion curve for VAEs (Alemi et al., 2018).”
Lifelong Generative Modeling (2017, background): “This has been attributed to the fact that VAEs don’t learn the true posterior and make a simplistic assumption regarding the reconstruction distribution Pθ(x|z) [53, 54].”
On Variational Bounds of Mutual Information (2019, methodology): “…x ∼ p ( x ) , the goal is to learn a stochastic representation of the data p θ ( y | x ) that has maximal MI with X subject to constraints on the mapping (e.g. Bell & Sejnowski, 1995; Krause et al., 2010; Hu et al., 2017; van den Oord et al., 2018; Hjelm et al., 2018; Alemi et al., 2017).”
Recent Advances in Autoencoder-Based Representation Learning (2018, methodology): “Finally, we provide a critique of unsupervised representation learning through the rate-distortion framework of Alemi et al. [30] and discuss the implications in Section 7.”
Self-Supervised Learning: Generative or Contrastive (2020, background): “On the other hand, achieving the same lower bound value can lead to vastly different representations and performance on downstream tasks, which indicates that it does not necessarily capture useful information of data [3] [128] [13].”
Structured Disentangled Representations (2018, methodology): “In settings where we are not primarily interested in inducing disentangled representations, the β-VAE objective has also been used with β < 1 in order to improve the quality of reconstructions [Alemi et al., 2016, Engel et al., 2017, Liang et al., 2018]. While this also decreases the relative weight”
Learning Controllable Fair Representations (2018, methodology): “We note that similar bounds have been proposed in Alemi et al. (2016, 2017); Zhao et al. (2018); Grover et al. (2019).”
Taming VAEs (2018, methodology): “,Similarly to [36] we study VAEs with supplementary constraints in addition to the ELBO objective function and we study its behaviour in the information plane as in [31].”
Preventing Posterior Collapse with delta-VAEs (2019, background): “Most common approaches either change the objective (Higgins et al., 2017; Alemi et al., 2017; Zhao et al., 2017; Chen et al., 2016; Lucas & Verbeek, 2017), or weaken the decoder (Bowman et al., 2015; Gulrajani et al., 2016).”
Emergence of Invariance and Disentanglement in Deep Representations (2017, methodology): “In parallel developments, the IB Lagrangian was used as a regularized loss function for learning representation, leading to new information theoretic regularizers (Achille and Soatto, 2016; Alemi et al., 2017).”