Understanding disentangling in β-VAE

By Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nicholas Watters, Guillaume Desjardins, Alexander Lerchner. 2018

962 citations recorded by the source corpus.

We present new intuitions and theoretical assessments of the emergence of disentangled representation in variational autoencoders. Taking a rate-distortion theory perspective, we show the circumstances under which representations aligned with the underlying generative factors of variation of data emerge when optimising the modified ELBO bound in $\beta$-VAE, as training progresses. From these insights, we propose a modification to the training regime of $\beta$-VAE, that progressively increases the information capacity of the latent code during training. This modification facilitates the robust learning of disentangled representations in $\beta$-VAE, without the previous trade-off in reconstruction accuracy.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “The AnnealedVAE (Burgess et al., 2018) progressively increase the bottleneck capacity so that the encoder can focus on learning one factor of variation at the time (the one that most contribute to a small reconstruction error).”
Flexibly Fair Representation Learning by Disentanglement (2019, background): “, 2017), as discussed in section 2, motivated a number of subsequent studies that examine why adding additional weight on the KL-divergence of the ELBO encourages disentangled representations (Alemi et al., 2018; Burgess et al., 2017).”
On the Transfer of Disentangled Representations in Realistic Settings (2020, background): “…to p ( z ), diﬀerent regularization strategies have been proposed, along with evaluation metrics to gauge the disentanglement of the learned representations (Higgins et al., 2017a; Kim and Mnih, 2018; Burgess et al., 2018; Kumar et al., 2018; Chen et al., 2018; Eastwood and Williams, 2018).”
Weakly-Supervised Disentanglement Without Compromises (2020, methodology): “Recently, there has been a renewed interest in the unsupervised learning of disentangled representations (Higgins et al., 2017a; Burgess et al., 2018; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2018) along with quantitative evaluation (Kim & Mnih, 2018; Eastwood & Williams, 2018; Kumar et al”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, methodology): “In this paper, we consider the setup of the recent unsupervised methods [27, 26, 48, 42, 9, 52, 71, 10].”
MONet: Unsupervised Scene Decomposition and Representation (2019, methodology): “As discussed above, we utilised a weight modulating the latent KL in the loss [Higgins et al., 2017, Burgess et al., 2018] and a broadcast decoder [Watters et al., 2019] in MONet to encourage disentangling of the latent representation at the feature level.”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, background): “In the context of learning interpretable representations [2, 17, 5, 40, 38] it is useful to assume that the data originates from a process with some generating factors.”
Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness (2018, methodology): “Visualization of interventional effects in the annealed β-VAE model (Higgins et al., 2017; Burgess et al., 2018).”
Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems (2019, background): “…studies that aim to further elucidate the capabilities and limitations of such models [31–36], enhance the interpretability of their results [37–39], as well as establish formal connections with well studied topics in mathematics and statistics, including importance sampling [40,41] and…”
Towards a Definition of Disentangled Representations (2018, methodology): “We trained one of the current state of the art disentangling models CCI-VAE [13] on a dataset of observations from such a grid world.”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, methodology): “For learning disentangled representations, we employ the Minimum Description Length (MDL) principle [7, 57] and replace the second term from equation (20) by γ|DKL[qς(z|x)||p(z)]−C|, where γ and C are a multiplicative and a linear constant used for controling the degree of disentanglement.”
Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies (2018, methodology): “VASE also has moving Fashion- and MNIST-specific latents, while CCI-VAE shares all latents between all datasets.”
Where is the Information in a Deep Neural Network? (2019, background): “Note that “phase transitions” in feature learning – where a feature is suddenly learned, following a period in which the representation is relatively stable – are also observed when analyzing activations, rather than the weights, from an information theoretic point of view (Burgess et al., 2017; Rez”
On Variational Bounds of Mutual Information (2019, methodology): “This variational upper bound is often used as a regularizer to limit the capacity of a stochastic representation (e.g. Rezende et al., 2014; Kingma & Welling, 2013; Burgess et al., 2018).”
Variational Autoencoders and Nonlinear ICA: A Unifying Framework (2019, background): “The most prominent of such models are β-VAEs and their extensions (Burgess et al., 2018; Higgins et al., 2016, 2018; Esmaeili et al., 2018; Kim and Mnih, 2018; Chen et al., 2018), in which the authors introduce adjustable hyperparameters in the VAE objective to encourage disentanglement.”