Isolating Sources of Disentanglement in VAEs

By T. Chen, Xuechen Li, Roger Baker Grosse, David Kristjanson Duvenaud. 2018

451 citations recorded by the source corpus.

We decompose the evidence lower bound to show the existence of a term measuring the total correlation between latent variables. We use this to motivate our $\beta$-TCVAE (Total Correlation Variational Autoencoder), a refinement of the state-of-the-art $\beta$-VAE objective for learning disentangled representations, requiring no additional hyperparameters during training. We further propose a principled classifier-free measure of disentanglement called the mutual information gap (MIG). We perform extensive quantitative and qualitative experiments, in both restricted and non-restricted settings, and show a strong relation between total correlation and disentanglement, when the latent variables model is trained using our framework.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “The Mutual Information Gap (MIG) (Chen et al., 2018) measures for each factor of variation the normalized gap in mutual information between the highest and second highest coordinate in r(x).”
Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, methodology): “Two of the baselines considered had hyperparameters to tune, SlowVAE (Klindt et al., 2021) and TCVAE (Chen et al., 2018).”
Recurrent Independent Mechanisms (2019, background): “The assumption about the joint distribution between the high-level variables is different from the assumption commonly found in many papers on disentangling factors of variation (Higgins et al., 2016; Burgess et al., 2018; Chen et al., 2018), where the high level variables are assumed to be marginal”
On the Fairness of Disentangled Representations (2019, background): “The Mutual Information Gap (MIG) [13] computes for each factor of variation the normalized gap on the top two entries in the matrix of pairwise mutual information between z and r(x).”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, methodology): “[52] studied several of these metrics, which we will adopt for our purposes in this work: the BetaVAE score [27], the FactorVAE score [42], the Mutual Information Gap (MIG) [10], the disentanglement score from Eastwood et al.”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, background): “VAE framework in order to learn a meaningful representation of the data by imposing a large penalty on the Kullback-Leibler (KL) divergence term [7,17,25] or on the total correlation latent variables [12,20,28,30]. These approaches perform well on independent and identically distributed data samples”
Disentangling Factors of Variation Using Few Labels (2019, methodology): “We conclude that the semi-supervised loss improves the training and transfer better to different metrics 0.84 0.86 0.88 0.90 0.92 0.94 0.96 V a lu e Metric = BetaVAE Score 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 Metric = FactorVAE Score 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 Metric = MIG 0.15 0.20 ”
Inductive biases for deep learning of higher-level cognition (2020, background): “el variables at the top of our deep learning hierarchy is diﬀerent from the assumption commonly found in many papers on disentangling factors of variation (Higgins et al., 2016; Burgess et al., 2018; Chen et al., 2018; Kim and Mnih, 2018; Locatello et al., 2019), where the high-level variables are a”
Lifelong Teacher-Student Network Learning (2021, background): “gories of characteristics in the feature space [15], [16], [17].”
Unsupervised Object Segmentation by Redrawing (2019, background): “FactorVAE [30] and β-TCVAE [7] extract a total correlation term from the KL term of the VAE objective and specifically re-weight it instead of the whole KL term.”
Multimodal Representation Learning via Maximization of Local Mutual Information (2021, unknown): “The definition of usefulness is casedriven [3,5,26].”
Compositionality and Generalization In Emergent Languages (2020, methodology): “ et al.,2019), several interesting metrics have been proposed to quantify disentanglement, as reviewed inSuter et al. (2019). We build in particular upon the Information Gap disentanglement measure ofChen et al. (2018), evaluating how well representations capture independence in the input sets. Our ”
Disentangling Influence: Using Disentangled Representations to Audit Model Predictions (2019, methodology): “Common measures for disentanglement such as the mutual information gap (MIG) do not apply well to our method since we disentangle the features one at a time, as opposed to simultaneously [5].”
Learning to Manipulate Individual Objects in an Image (2020, background): “To encourage the alignment of the learned representation with the underlying generative factors, several constraints have been proposed to enforce the disentanglement of the latent codes [17, 5, 1, 30, 22, 9, 4, 18].”
Controlling Text Edition by Changing Answers of Specific Questions (2021, background): “Implicit disentanglement (Higgins et al., 2017; Chen et al., 2018; Moyer et al., 2018; Mathieu et al., 2018; Kim and Mnih, 2018) separates the latent space into several components in a purely unsupervised way, expecting that each component corresponds to an attribute.”
Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing (2019, background): “The KL term in (5) can be decomposed into two reﬁned terms (Chen et al., 2018; Hoffman and Johnson, 2016): where F 1 is the mutual information (MI) measured by q .”
Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning (2020, background): “For the former, several metrics have been proposed in VAE-based disentangled representation learning, such as factor score [25] and mutual information gap [5].”