A Framework for the Quantitative Evaluation of Disentangled Representations

By Cian Eastwood, Christopher K. I. Williams. 2018

556 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “These results may not be surprising given the insights presented by Eastwood and Williams (2018).”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, background): “Unfortunately the metrics recently proposed for quantifying disentanglement [25, 30, 6, 18] require an encoder network that maps input images to latent codes.”
Toward Causal Representation Learning (2021, background): “Much of the existing work on disentanglement [62], [110], [135], [157]–[159], [202], [256] focuses on independent factors of variation.”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “If we use an unsupervised algorithm for disentangling the space, and then evaluate whether it indeed disentangled the known concepts, then we have a useful quantitative evaluation (Higgins et al., 2017; Chen et al., 2018b; Kim and Mnih, 2018; Eastwood and Williams, 2018; Do and Tran, 2019).”
Towards Causal Representation Learning (2021, background): “Much of the existing work on disentanglement [109, 158, 159, 256, 157, 135, 202, 61] focuses on independent factors”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “When ground truth information is available, an alternative is to directly measure selected properties of the object representations, such as local correspondence between ground-truth factors of variation and features for disentanglement (Eastwood and Williams, 2018).”
On the Transfer of Disentangled Representations in Realistic Settings (2020, methodology): “…to p ( z ), diﬀerent regularization strategies have been proposed, along with evaluation metrics to gauge the disentanglement of the learned representations (Higgins et al., 2017a; Kim and Mnih, 2018; Burgess et al., 2018; Kumar et al., 2018; Chen et al., 2018; Eastwood and Williams, 2018).”
Weakly-Supervised Disentanglement Without Compromises (2020, methodology): “…disentanglement metrics in Locatello et al. (2019b): BetaVAE score (Higgins et al., 2017a), FactorVAE score (Kim & Mnih, 2018), Mutual Information Gap (MIG) (Chen et al., 2018), Modularity (Ridgeway & Mozer, 2018), DCI Disentanglement (Eastwood & Williams, 2018) and SAP score (Kumar et al., 2018).”
Weakly Supervised Disentanglement with Guarantees (2019, neutral): “This is in contrast to the existing disentanglement deﬁnitions and metrics, which require access to the ground truth factors (Higgins et al., 2017; Kumar et al., 2018; Kim & Mnih, 2018; Chen et al., 2018a; Suter et al., 2018; Ridgeway & Mozer, 2018; Eastwood & Williams, 2018).”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, neutral): “[11] Cian Eastwood and Christopher K.”
On the Fairness of Disentangled Representations (2019, methodology): “The Disentanglement metric of [22] (which we call DCI Disentanglement following [61]) is based on the entropy of the probability that a dimension of r(x) is useful for predicting z.”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, methodology): “Despite an increasing interest in learning disentangled representations, a precise deﬁnition is still a topic of debate [16, 26, 50, 61].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, methodology): “Disentanglement is then measured as a particular structural property of these relations (Higgins et al., 2017a; Kim & Mnih, 2018; Eastwood & Williams, 2018; Kumar et al., 2017; Chen et al., 2018; Ridgeway & Mozer, 2018).”
Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness (2018, methodology): “We compute the feature importance based disentanglement scores, as discussed by Eastwood & Williams (2018), using random forests with 50 decision trees that are split up to a minimal leaf size of 500.”
Disentangling by Factorising (2018, background): “More recently Eastwood & Williams (2018) have extended this idea to disentanglement by training a Lasso regressor to map z to f and using its trained weights to quantify disentanglement.”
Learning Deep Disentangled Embeddings with the F-Statistic Loss (2018, background): “In Eastwood & Williams (2018), all three are included; in Kim & Mnih (2017), modularity and compactness are included, but not explicitness; and in Higgins et al. (2017), modularity is included, but not compactness or explicitness.”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, background): “The notions in [36] follow those in [12], hence, suffer from the same disadvantage.”
Towards a Definition of Disentangled Representations (2018, background): “To date, most work on disentangling has relied on comparison with human intuition for evaluation, though a number of metrics have been proposed in the case where the ground truth factors are known [27, 65, 51, 47, 55, 72].”
Isolating Sources of Disentanglement in VAEs (2018, background): “Many have proposed classiﬁer-based metrics for assessing the quality of disentanglement [7, 8, 12, 13, 14, 15].”