Learning Deep Disentangled Embeddings with the F-Statistic Loss

By Karl Ridgeway, Michael C. Mozer. 2018

238 citations recorded by the source corpus.

Deep-embedding methods aim to discover representations of a domain that make explicit the domain's class structure. Disentangling methods aim to make explicit compositional or factorial structure. We combine these two active but independent lines of research and propose a new paradigm for discovering disentangled representations of class structure; these representations reveal the underlying factors that jointly determine class. We propose and evaluate a novel loss function based on the $F$ statistic, which describes the separation of two or more distributions. By ensuring that distinct classes are well separated on a subset of embedding dimensions, we obtain embeddings that are useful for few-shot learning. By not requiring separation on all dimensions, we encourage the discovery of disentangled representations. Our embedding procedure matches or beats state-of-the-art procedures on deep embeddings, as evaluated by performance on recall@$k$ and few-shot learning tasks. To evaluate alternative approaches on disentangling, we formalize two key properties of a disentangled representation: modularity and explicitness. By these criteria, our procedure yields disentangled representations, whereas traditional procedures fail. The goal of our work is to obtain more interpretable, manipulable, and generalizable deep representations of concepts and categories.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “The scores proposed by Eastwood and Williams (2018); Kumar et al. (2018); Chen et al. (2018); Ridgeway and Mozer (2018) first construct a matrix of relation between factors of variation and codes (for example pairwise mutual information) and then aggregate this matrix into a single final number.”
Toward Causal Representation Learning (2021, background): “Much of the existing work on disentanglement [62], [110], [135], [157]–[159], [202], [256] focuses on independent factors of variation.”
Towards Causal Representation Learning (2021, background): “Much of the existing work on disentanglement [109, 158, 159, 256, 157, 135, 202, 61] focuses on independent factors”
On the Binding Problem in Artificial Neural Networks (2020, background): “A disentangled representation aims to make these attributes explicit by establishing a local correspondence between (independent) factors of variation and features (Barlow et al., 1989; Schmidhuber, 1992c; Higgins et al., 2017a, 2018; Ridgeway and Mozer, 2018).”
On the Transfer of Disentangled Representations in Realistic Settings (2020, methodology): “The Mutual Information Gap (MIG) (Chen et al., 2018), Modularity (Ridgeway and Mozer, 2018), DCI Disentanglement (Eastwood and Williams, 2018) and SAP score (Kumar et al., 2018) are based on matrices relating factors of variation and codes (e.g. pairwise mutual information, feature importance and pr”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “…of disentangled representations (Higgins et al., 2017a; Burgess et al., 2018; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2018) along with quantitative evaluation (Kim & Mnih, 2018; Eastwood & Williams, 2018; Kumar et al., 2018; Chen et al., 2018; Ridgeway & Mozer, 2018; Duan et al., 2019).”
Weakly Supervised Disentanglement with Guarantees (2019, methodology): “We propose a set of deﬁnitions for disentanglement that can handle correlated factors and are inspired by many existing deﬁnitions in the literature (Higgins et al., 2018; Suter et al., 2018; Ridgeway & Mozer, 2018).”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “[47] Karl Ridgeway and Michael C Mozer.”
On the Fairness of Disentangled Representations (2019, methodology): “In Figure 5 (left), we observe that the rank correlation between the adjusted disentanglement scores (except Modularity) on Color-dSprites is consistenly positive.”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Although the precise circumstances that give rise to disentanglement are still being debated, the core concept of a local correspondence between data-generative factors and learned latent codes is generally agreed upon [16, 26, 52, 63, 71].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “We observe that all metrics except Modularity seem to be correlated strongly on the data sets dSprites, Color-dSprites and Scream-dSprites and mildly on the other data sets.”
Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness (2018, methodology): “(Ridgeway & Mozer, 2018).”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, background): “The notions in [36] follow those in [12], hence, suffer from the same disadvantage.”
Towards a Definition of Disentangled Representations (2018, background): “These intuitions can be characterised along three dimensions: modularity , compactness and explicitness 2 [65].”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “Speciﬁcally, disentanglement metrics measure how predictive the individual latent factors are for the ground-truth factors, see, e.g., [2, 3, 6, 41, 4, 42].”
Disentangling Factors of Variation Using Few Labels (2019, background): “We conclude that the semi-supervised loss improves the training and transfer better to different metrics 0.84 0.86 0.88 0.90 0.92 0.94 0.96 V a lu e Metric = BetaVAE Score 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 Metric = FactorVAE Score 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 Metric = MIG 0.15 0.20 ”
StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation (2020, background): “Recent studies of disentangled representations [8, 27] consider a latent representation to be perfectly disentangled if each latent dimension controls a single visual attribute (disentanglement), and each attribute is controlled by a single dimension (completeness).”
Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs (2019, neutral): “This has been noted by others in the disentangling literature [Ridgeway and Mozer, 2018, Eastwood and Williams, 2018].”
Unsupervised Model Selection for Variational Disentangled Representation Learning (2019, background): “When talking about disentangled representations, three properties are generally considered: modularity , compactness and explicitness 2 (Ridgeway & Mozer, 2018).”