Towards a Definition of Disentangled Representations

By Irina Higgins, David Amos, David Pfau, Sébastien Racanière, Loic Matthey, Danilo Jimenez Rezende, Alexander Lerchner. 2018

546 citations recorded by the source corpus.

How can intelligent agents solve a diverse set of tasks in a data-efficient manner? The disentangled representation learning approach posits that such an agent would benefit from separating out (disentangling) the underlying structure of the world into disjoint parts of its representation. However, there is no generally agreed-upon definition of disentangling, not least because it is unclear how to formalise the notion of world structure beyond toy datasets with a known ground truth generative process. Here we propose that a principled solution to characterising disentangled representations can be found by focusing on the transformation properties of the world. In particular, we suggest that those transformations that change only some properties of the underlying world state, while leaving all other properties invariant, are what gives exploitable structure to any kind of data. Similar ideas have already been successfully applied in physics, where the study of symmetry transformations has revolutionised the understanding of the world structure. By connecting symmetry transformations to vector representations using the formalism of group and representation theory we arrive at the first formal definition of disentangled representations. Our new definition is in agreement with many of the current intuitions about disentangling, while also providing principled resolutions to a number of previous points of contention. While this work focuses on formally defining disentangling - as opposed to solving the learning problem - we believe that the shift in perspective to studying data transformations can stimulate the development of better representation learning algorithms.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “While prior work (Steenbrugge et al., 2018; LaversanneFinot et al., 2018; Nair et al., 2018; Higgins et al., 2017b, 2018b) successfully applied disentanglement methods such as β-VAE on a variety of downstream tasks, it is not clear to us that these approaches and trained models performed well…”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “” Such a latent space, made of disjoint data generation factors, is a disentangled latent space [26, 123].”
On the Binding Problem in Artificial Neural Networks (2020, background): “A disentangled representation aims to make these attributes explicit by establishing a local correspondence between (independent) factors of variation and features (Barlow et al., 1989; Schmidhuber, 1992c; Higgins et al., 2017a, 2018; Ridgeway and Mozer, 2018).”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “The generative model (1)–(2) does not model additive noise; we assume that noise is explicitly modeled as a latent variable and its effect is manifested through g as done by (Bengio et al., 2013; Locatello et al., 2019b; Higgins et al., 2018; 2017a; Suter et al., 2019; Reed et al., 2015; LeCun et al”
Weakly Supervised Disentanglement with Guarantees (2019, methodology): “We propose a set of definitions for disentanglement that can handle correlated factors and are inspired by many existing definitions in the literature (Higgins et al., 2018; Suter et al., 2018; Ridgeway & Mozer, 2018).”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “[17] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner.”
On the Fairness of Disentangled Representations (2019, background): “Indeed, isolating each independent factor of variation into the independent components of a representation vector should make it both interpretable and simplify downstream prediction tasks [6, 7, 29, 35, 56, 58, 60, 74, 82, 87, 89, 90, 1, 28].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, methodology): “Despite an increasing interest in learning disentangled representations, a precise definition is still a topic of debate [16, 26, 52, 63].”
Multi-Object Representation Learning with Iterative Variational Inference (2019, background): “While its precise definition is still highly debated (Higgins et al., 2018; Eastwood & Williams, 2018; Ridgeway & Mozer, 2018; Locatello et al., 2018), the concept of disentanglement has generated a lot of interest recently.”
Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness (2018, background): “It is striking that despite the recent popularity of disentangled representation learning approaches, a commonly accepted definition and validation metric is missing (Higgins et al., 2018).”
Theory and Evaluation Metrics for Learning Disentangled Representations (2019, background): “[15] proposed a definition based on group theory [10] which is (informally) stated as follows: “A representation z is disentangled w.”
Equivariant Hamiltonian Flows (2019, background): “By definition in [10], these will act on independent subspaces of a disentangled q.”
On Variational Bounds of Mutual Information (2019, background): “While the exact definition of disentangling remains elusive (Locatello et al., 2018; Higgins et al., 2018; Mathieu et al., 2018), many papers have focused on reducing statistical dependency between latent variables as a proxy (Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2017).”
Variational Autoencoders and Nonlinear ICA: A Unifying Framework (2019, background): “The most prominent of such models are β-VAEs and their extensions [4, 9, 8, 7, 15, 5], in which the authors introduce adjustable hyperparameters in the VAE objective to encourage disentanglement.”
Disentangling Factors of Variation Using Few Labels (2019, background): “The hope is that such representations will be interpretable, maximally compact, allow for counterfactual reasoning and be useful for a large variety of downstream task [2, 49, 39, 3, 56, 37, 17, 41, 62, 19, 60].”
Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs (2019, background): “This naturally poses a great challenge for both designing disentangling metrics and formally deﬁning disentangling [cf. Higgins et al., 2018].”
Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere (2020, background): “…the intuition that connected sets with smooth boundaries are nearly linearly separable in the hyperspherical geometry (see Figure 2), since linear separability is one of the most widely used criteria for representation quality and is related to the notion of disentanglement (Higgins et al., 2018).”
Editing in Style: Uncovering the Local Semantics of GANs (2020, background): “While definitions of disentanglement are many and varied [12], the common idea is that an attribute of interest, which we often consider semantic, can be manipulated independently of other attributes.”