Disentangling Factors of Variation Using Few Labels

By Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem. 2019

129 citations recorded by the source corpus.

Learning disentangled representations is considered a cornerstone problem in representation learning. Recently, Locatello et al. (2019) demonstrated that unsupervised disentanglement learning without inductive biases is theoretically impossible and that existing inductive biases and unsupervised methods do not allow to consistently learn disentangled representations. However, in many practical settings, one might have access to a limited amount of supervision, for example through manual labeling of (some) factors of variation in a few training examples. In this paper, we investigate the impact of such supervision on state-of-the-art disentanglement methods and perform a large scale study, training over 52000 models under well-defined and reproducible experimental conditions. We observe that a small number of labeled examples (0.01--0.5\% of the data set), with potentially imprecise and incomplete labels, is sufficient to perform model selection on state-of-the-art unsupervised models. Further, we investigate the benefit of incorporating supervision into the training process. Overall, we empirically validate that with little and imprecise supervision it is possible to reliably learn disentangled representations.

Subfields: Analytical Chemistry, Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Analytical Chemistry

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “Finding sample efficient disentanglement scores is an important research direction for practical semi-supervised disentanglement (Locatello et al., 2020b).”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “…representations with small weakly-supervised reconstruction loss (the sum of the first two terms in (6)) achieve improved downstream performance (Locatello et al., 2019b; 2020), improved downstream generalization (Peters et al., 2017) under covariate shifts (Shimodaira, 2000; Quionero-Candela…”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “Disentangled representations may be useful for (semi-)supervised learning of downstream tasks, transfer and few shot learning [1, 43, 34].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Following the successes in learning distributed representations that efficiently encode the content of high-dimensional sensory data [45, 56, 76], recent work has focused on learning representations that are disentangled [6, 69, 68, 73, 71, 26, 27, 42, 10, 63, 16, 52, 53, 48, 9, 51].”
Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness (2018, background): “…effects on encodings may give rise to the development of novel, robustly disentangled representation learning algorithms, for example in the interactive learning environment (Thomas et al., 2017) or when weak forms of supervision are available (Bouchacourt et al., 2017; Locatello et al., 2019).”
Is Independence all you need? On the Generalization of Representations Learned from Correlated Data (2020, methodology): “For this we use a variant of demographic parity Dwork et al. (2012) that computes pairwise mutual information between latents and FoV (Locatello et al., 2019a).”
CausalVAE: Disentangled Representation Learning via Neural Structural Causal Models (2020, unknown): “[19] use additional labels to reduce the model ambiguity.”
Learning Disentangled Representations for Recommendation (2019, methodology): “We thus encourage future efforts to focus more on (semi-)supervised methods [35].”
Learning joint latent representations based on information maximization (2021, background): “Supervised learning is based on the notion of latent variable predictability when labels are available [12].”
Low-Resource Knowledge-Grounded Dialogue Generation (2020, methodology): “State-of-the-art models are built within the framework of variational auto-encoding (Kingma & Welling, 2013) either under an unsupervised assumption (Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2016; 2018) or aided by a few labels (Narayanaswamy et al., 2017; Locatello et al., 2019).”