Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations

By Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, Olivier Bachem. 2018

1,814 citations recorded by the source corpus.

The key idea behind the unsupervised learning of disentangled representations is that real-world data is generated by a few explanatory factors of variation which can be recovered by unsupervised learning algorithms. In this paper, we provide a sober look at recent progress in the field and challenge some common assumptions. We first theoretically show that the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data. Then, we train more than 12000 models covering most prominent methods and evaluation metrics in a reproducible large-scale experimental study on seven different data sets. We observe that while the different methods successfully enforce properties ``encouraged'' by the corresponding losses, well-disentangled models seemingly cannot be identified without supervision. Furthermore, increased disentanglement does not seem to lead to a decreased sample complexity of learning for downstream tasks. Our results suggest that future work on disentanglement learning should be explicit about the role of inductive biases and (implicit) supervision, investigate concrete benefits of enforcing disentanglement of the learned representations, and consider a reproducible experimental setup covering several data sets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Other intrinsic scores have been proposed to measure the disentanglement of representations (Locatello et al., 2019), or the mutual information between the input and representations (Hjelm et al., 2018).”
Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, positive): “In the quest to draw sound and robust conclusions, we join other recent papers in highlighting the under-acknowledged existence of conclusion replication failure [47, 36, 39, 26, 19].”
Diagnosing and Enhancing VAE Models (2019, background): “It was our original intent to include a thorough presentation of these results; however, in the process of preparing this manuscript we became aware of a contemporary work with extensive testing in this area (Locatello et al., 2018).”
Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, background): “…machine learning setting, hyperparameter selection for disentanglement cannot be performed simply by evaluating goodness of fit on a validation set and selecting the highest scoring model since there is usually a trade-off between goodness of fit and disentanglement (Locatello et al., 2019, Sec.”
Toward Causal Representation Learning (2021, background): “case as there are infinitely many equivalent solutions yielding the same observational distribution [117], [158], [188].”
Flexibly Fair Representation Learning by Disentanglement (2019, background): “…literature, where the goal is for each dimension of the representation (also called the “latent code”) to correspond to no more than one semantic factor of variation in the data (for example, independent visual features like object shape and position) (Higgins et al., 2017; Locatello et al., 2019).”
Towards Causal Representation Learning (2021, background): “help favoring certain solutions over others [158, 205].”
On the Transfer of Disentangled Representations in Realistic Settings (2020, methodology): “In particular, using the architecture from Locatello et al. (2019b), none of the models we trained correctly captured all factors of variation or yielded high-quality reconstructions.”
Towards causal generative scene models via competition of experts (2020, background): “…(Bengio et al., 2013) has largely focused on disentangling factors of variation within a single shared representation space, e.g., by training a large encoder-decoder architecture with different forms of regularization (Higgins et al.; Kim & Mnih, 2018; Chen et al., 2018; Locatello et al., 2019).”
A theory of independent mechanisms for extrapolation in generative models (2020, methodology): “2018) and its superiority to statistical independence has been emphasized (Besserve et al. 2020; Locatello et al. 2018).”
Weakly-Supervised Disentanglement Without Compromises (2020, methodology): “Further, we show that these representations are useful on a diverse suite of downstream tasks, including a novel experiment targeting strong generalization under covariate shifts, fairness (Locatello et al., 2019a) and abstract visual reasoning (van Steenkiste et al., 2019).”
Causality for Machine Learning (2019, background): “A sensible inductive bias to learn such models is to look for independent causal mechanisms (Locatello et al., 2018b), and competitive training can play a role in this: for a pattern recognition task, Parascandolo et al. (2018) show that learning causal models that contain independent mechanisms hel”
Weakly Supervised Disentanglement with Guarantees (2019, methodology): “This challenge was ﬁrst put forth by Locatello et al. (2019), who noted that unsupervised disentanglement is heavily reliant on model inductive bias.”
On the Fairness of Disentangled Representations (2019, methodology): “For the prediction, we consider the same gradient boosting classifier [27] as in [61] which was trained on 10 000 labeled examples (subsequently denoted by GBT10000) and which achieves higher accuracy than the crossvalidated logistic regression.”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, methodology): “In prior work it was found that there is a positive correlation between disentanglement metrics and down-stream performance on single factor classification [52].”
Multi-Object Representation Learning with Iterative Variational Inference (2019, background): “While its precise definition is still highly debated (Higgins et al., 2018; Eastwood & Williams, 2018; Ridgeway & Mozer, 2018; Locatello et al., 2018), the concept of disentanglement has generated a lot of interest recently.”
A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms (2019, background): “As stated already by Bengio et al. (2013) and clearly demonstrated by Locatello et al. (2018), assumptions, priors or biases are necessary to identify the underlying explanatory For the former, we obtain θ = 0 corresponding to the correct causal graph U → V ; and for the latter, θ = π 2 ,…”
MONet: Unsupervised Scene Decomposition and Representation (2019, background): “Recent progress has been made on learning object representations that support feature compositionality, for which a variety of VAE-based methods have become state-of-the-art [Higgins et al., 2017, Kim and Mnih, 2017, Chen et al., 2016, Locatello et al., 2018].”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, methodology): “Under the current disentan-glement metrics [17, 21, 6, 29], VAE-based architectures ( β -VAE [17], TCVAE [6], FactorVAE [21]) dominate the benchmarks, leaving behind other approaches such as Info-GAN [7] and DCIGN [25].”