Disentangling the independently controllable factors of variation by interacting with the world

By Valentin Thomas, Emmanuel Bengio, William Fedus, Jules Pondard, Philippe Beaudoin, Hugo Larochelle, Joelle Pineau, Doina Precup, Yoshua Bengio. 2018

69 citations recorded by the source corpus.

It has been postulated that a good representation is one that disentangles the underlying explanatory factors of variation. However, it remains an open question what kind of training framework could potentially achieve that. Whereas most previous work focuses on the static setting (e.g., with images), we postulate that some of the causal factors could be discovered if the learner is allowed to interact with its environment. The agent can experiment with different actions and observe their effects. More specifically, we hypothesize that some of these factors correspond to aspects of the environment which are independently controllable, i.e., that there exists a policy and a learnable feature for each such aspect of the environment, such that this policy can yield changes in that feature with minimal changes to other features that explain the statistical variations in the observed data. We propose a specific objective function to find such factors, and verify experimentally that it can indeed disentangle independently controllable aspects of the environment without any extrinsic reward signal.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “Promising settings (that have been explored to some degree) seem to be for example (i) disentanglement learning with interactions (Thomas et al., 2017), (ii) when weak forms of supervision like grouping information are available (Bouchacourt et al., 2018; Shu et al., 2020; Hosoya, 2019; Locatello…”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “On the other hand, many data modalities are not observed as i.i.d. samples from a distribution (Dayan, 1993; Storck et al., 1995; Hochreiter & Schmidhuber, 1999; Bengio et al., 2013; Peters et al., 2017; Thomas et al., 2017; Schölkopf, 2019).”
Learning First-Order Symbolic Representations for Planning from the Structure of the State Space (2019, background): “Something similar occurs with work aimed at learning low-dimensional representations that disentangle the factors of variations in the data [35].”
On the Fairness of Disentangled Representations (2019, methodology): “Weaker forms of supervision like relational information or additional assumptions on the effect of the factors of variation were also studied [39, 16, 47, 31, 91, 26, 19, 41, 93, 62, 53, 81, 8] and applied in the sequential data and reinforcement learning settings [88, 85, 57, 69, 37, 38].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “Promising settings (that have been explored to some degree) seem to be for example (i) disentanglement learning with interactions [50], (ii) when weak forms of supervision e.”
Recurrent Neural Filters: Learning Independent Bayesian Filtering Steps for Time Series Prediction (2019, background): “Recent works in deep generative modelling have focused on the use of neural networks to learn independent factors of variation in static datasets – through the encouragement of disentangled representations [16], [17] or by learning causal mechanisms [18], [19].”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “…et al., 2018; Sukhbaatar et al., 2018; Dwiel et al., 2019; Nair & Finn, 2019; Nasiriany et al., 2019; Dilokthanakul et al., 2019) or by discovering reusable skills (Daniel et al., 2012; Gregor et al., 2016; Warde-Farley et al., 2018; Shankar & Gupta, 2020; Thomas et al., 2018; Sharma et al., 2020).”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “This principle, proposed in prior work that utilized purely model-free unsupervised RL methods (Daniel et al., 2012; Florensa et al., 2017; Eysenbach et al., 2018; Gregor et al., 2016; Warde-Farley et al., 2018; Thomas et al., 2018), aims to learn diverse skills via a discriminability objective: a g”
Curiosity Driven Exploration of Learned Disentangled Goal Spaces (2018, background): “However, identifying such independantly controllable features [27] is of paramount importance for agents to develop compact world models that generalize well, as well as to grow efficiently their repertoire of skills.”
Disentangling Factors of Variation Using Few Labels (2019, background): “These are not limiting assumption especially for sequential data or reinforcement learning [61, 59, 38, 46, 21, 22].”
Provable Rich Observation Reinforcement Learning with Combinatorial Latent States (2021, background): “There has also been several empirical works that either focus on the factored state space setting (e.g., Kim & Mnih (2018); Thomas et al. (2018); Laversanne-Finot et al. (2018); Miladinović et al. (2019)), or the factored action space setting (e.g., He et al. (2016); Sharma et al. (2017)).”
Contrastive Learning of Structured World Models (2019, background): “By establishing a connection between contrastive learning of state abstractions (François-Lavet et al., 2018; Thomas et al., 2018) and relational graph embeddings (Nickel et al.”