A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms

By Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Nan Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, C. Pal. 2019

379 citations recorded by the source corpus.

We propose to meta-learn causal structures based on how fast a learner adapts to new distributions arising from sparse distributional changes, e.g. due to interventions, actions of agents and other sources of non-stationarities. We show that under this assumption, the correct causal structural choices lead to faster adaptation to modified distributions because the changes are concentrated in one or just a few mechanisms when the learned knowledge is modularized appropriately. This leads to sparse expected gradients and a lower effective number of degrees of freedom needing to be relearned while adapting to the change. It motivates using the speed of adaptation to a modified distribution as a meta-learning objective. We demonstrate how this can be used to determine the cause-effect relationship between two observed variables. The distributional changes do not need to correspond to standard interventions (clamping a variable), and the learner has no direct knowledge of these interventions. We show that causal structures can be parameterized via continuous variables and learned end-to-end. We then explore how these ideas could be used to also learn an encoder that would map low-level observed variables to unobserved causal variables leading to faster adaptation out-of-distribution, learning a representation space where one can satisfy the assumptions of independent mechanisms and of small and sparse changes in these mechanisms due to actions and non-stationarities.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Shortcut learning in deep neural networks (2020, background): “This ability is connected to the identification of causal graphs [138] since learning causal features allows for small changes when changing environments.”
The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, methodology): “A key weakness, as Yoshua Bengio put it in a recent article (Bengio et al., 2019), is that Current machine learning methods seem weak when they are required to generalize beyond the training distribution, which is what is often needed in practice.”
Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, methodology): “Bengio et al. (2020) suggests using adaptation speed as a heuristic objective to disentangle latent factors and their causal relationship in the bivariate case.”
Differentiable Causal Discovery from Interventional Data (2020, background): “[1] leveraged interventions and continuous optimization to learn the causal direction in the bivariate setting.”
An Analysis of the Adaptation Speed of Causal Models (2020, methodology): “This is the choice made in Bengio et al. (2020), as well as Chalupka et al. (2016). The latter work reports that distributions sampled with from this prior exhibit some asymmetry between X and Y .”
Toward Causal Representation Learning (2021, methodology): “While Bengio et al. [25] and Ke et al. [131] focused on learning a causal model using neural networks with an unsupervised loss, the work of Dasgupta et al. [51] explores learning a causal model using an RL agent.”
Towards Causal Representation Learning (2021, methodology): “While [24, 131] focus on learning a causal model using neural networks with an unsupervised loss, the work of Dasgupta et al.”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “In this paper, we consider learning disentangled representations from pairs of observations which differ by a few factors of variation (Bengio, 2017; Schmidt et al., 2007; Bengio et al., 2019) as in Figure 1.”
Causality for Machine Learning (2019, background): “Our opinions have since then converged, and research has started to appear that combines both fields (Bengio et al., 2019; Goyal et al., 2019; Locatello et al., 2018a; Suter et al., 2018; Goudet et al., 2017).”
Learning Neural Causal Models from Unknown Interventions (2019, methodology): “Bengio et al. (2019) propose to learn them by using observations of how the distribution changes sparsely, with the transfer generalization (the adaptation rate after the sparse change) being the training objective for γ .”
Recurrent Independent Mechanisms (2019, background): “ can specialize on simple sub-problems, similar to Parascandolo et al. (2018). This specialization and modularization not only has computational and statistical advantages (Baum & Haussler, 1989; Bengio et al., 2019), but also prevents individual RIMs from dominating and modelling complex, compo”
Invariant Risk Minimization (2019, background): “Some works in machine learning [45, 18, 21, 26, 36, 43, 34, 7] pursue similar questions.”
Meta-Learning Representations for Continual Learning (2019, methodology): “After learning θ , we learn g W from R d → Y for a CLP problem from a single trajectory S using fully online SGD updates in a single pass. similar ( A idea has been proposed by Bengio et al. 2019) for learning causal structures.”
Inductive biases for deep learning of higher-level cognition (2020, methodology): “eﬁned on the appropriate causal variables would be more eﬃciently learned than one deﬁned on the wrong representation. Preliminary work based on meta-learning (Ke et al., 2019; Dasgupta et al., 2019; Bengio et al., 2019) suggests that, parameterizing the correct variables and causal structures, the ”
A large-scale benchmark for few-shot program induction and synthesis (2021, methodology): “To improve this, more compositional methods to meta-learning are increasingly being proposed (Alet et al., 2018; Bengio et al., 2019; Ke et al., 2019; Mendez & Eaton, 2020; Ruis et al., 2020).”
Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers (2019, background): “More generally, there has been increasing work to incorporate causal knowledge in machine learning, such as by causally invariant representations [3] or modelling causal knowledge for generalization [4].”
Disentangling User Interest and Conformity for Recommendation with Causal Embedding (2021, background): “Learning representations in which different semantics are disentangled is crucial for robust use of neural models [6, 32, 41, 44].”
Learning Compositional Neural Programs with Recursive Tree Search and Planning (2019, background): “As argued in Bengio et al. [2019], beyond achieving good generalization when both the training and test data come from the same distribution, we want knowledge acquired in one setting to transfer to other settings with different but possibly related distributions.”