Learning Independent Causal Mechanisms

By Giambattista Parascandolo, Mateo Rojas-Carulla, Niki Kilbertus, Bernhard Schölkopf. 2017

205 citations recorded by the source corpus.

Statistical learning relies upon data sampled from a distribution, and we usually do not care what actually generated it in the first place. From the point of view of causal modeling, the structure of each distribution is induced by physical mechanisms that give rise to dependences between observables. Mechanisms, however, can be meaningful autonomous modules of generative models that make sense beyond a particular entailed data distribution, lending themselves to transfer between problems. We develop an algorithm to recover a set of independent (inverse) mechanisms from a set of transformed data points. The approach is unsupervised and based on a set of experts that compete for data generated by the mechanisms, driving specialization. We analyze the proposed method in a series of experiments on image data. Each expert learns to map a subset of the transformed data back to a reference distribution. The learned mechanisms generalize to novel domains. We discuss implications for transfer learning and links to recent trends in generative modeling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Causal Inference & Discovery

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, methodology): “For CLEVR (with masks), we perform a center-crop with boundaries [29, 221] (width) and [64, 256] (height), as done in [16], and afterwards resize the cropped images to a resolution of 128× 128.”
Toward Causal Representation Learning (2021, methodology): “The result may be a modular architecture, where the different modules can be individually ﬁne-tuned and repurposed for new tasks [85], [180], and the SMS hypothesis can be used to enforce the appropriate structure.”
Towards Causal Representation Learning (2021, background): “[180] and competitive training can play a role in this.”
Learning explanations that are hard to vary (2020, background): “17 Causal models and robust regression Based on this insight, it was proposed that regression based on causal features should presents desirable invariance and robustness properties (Mooij et al., 2009; Schölkopf et al., 2012; Peters et al., 2016; Rojas-Carulla et al., 2018; Heinze-Deml et al., 2018”
Towards causal generative scene models via competition of experts (2020, background): “During inference, this process is reversed: at each step, experts compete for explaining part of the remaining scene, and only the winning expert is further trained on the explained part (Parascandolo et al., 2018).”
A theory of independent mechanisms for extrapolation in generative models (2020, background): “In order to characterize and enforce such causal disentanglement properties, the IM principle has been exploited in empirical studies (Goyal et al. 2019; Parascandolo et al. 2018) and its superiority to statistical independence has been emphasized (Besserve et al. 2020; Locatello et al. 2018).”
Causality for Machine Learning (2019, background): “…look for independent causal mechanisms (Locatello et al., 2018b), and competitive training can play a role in this: for a pattern recognition task, Parascandolo et al. (2018) show that learning causal models that contain independent mechanisms helps in transferring modules across substantially…”
Recurrent Independent Mechanisms (2019, methodology): “…learned this way may be more likely to capture the compositional generative (or causal) structure of the world, and thus better generalize across tasks where a (small) subset of mechanisms change while most of them remain invariant (Simon, 1991; Peters et al., 2017; Parascandolo et al., 2018).”
Semi-supervised learning, causality, and the conditional cluster assumption (2019, background): “In this work [25], the functions f0, f1 are generative models competing for data that has undergone unknown transformations, and eventually each specializing on how to invert one of those transformations.”
A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms (2019, background): “Parascandolo et al. (2017) propose an algorithm to recover a set of independent causal mechanisms by establishing competition between mechanisms, hence driving specialization.”
Counterfactuals uncover the modular structure of deep generative models (2018, background): “This amounts to saying that the transformations follow the causal principle of independent mechanisms (Peters et al., 2017; Parascandolo et al., 2018).”
Continual Learning via Local Module Composition (2021, background): “[74], who propose a way to learn a set of causal independent mechanisms as mixture-of-experts.”
Interventional Few-Shot Learning (2020, background): “, Y → X , where the assumption is that labels Y are disentangled enough to be treated as Independent Mechanism (IM) [43, 55], which generates observed images X through Y → X .”
Competitive Training of Mixtures of Independent Deep Generative Models (2018, background): “inference and proposed a semi-distance based on algorithmic complexity estimates. Our framework is related to competition of experts, which has been used to invert independent causal mechanisms, see [Parascandolo et al., 2018]. Mixtures of experts with a gating network trained with EM are a classica”
CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting (2022, background): “Secondly, by the independent mechanisms assumption (Peters et al., 2017; Parascandolo et al., 2018), we can see that the seasonal and trend modules do not influence or inform each other.”
Deep Structural Causal Models for Tractable Counterfactual Inference (2020, background): “For example, research has studied the use of causality for disentanglement [6, 7], causal discovery [8, 9], and for deriving causality-inspired explanations [10, 11] or data augmentations [12].”
Is Independence all you need? On the Generalization of Representations Learned from Correlated Data (2020, background): “The higher-level factors of this representation are in fact correlated and what we actually want to infer are independent (causal) mechanisms (Peters et al., 2017; Suter et al., 2019; Goyal et al., 2019; Parascandolo et al., 2018).”
Learning Compositional Neural Programs with Recursive Tree Search and Planning (2019, background): “Modularity is a powerful inductive bias for achieving this goal with neural networks [Parascandolo et al., 2018, Bengio et al., 2019].”