Recurrent Independent Mechanisms

By Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, Bernhard Schölkopf. 2019

369 citations recorded by the source corpus.

Learning modular structures which reflect the dynamics of the environment can lead to better generalization and robustness to changes which only affect a few of the underlying causes. We propose Recurrent Independent Mechanisms (RIMs), a new recurrent architecture in which multiple groups of recurrent cells operate with nearly independent transition dynamics, communicate only sparingly through the bottleneck of attention, and are only updated at time steps where they are most relevant. We show that this leads to specialization amongst the RIMs, which in turn allows for dramatically improved generalization on tasks where some factors of variation differ systematically between training and evaluation.

Subfields: Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, methodology): “…of deep learning, including developing techniques for statistically extracting causal relationships through a sensitivity to distributional changes (Bengio et al., 2019) and techniques for automatically extracting modular structure (Goyal et al., 2019), both of which I am quite sympathetic to.”
Object-Centric Learning with Slot Attention (2020, background): “[66] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf.”
Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, background): “The hypothesis that high-level concepts can be described by a sparse dependency graph has been described and leveraged for out-of-distribution generalization originally by Bengio (2019) and Goyal et al. (2021b), which were early sources of inspiration for this work.”
Toward Causal Representation Learning (2021, methodology): “[85] suggested that learning causal models that contain independent mechanisms may help in transferring modules across substantially different domains.”
Towards Causal Representation Learning (2021, methodology): “to adapt a few modules in its internal representation of the world [220, 84].”
CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, background): “…has been historically studied from the perspective of probabilistic and causal reasoning (Pearl, 2009), cognitive psychology (Grifﬁths & Tenenbaum, 2005), and more recently in the context of machine learning (Goyal et al., 2019b; Sch ¨ olkopf, 2019; Baradel et al., 2019; Bakhtin et al., 2019).”
Towards causal generative scene models via competition of experts (2020, methodology): “…mixtures of generative models (Locatello et al., 2018), as well as to dynamical systems via sparsely-interacting recurrent independent mechanisms (Goyal et al., 2019). and a second model which uses a softmax combination akin to the spatial mixture models used in IODINE and GENESIS , although the…”
A theory of independent mechanisms for extrapolation in generative models (2020, background): “In order to characterize and enforce such causal disentanglement properties, the IM principle has been exploited in empirical studies (Goyal et al. 2019; Parascandolo et al. 2018) and its superiority to statistical independence has been emphasized (Besserve et al. 2020; Locatello et al. 2018).”
Causality for Machine Learning (2019, background): “This has recently been taken to the next step, embedding a set of dynamic modules into a recurrent neural network, coordinated by a so-called attention mechanism (Goyal et al., 2019).”
Continual Learning via Local Module Composition (2021, background): “Modularity in neural networks is studied in the context of scalability [9], and more recently as a way to achieve compositionality and systematic generalization [6, 47, 15, 8, 27, 19] as well as for multi-task learning [65, 82].”
Few-Shot and Continual Learning with Attentive Independent Mechanisms (2021, background): “AIM can be viewed as a static version of RIMs [16], i.e. temporal modeling of hidden states using LSTM [18] found in RIMs is removed.”
Coordination Among Neural Modules Through a Shared Global Workspace (2021, methodology): “The proposed model can also be seen as integrating out different ideas popular in modular architectures (Andreas et al., 2016; Goyal et al., 2019), memory networks (Graves et al., 2014; Santoro et al., 2018) and mixture of experts (Jacobs et al., 1991), and hence combining some of their beneﬁts in…”
Inductive biases for deep learning of higher-level cognition (2020, background): “Any change in such a distribution (e.g., when moving from one setting/domain to a related one) will always be due to changes in at least one of those mechanisms (Goyal et al., 2019; Bengio et al., 2019; Ke et al., 2019).”
Neuroevolution of self-interpretable agents (2020, background): “[3, 32, 45, 46, 52, 58, 70, 87, 102, 106]) that approach the generalization problem, and future work will continue to develop on these ideas to push the generalization abilities proposed in more general domains [7, 16, 17, 49, 59, 70, 105].”
Context-Specific Representation Abstraction for Deep Option Learning (2021, methodology): “While the CRADOL framework is certainly compatible with this kind of explicit belief state modeling, we have chosen to implement the belief state implicitly and denote it as the factored belief state in order to have a fair empirical comparison to the RIMs method (Goyal et al. 2021) that we build of”
A Practical Survey on Faster and Lighter Transformers (2021, background): “A recent structural inductive bias inspired by independent mechanisms in the causality literature consists of designing an architecture that learns sparsely interacting modules, each one of them specialized in a different mechanism [37].”