Fast and Slow Learning of Recurrent Independent Mechanisms

By Kanika Madan, R. Ke, Anirudh Goyal, Bernhard Schölkopf, Yoshua Bengio. 2021

44 citations recorded by the source corpus.

Decomposing knowledge into interchangeable pieces promises a generalization advantage when there are changes in distribution. A learning agent interacting with its environment is likely to be faced with situations requiring novel combinations of existing pieces of knowledge. We hypothesize that such a decomposition of knowledge is particularly relevant for being able to generalize in a systematic manner to out-of-distribution changes. To study these ideas, we propose a particular training framework in which we assume that the pieces of knowledge an agent needs and its reward function are stationary and can be re-used across tasks. An attention mechanism dynamically selects which modules can be adapted to the current task, and the parameters of the selected modules are allowed to change quickly as the learner is confronted with variations in what it experiences, while the parameters of the attention mechanisms act as stable, slowly changing, meta-parameters. We focus on pieces of knowledge captured by an ensemble of modules sparsely communicating with each other via a bottleneck of attention. We find that meta-learning the modular aspects of the proposed system greatly helps in achieving faster adaptation in a reinforcement learning setup involving navigation in a partially observed grid world with image-level input. We also find that reversing the role of parameters and meta-parameters does not work nearly as well, suggesting a particular role for fast adaptation of the dynamically selected modules.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA (2021, background): “The assumption that high-level variables are sparsely related to one another has been leveraged also by Goyal et al. (2021b,a); Madan et al. (2021) via attention mechanisms.”
Continual Learning via Local Module Composition (2021, background): “Building up on this ideas, others show evidence of compositional OOD generalization [61].”
Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domains (2021, methodology): “Previous work [41, 42, 51] used Proximal Policy Optimization (PPO) [52] to solve the MiniGrid problems.”
History Compression via Language Models in Reinforcement Learning (2022, methodology): “In accordance with recent work (Jiang et al., 2021; Goyal et al., 2021; Madan et al., 2021), we use PPO as on-policy algorithm for our Minigrid experiments.”
Learning where to learn: Gradient sparsity in meta and continual learning (2021, result): “This finding complements recent work showing that modular recurrent networks with sparse updating mechanisms hold great promise in improving out-of-distribution generalization performance [14, 38].”