Modular Networks: Learning to Decompose Neural Computation

By Louis Kirsch, Julius Kunze, David Barber. 2018

127 citations recorded by the source corpus.

Scaling model capacity has been vital in the success of deep learning. For a typical network, necessary compute resources and training time grow dramatically with model size. Conditional computation is a promising way to increase the number of parameters with a relatively small increase in resources. We propose a training algorithm that flexibly chooses neural modules based on the data to be processed. Both the decomposition and modules are learned end-to-end. In contrast to existing approaches, training does not rely on regularization to enforce diversity in module use. We apply modular networks both to image recognition and language modeling tasks, where we achieve superior performance compared to several baselines. Introspection reveals that modules specialize in interpretable contexts.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems (2020, background): “ule network [Jacobs et al., 1991, Bottou and Gallinari, 1991, Ronco et al., 1997, Reed and De Freitas, 2015, Andreas et al., 2016, Rosenbaum et al., 2017, Fernando et al., 2017, Shazeer et al., 2017, Kirsch et al., 2018, Rosenbaum et al., 2019] has an architecture which is composed dynamically from ”
Recurrent Independent Mechanisms (2019, background): “Modularity and Neural Networks: A neural network is composed of several neural modules, where each module is meant to perform a distinct function, and hence can be seen as a combination of experts (Jacobs et al., 1991; Bottou & Gallinari, 1991; Ronco et al., 1997; Reed & De Freitas, 2015; Andreas et”
Continual Learning via Local Module Composition (2021, methodology): “Note that in the case of global structural objective, hard module selection would require applying tools for non-differentiable learning such as Expectation Maximization [47] or reinforcement-learning based methods [82, 5].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, neutral): “…Fernando, Banarse, Blundell, Zwols, Ha, Rusu, Pritzel, and Wierstra, 2017; Rosenbaum, Klinger, and Riemer, 2017; Meyerson and Miikkulainen, 2017; Kirsch, Kunze, and Barber, 2018; Ramachandran and Le, 2018; Chang, Gupta, Levine, and Griffiths, 2018; Liang, Meyerson, and Miikkulainen, 2018; Alet,…”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “RL can be viewed as a recurrent mixture of experts (Jacobs et al., 1991), where each expert is a module, similar to other recent and contemporaneous work (Hinton et al., 2018; Rosenbaum et al., 2018; Kirsch et al., 2018; Fernando et al., 2017) that route through a choices of layers of a ﬁxed-depth a”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, methodology): “There has been some progress on each of these problems (for example, module collapse was discussed in [Kirsch et al., 2018]).”
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021, background): “Various modular network architectures have been proposed to exploit compositionality (Rosenbaum et al., 2018; Kirsch et al., 2018).”
Lifelong Learning of Compositional Structures (2020, background): “…include treating the component selection as a reinforcement learning [26, 6, 11, 22], learning it via stochastic search [7, 2], or deﬁning it as a differentiable gating network [28], probabilistic model [13], or softmax selector [19] and training it jointly with the components via backpropagation.”
Recursive Routing Networks: Learning to Compose Modules for Language Understanding (2019, methodology): “lizes curriculum learning to solve arithmetic and vision problems; and ModularNetworks (Kirsch et al., 2018), which extend Routing Networks with an EM-like training approach.”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, background): “The architecture of (Kirsch et al., 2018) is similarly inspired to Routing Networks and the CRL, but takes a local view of routing rather than a global one.”
Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Experts (2021, background): “Mixtures of experts and the more general routing or modular networks [37, 19, 34], sometimes treat expert assignments as latent variables and use EM or variational methods [19] for training them.”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “Recently, an increasing number of works have focused on the problem of learning compositional chunks of knowledge to share across different tasks (Andreas et al., 2016; Hu et al., 2017; Kirsch et al., 2018; Meyerson and Miikkulainen, 2018).”
Neural Production Systems (2021, background): “A network can be composed of several modules, each meant to perform a distinct function, and hence can be seen as a combination of experts (Jacobs et al., 1991; Bottou & Gallinari, 1991; Ronco et al., 1997; Reed & De Freitas, 2015; Andreas et al., 2016; Rosenbaum et al., 2017; Fernando et al., 2017;”
Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks (2020, background): “At ﬁrst glance, explicitly modular networks (Clune et al., 2013; Andreas et al., 2016; Kirsch et al., 2018; Chang et al., 2019; Bahdanau et al., 2019) could provide a solution for the discovered problems.”
Sharing Less is More: Lifelong Learning in Deep Networks with Selective Layer Transfer (2020, methodology): “, 2018), expectationmaximization (Kirsch et al., 2018) and reinforcement learning (Rosenbaum et al.”
A systematic review on overfitting control in shallow and deep neural networks (2021, background): “As an instance, (Kirsch et al. 2018), proposes a modularization scheme to design a network with high performance.”
Fast and Slow Learning of Recurrent Independent Mechanisms (2021, methodology): “The generalization capabilities of neural models may benefit from structure in the design of models, to make them resemble the kind of rules they are supposed to learn (Andreas et al., 2016; Johnson et al., 2017; Jacobs et al., 1991; Bottou & Gallinari, 1991; Ronco et al., 1997; Reed & De Freitas, 2”