Routing Networks and the Challenges of Modular and Compositional Computation

By Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, Tim Klinger. 2019

93 citations recorded by the source corpus.

Compositionality is a key strategy for addressing combinatorial complexity and the curse of dimensionality. Recent work has shown that compositional solutions can be learned and offer substantial gains across a variety of domains, including multi-task learning, language modeling, visual question answering, machine comprehension, and others. However, such models present unique challenges during training when both the module parameters and their composition must be learned jointly. In this paper, we identify several of these issues and analyze their underlying causes. Our discussion focuses on routing networks, a general approach to this problem, and examines empirically the interplay of these challenges and a variety of design decisions. In particular, we consider the effect of how the algorithm decides on module composition, how the algorithm updates the modules, and if the algorithm uses regularization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

In Neural Architecture Search & AutoML

Citations in the corpus

Continual Learning via Local Module Composition (2021, methodology): “Modularity in neural networks is studied in the context of scalability [9], and more recently as a way to achieve compositionality and systematic generalization [6, 47, 15, 8, 27, 19] as well as for multi-task learning [65, 82].”
Sharing Less is More: Lifelong Learning in Deep Networks with Selective Layer Transfer (2020, unknown): “…challenge of training modular networks is the interdependence of searching for the optimal compositional structure and learning the optimal modules (Rosenbaum et al., 2019), so algorithms such as meta-learning (Alet et al., 2018), expectation-maximization (Kirsch et al., 2018) and reinforcement…”
Neural Production Systems (2021, background): “…et al., 1991; Bottou & Gallinari, 1991; Ronco et al., 1997; Reed & De Freitas, 2015; Andreas et al., 2016; Rosenbaum et al., 2017; Fernando et al., 2017; Shazeer et al., 2017; Kirsch et al., 2018; Rosenbaum et al., 2019; Lamb et al., 2020) routing information through a gated activation of modules.”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “One popular approach to multi-task RL is to employ hierarchical [23], [47] or modular neural networks [31], [48], [2].”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “Rosenbaum et al. (2019) discussed the challenges of optimizing modular architectures via an extensive evaluation with and without task indicators on vision and language tasks.”
Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Experts (2021, background): “Mixtures of experts and the more general routing or modular networks [37, 19, 34], sometimes treat expert assignments as latent variables and use EM or variational methods [19] for training them.”
Multi-Task Reinforcement Learning with Soft Modularization (2020, methodology): “al [30, 29] proposes to use RL to learn a routing policy, which can be considered as a hard version of our soft modularization method.”
Scaling Vision with Sparse Mixture of Experts (2021, background): “During training, sparse models may favor only a small set of experts [26, 52].”