Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning

By Clemens Rosenbaum, Tim Klinger, Matthew Riemer. 2017

277 citations recorded by the source corpus.

Multi-task learning (MTL) with neural networks leverages commonalities in tasks to improve performance, but often suffers from task interference which reduces the benefits of transfer. To address this issue we introduce the routing network paradigm, a novel neural network and training algorithm. A routing network is a kind of self-organizing neural network consisting of two components: a router and a set of one or more function blocks. A function block may be any neural network - for example a fully-connected or a convolutional layer. Given an input the router makes a routing decision, choosing a function block to apply and passing the output back to the router recursively, terminating when a fixed recursion depth is reached. In this way the routing network dynamically composes different function blocks for each input. We employ a collaborative multi-agent reinforcement learning (MARL) approach to jointly train the router and function blocks. We evaluate our model against cross-stitch networks and shared-layer baselines on multi-task settings of the MNIST, mini-imagenet, and CIFAR-100 datasets. Our experiments demonstrate a significant improvement in accuracy, with sharper convergence. In addition, routing networks have nearly constant per-task training cost while cross-stitch networks scale linearly with the number of tasks. On CIFAR-100 (20 tasks) we obtain cross-stitch performance levels with an 85% reduction in training time.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

A Review of Sparse Expert Models in Deep Learning (2022, neutral): “The problem of expert selection can be recast as a Bandit problem and several works have used reinforcement learning to learn the selection (Bengio et al., 2016; Rosenbaum et al., 2017; 2019; Clark et al., 2022).”
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “Existing sparse gates are not differentiable, and reinforcement learning algorithms are commonly used to train these gates [1, 29].”
Dynamic Neural Networks: A Survey (2021, methodology): “For another example, the idea of dynamic routing is implemented for: 1) reducing the training cost under a multi-task setting [269] and 2) finding the optimal fine-tuning strategy for per example in transfer learning [270].”
Recurrent Independent Mechanisms (2019, background): “Modularity and Neural Networks: A neural network is composed of several neural modules, where each module is meant to perform a distinct function, and hence can be seen as a combination of experts (Jacobs et al., 1991; Bottou & Gallinari, 1991; Ronco et al., 1997; Reed & De Freitas, 2015; Andreas et”
Continual Learning with Recursive Gradient Optimization (2022, background): “Expansionbased approaches select the network parameters dynamically (Yoon et al., 2018; Rosenbaum et al., 2018; Serra et al., 2018; Kaushik et al., 2021), add additional components as new tasks arrive (Rusu et al., 2016; Fernando et al., 2017; Alet et al., 2018; Chang et al., 2019; Li et al.,…”
Wide Neural Networks Forget Less Catastrophically (2021, background): “Expansion-based methods (Rusu et al., 2016; Fernando et al., 2017; Aljundi et al., 2017; Rosenbaum et al., 2018; Chang et al., 2018; Xu and Zhu, 2018; Ferran Alet, 2018; Li et al., 2019; Veniat et al., 2021) add new network modules or experts for new tasks.”
Model Zoo: A Growing Brain That Learns Continually (2021, methodology): “…has been recognized that this is not suﬃcient; this has given rise to methods for soft-parameter sharing that either design or learn specialized routing architectures (Rosenbaum et al., 2017; Sun et al., 2019; Fernando et al., 2017; Devin et al., 2017; Misra et al., 2016; Vandenhende et al., 2019).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “More recent work (Devin et al., 2017; Frans et al., 2017; Fernando et al., 2017; Rosenbaum et al., 2017; Meyerson and Miikkulainen, 2017; Ramachandran and Le, 2018; Chang et al., 2018; Liang et al., 2018; Alet et al., 2018; Cases et al., 2019; Yang et al., 2020) has focused on training neural networ”
Continual Learning in Low-rank Orthogonal Subspaces (2020, neutral): “Second, modular approaches use different network modules that can be extended for each new task [Fernando et al., 2017, Aljundi et al., 2017, Rosenbaum et al., 2018, Chang et al., 2018, Xu and Zhu, 2018, Ferran Alet, 2018].”
Meta-Consolidation for Continual Learning (2020, background): “This opened up an interesting research direction, in exploiting the meta-manifold of model parameters for multi-task learning [67], neural architecture search [8], zero-shot learning [58, 82] and style transfer [73] in recent years.”
Routing Networks with Co-training for Continual Learning (2020, background): “Continual learning is inherently multi-task, routing networks have been shown to outperform densely connected networks for multi-task learning when the data is not presented continually (Rosenbaum et al., 2017; Maziarz et al., 2019).”
Using Hindsight to Anchor Past Knowledge in Continual Learning (2019, methodology): “Second, modular approaches use different parts of the prediction function for each new task (Fernando et al., 2017; Aljundi et al., 2017; Rosenbaum et al., 2018; Chang et al., 2018; Xu & Zhu, 2018; Ferran Alet, 2018).”
Efficient Lifelong Learning with A-GEM (2018, methodology): “One approach to LLL uses modular compositional models (Fer-nando et al., 2017; Aljundi et al., 2017; Rosenbaum et al., 2018; Chang et al., 2018; Xu & Zhu, 2018; Ferran Alet, 2018), which limit interference among tasks by using different subset of modules for each task.”
Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference (2018, methodology): “In this section we would like to generalize our interpretation of a large set of different weight sharing schemes including (Riemer et al., 2015; Bengio et al., 2015; Rosenbaum et al., 2018; Serrà et al., 2018) and how the concept of weight sharing impacts the dynamics of transfer (equation 1) and i”
Gradient Surgery for Multi-Task Learning (2020, methodology): “A number of architectural solutions have been proposed to the multi-task learning problem based on multiple modules or paths [19, 14, 40, 51, 46, 57, 46], or using attention-based architectures [33, 37].”
Automatically Composing Representation Transformations as a Means for Generalization (2018, methodology): “We compare our approach to Routing Networks (RN) [67], which we can implement as an ablation of our method by ﬁxing the number of computation steps and using a different controller per target language and per computation step.”