Recursive Routing Networks: Learning to Compose Modules for Language Understanding

By Ignacio Cases, Clemens Rosenbaum, Matthew Riemer, Atticus Geiger, Tim Klinger, Alex Tamkin, Olivia Li, S. Agarwal, Joshua D. Greene, Dan Jurafsky and 2 more. 2019

29 citations recorded by the source corpus.

We introduce Recursive Routing Networks (RRNs), which are modular, adaptable models that learn effectively in diverse environments. RRNs consist of a set of functions, typically organized into a grid, and a meta-learner decision-making component called the router. The model jointly optimizes the parameters of the functions and the meta-learner’s policy for routing inputs through those functions. RRNs can be incorporated into existing architectures in a number of ways; we explore adding them to word representation layers, recurrent network hidden layers, and classifier layers. Our evaluation task is natural language inference (NLI). Using the MultiNLI corpus, we show that an RRN’s routing decisions reflect the high-level genre structure of that corpus. To show that RRNs can learn to specialize to more fine-grained semantic distinctions, we introduce a new corpus of NLI examples involving implicative predicates, and show that the model components become fine-tuned to the inferential signatures that are characteristic of these predicates.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “More recent work (Devin et al., 2017; Frans et al., 2017; Fernando et al., 2017; Rosenbaum et al., 2017; Meyerson and Miikkulainen, 2017; Ramachandran and Le, 2018; Chang et al., 2018; Liang et al., 2018; Alet et al., 2018; Cases et al., 2019; Yang et al., 2020) has focused on training neural networ”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, methodology): “Although in practice this strategy has been shown to work in any case [Rosenbaum et al., 2017, Chang et al., 2019, Cases et al., 2019], it lacks theoretical justification, and more principled approaches may yield superior performance.”
On the Role of Weight Sharing During Deep Option Learning (2019, methodology): “…Riemer et al., 2015; Misra et al., 2016; Rusu et al., 2016; Shazeer et al., 2017; Fernando et al., 2017; Rosenbaum et al., 2018; Ruder et al., 2017; Ramachandran & Le, 2018; Cases et al., 2019) that directly addresses catastrophic forgetting, increasing representation stability during learning.”