Latent Multi-Task Architecture Learning

By Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, Anders Søgaard. 2017

414 citations recorded by the source corpus.

Multi-task learning (MTL) allows deep neural networks to learn from related tasks by sharing parameters with other networks. In practice, however, MTL involves searching an enormous space of possible parameter sharing architectures to find (a) the layers or subspaces that benefit from sharing, (b) the appropriate amount of sharing, and (c) the appropriate relative weights of the different task losses. Recent work has addressed each of the above problems in isolation. In this work we present an approach that learns a latent multi-task architecture that jointly addresses (a)–(c). We present experiments on synthetic data and data from OntoNotes 5.0, including four different tasks and seven different domains. Our extension consistently outperforms previous approaches to learning latent architectures for multi-task problems and achieves up to 15% average error reductions over common approaches to MTL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “More recent work has explored using labels from core NLP tasks to supervise training of lower levels of deep neural networks (Søgaard and Goldberg, 2016; Hashimoto et al., 2016) and automatically learning cross-task sharing mechanisms for multi-task learning (Ruder et al., 2017).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “While “learning to share” approaches (Kang et al., 2011; Ruder et al., 2017; Platanios et al., 2018) are promising alternatives, a more straightforward solution could be implicitly increasing the per-task capacity by increasing over-all model capacity.”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “…recent work for NLP has centered on neural architecture design: e.g., ensuring only beneﬁcial information is shared across tasks (Liu et al., 2017; Ruder et al., 2019) or arranging tasks in linguistically-motivated hierarchies (Søgaard and Goldberg, 2016; Hashimoto et al., 2017; Sanh et al.,…”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, background): “In this section, we formalize our goal of transfer learning (Pan and Yang, 2010; Torrey and Shavlik, 2010; Ruder, 2019), highlight its key challenges, and provide a brief overview of common methods that can be used to address them.”
Continual and Multi-Task Architecture Search (2019, positive): “…is primarily used to improve the generalization performance of a task by leveraging knowledge from related tasks (Caru-ana, 1998; Collobert and Weston, 2008; Girshick, 2015; Luong et al., 2015; Ruder et al., 2017; Au-genstein et al., 2018; Guo et al., 2018; Oh et al., 2017; Ruder and Plank, 2017).”
Efficient Continual Learning with Modular Networks and Task-Driven Priors (2020, background): “There is also a body of literature on modular networks for multi-task and multi-domain learning (Ruder et al., 2019; Rebufﬁ et al., 2017; Zhao et al., 2020).”
Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, background): “Routing networks share a common goal with techniques for automated selective transfer learning using attention (Rajendran et al., 2017) and learning gating mechanisms between representations (Stollenga et al., 2014), (Misra et al., 2016), (Ruder et al., 2017).”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, background): “…soft parameter sharing models, have been considered in the multi-task and lifelong learning setting [Stollenga et al., 2014, Aljundi et al., 2017, Misra et al., 2016, Ruder et al., 2017, Rajendran et al., 2017], but do not allow for nearly the level of specialization as those based on routing.”
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks (2018, methodology): “We selected the simple yet effective training method described in (Søgaard and Goldberg 2016; Ruder et al. 2017): after each parameter update, a task is randomly selected and a batch of the dataset attached to this task is also sampled at random to train the model.”
Visual Reasoning by Progressive Module Networks (2018, methodology): “The process of training a machine learning model to be able to solve multiple tasks, or multi-task learning (MTL), has been widely studied (Long et al., 2017; Ruder, 2017; Ruder et al., 2017; Rusu et al., 2016).”
Generating Fact Checking Explanations (2020, background): “…) takes the contextual embeddings c E and c F produced by the last layer of DistilBERT and feeds them into a cross-stitch layer (Misra et al., 2016; Ruder et al., 2019), which consists of two layers with two shared subspaces each - h 1 E and h 2 E for the explanation task and h 1 F and h 2 F for…”
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (2019, methodology): “Finally, we compare our method with state-of-the-art multi-task learning approaches, including Cross-Stitch Networks [39] (CVPR’16), Sluice Networks [48] (AAAI’19), and NDDR-CNN [16] (CVPR’19), which adopt several feature fusion layers between task-specific backbones, MTAN [33] (CVPR’19), which intr”
Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (2021, background): “Ruder et al. (2019) propose the sluice network that learns the inter-task (shared) sub-spaces on top of task-speciﬁc models.”
Branched Multi-Task Networks: Deciding what layers to share (2019, methodology): “ask talk. Cross-stitch networks (Misra et al., 2016) softly share their features among tasks, by using a linear combination of the activations found in multiple single task networks. Sluice networks (Ruder et al., 2019) extend cross-stitch networks and allow to learn the selective sharing of layers,”
Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, methodology): “This was followed by a number of recent works, which further analyzed task relationships [1, 5, 24, 17, 23, 22] for transfer learning.”
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021, background): “This approach has seen success in a variety of domains such as classification, recommender systems and NLP ((Ma et al., 2019, 2018; Clark et al., 2019; Collobert and Weston, 2008; Ruder et al., 2019; Tan et al., 2019)).”
AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning (2019, methodology): “Multi-task learning has been applied to a wide range of natural language processing problems (Luong et al., 2015; Pasunuru and Bansal, 2017; Hashimoto et al., 2017; Ruder et al., 2017b; Kaiser et al., 2017; McCann et al., 2018).”
MTL-NAS: Task-Agnostic Neural Architecture Search Towards General-Purpose Multi-Task Learning (2020, methodology): “Our work leverages NAS to move towards GP-MTL, which (or whose search space) is compatible with any task combinations [27,29,36–38,40,42,52,67].”