Sluice networks: Learning what to share between loosely related tasks

By Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, Anders Søgaard. 2017

177 citations recorded by the source corpus.

Multi-task learning is partly motivated by the observation that humans bring to bear what they know about related problems when solving new ones. Similarly, deep neural networks can profit from related tasks by sharing parameters with other networks. However, humans do not consciously decide to transfer knowledge between tasks (and are typically not aware of the transfer). In machine learning, it is hard to estimate if sharing will lead to improvements; especially if tasks are only loosely related. To overcome this, we introduce Sluice Networks, a general framework for multi-task learning where trainable parameters control the amount of sharing -- including which parts of the models to share. Our framework goes beyond and generalizes over previous proposals in enabling hard or soft sharing of all combinations of subspaces, layers, and skip connections. We perform experiments on three task pairs from natural language processing, and across seven different domains, using data from OntoNotes 5.0, and achieve up to 15% average error reductions over common approaches to multi-task learning. We analyze when the architecture is particularly helpful, as well as its ability to fit noise. We show that a) label entropy is predictive of gains in sluice networks, confirming findings for hard parameter sharing, and b) while sluice networks easily fit noise, they are robust across domains in practice.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “More recent work has explored using labels from core NLP tasks to supervise training of lower levels of deep neural networks (Søgaard and Goldberg, 2016; Hashimoto et al., 2016) and automatically learning cross-task sharing mechanisms for multi-task learning (Ruder et al., 2017).”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “Learning such modularization can further mitigate interference between tasks (Ruder et al., 2017).”
Continual and Multi-Task Architecture Search (2019, positive): “…is primarily used to improve the generalization performance of a task by leveraging knowledge from related tasks (Caru-ana, 1998; Collobert and Weston, 2008; Girshick, 2015; Luong et al., 2015; Ruder et al., 2017; Au-genstein et al., 2018; Guo et al., 2018; Oh et al., 2017; Ruder and Plank, 2017).”
Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, background): “Routing networks share a common goal with techniques for automated selective transfer learning using attention (Rajendran et al., 2017) and learning gating mechanisms between representations (Stollenga et al., 2014), (Misra et al., 2016), (Ruder et al., 2017).”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, background): “…soft parameter sharing models, have been considered in the multi-task and lifelong learning setting [Stollenga et al., 2014, Aljundi et al., 2017, Misra et al., 2016, Ruder et al., 2017, Rajendran et al., 2017], but do not allow for nearly the level of specialization as those based on routing.”
Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation (2018, background): “Several recent works have adopted MTL in neural models (Luong et al., 2016; Misra et al., 2016; Hashimoto et al., 2017; Pasunuru and Bansal, 2017; Ruder et al., 2017; Kaiser et al., 2017).”
Multi-task learning for natural language processing in the 2020s: where are we going? (2020, background): “There are two primary families of MTL neural network approaches: hard parameter sharing and soft parameter sharing (Ruder, 2017).”
Towards multidomain and multilingual abusive language detection: a survey (2021, background): “Finally, Ousidhoum et al. [89] proposed a multitask architecture based on Sluice Network [118] coupled with Babylon cross-lingual word embedding [129], which allows the model to share the same parameters from other related tasks.”
Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation (2018, methodology): “Recently, (Ruder et al., 2017) tried to address this issue; however, their method is restrictive for S EQ 2S EQ scenarios and does not consider the input at each time step to modulate parameter sharing.”
Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations (2020, methodology): “Besides hard parameter sharing, cross-stitch [16], sluice network [18] and MMOE [13], we also evaluate two innovatively proposed structures called asymmetric sharing and customized sharing: • Asymmetric Sharing is a novel sharing mechanism to capture asymmetric relations between tasks.”
TranSlider: Transfer Ensemble Learning from Exploitation to Exploration (2020, background): “Yet more recent studies [11, 16, 26, 27, 33, 41] attempt to simultaneously learn a transfer strategy and the parameters for the target model.”
Multilingual and Multi-Aspect Hate Speech Analysis (2019, methodology): “We adopt (Ruder et al., 2017) as a learning algorithm adapted to loosely related tasks such as our ﬁve annotated aspects and, use the Babylon cross-lingual embeddings (Smith et al., 2017) to align the three languages.”
AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning (2019, methodology): “Although there are many variants of multi-task learning (Ruder et al., 2017b; Hashimoto et al., 2017; Luong et al., 2015; McCann et al., 2018), our goal is to improve the performance of a primary task”