Identifying beneficial task relations for multi-task learning in deep neural networks

By Anders Søgaard, Joachim Bingel. 2017

256 citations recorded by the source corpus.

Multi-task learning (MTL) in deep neural networks for NLP has recently received increasing interest due to some compelling benefits, including its potential to efficiently regularize models and to reduce the need for labeled data. While it has brought significant improvements in a number of NLP tasks, mixed results have been reported, and little is known about the conditions under which MTL leads to gains in NLP. This paper sheds light on the specific task relations that can lead to gains from MTL models over single-task setups.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks (2018, methodology): “In earlier work less closely tied to the unsupervised pretraining setup studied here, Bingel and Søgaard (2017) and Kerinec et al.”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, background): “…et al. (2016) show promising results combining translation and parsing; Subramanian et al. (2018) benefit from multitask learning in sentence-to-vector encoding; and Bingel and Søgaard (2017) and Changpinyo et al. (2018) offer studies of when multitask learning is helpful for lower-level NLP tasks.”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, methodology): “However, multi-task NLP remains difficult for many applications, with multi-task models often performing worse than their single-task counterparts (Plank and Alonso, 2017; Bingel and Søgaard, 2017; McCann et al., 2018).”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “tter cross-modal retrieval system. Our proposed approach is also motivated by learning using privileged information (LUPI) paradigm [45, 51] and multi-task learning strategies in deep neural networks [2, 43] that share representations between closely related tasks for enhanced learning performance. ”
Taskonomy: Disentangling Task Transfer Learning (2018, background): “compositional modeling [35, 10, 13, 23, 55, 92, 90], homomorphic cryptography [42], lifelong learning [93, 15, 85, 84], functional maps [71], certain aspects of Bayesian inference and Dirichlet processes [54, 91, 90, 89, 37, 39], few-shot learning [81, 25, 24, 70, 86], transfer learning [75, 84, 29,”
Latent Multi-Task Architecture Learning (2017, methodology): “Task Properties and Performance Bingel and Søgaard (2017) correlate meta-characteristics of task pairs and gains compared to hard parameter sharing across a large set of NLP task pairs.”
Exploring and Predicting Transferability across NLP Tasks (2020, methodology): “Bingel and Søgaard (2017) find such learning curve features to be good predictors of gains from multi-task learning.”
Linguistically-Informed Self-Attention for Semantic Role Labeling (2018, background): “, 2017), and we build off work that investigates where and when to combine different tasks to achieve the best results (Søgaard and Goldberg, 2016; Bingel and Søgaard, 2017; Alonso and Plank, 2017).”
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020, neutral): “Among the prior work on predicting transfer performance, Bingel and Søgaard (2017) is the most similar to ours.”
Sluice networks: Learning what to share between loosely related tasks (2017, background): “Task Properties and Performance [4] correlate meta-characteristics of task pairs and gains from hard parameter sharing across a large set of NLP task pairs.”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, background): “There have also been efforts to predict task transferability (Bingel and Søgaard, 2017; Vu et al., 2020; Poth et al., 2021).”
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks (2018, background): “Several prior works (Yang, Salakhutdinov, and Cohen 2016; Bingel and Søgaard 2017) avoid the question of the linguistic hierarchies between NLP tasks.”
When does deep multi-task learning work for loosely related document classification tasks? (2018, methodology): “haps a bit surprising in the light of recent results for sequence tagging (Alonso and Plank, 2017; Bingel and Søgaard, 2017).”
AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning (2019, background): “Bingel and Søgaard (2017) explored task relatedness via exhaustively experimenting with all possible two task tuples in a non-automated multi-task setup.”