Multi-Task Learning for Sequence Tagging: An Empirical Study

By Soravit Changpinyo, Hexiang Hu, Fei Sha. 2018

68 citations recorded by the source corpus.

We study three general multi-task learning (MTL) approaches on 11 sequence tagging tasks. Our extensive empirical results show that in about 50% of the cases, jointly learning all 11 tasks improves upon either independent or pairwise learning of the tasks. We also show that pairwise MTL can inform us what tasks can benefit others or what tasks can be benefited if they are learned jointly. In particular, we identify tasks that can always benefit others as well as tasks that can always be harmed by others. Interestingly, one of our MTL approaches yields embeddings of the tasks that reveal the natural clustering of semantic and syntactic tasks. Our inquiries have opened the doors to further utilization of MTL in NLP.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, background): “…et al. (2016) show promising results combining translation and parsing; Subramanian et al. (2018) beneﬁt from multitask learning in sentence-to-vector encoding; and Bingel and Søgaard (2017) and Changpinyo et al. (2018) offer studies of when multitask learning is helpful for lower-level NLP tasks.”
Exploring and Predicting Transferability across NLP Tasks (2020, background): “Within multi-task learning, several works (Luong et al., 2016; Mou et al., 2016; Martı́nez Alonso and Plank, 2017; Bingel and Søgaard, 2017; Kerinec et al., 2018; Changpinyo et al., 2018; Liu et al., 2019b) combine related tasks for better regularization and transfer.”
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling (2018, background): “…promising results from the combination of translation and parsing, Subramanian et al. (2018) show the beneﬁts of multitask learning in sentence-to-vector encoding, and Bingel & Søgaard (2017) and Changpinyo et al. (2018) offer studies of when multitask learning is helpful for lower-level NLP tasks.”
Drinking From a Firehose: Continual Learning With Web-Scale Natural Language (2020, methodology): “(1) Multi-task Encoder is a model that combines the user-speciﬁc information together with the word-embedding, before any sequence modeling happens.”
Low Resource Multi-Task Sequence Tagging - Revisiting Dynamic Conditional Random Fields (2020, background): “Bingel and Søgaard (2017) provide an in depth ablation study on which task combinations, such as POS, multi-word expressions, supersense tagging, etc., profit from one another, while Changpinyo et al. (2018) design different strategies for sharing weights between tasks.”