When is multitask learning effective? Semantic sequence prediction under varying data conditions

By Barbara Plank, Héctor Martínez Alonso. 2016

166 citations recorded by the source corpus.

Multitask learning has been applied successfully to a range of tasks, mostly morphosyntactic. However, little is known on when MTL works and whether there are data characteristics that help to determine the success of MTL. In this paper we evaluate a range of semantic sequence labeling tasks in a MTL setup. We examine different auxiliary task configurations, amongst which a novel setup, and correlate their impact to data-dependent conditions. Our results show that MTL is not always effective, because significant improvements are obtained only for 1 out of 5 tasks. When successful, auxiliary tasks with compact and more uniform label distributions are preferable.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

An Overview of Multi-Task Learning in Deep Neural Networks (2017, positive): “Recent work [Alonso and Plank, 2017] has found auxiliary tasks with compact and uniform label distributions to be preferable for sequence tagging problems in NLP, which we have conﬁrmed in experiments [Ruder et al., 2017].”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, methodology): “However, applying multi-task NLP remains difﬁcult for many applications, with multi-task models often performing worse than their single-task counterparts (Plank and Alonso, 2017; Bingel and Søgaard, 2017; McCann et al., 2018).”
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017, methodology): “deep BiLSTM-networks and achieved an improvement for Chunking and CCG supertagging, while for NER, super senses (SemCor), and MWE brackets & super sense tags no improvement was achieved.Mart´ınez Alonso and Plank (2017) achieved only for 1 out of 5 tasks signiﬁcant improvements. In the following”
Latent Multi-Task Architecture Learning (2017, methodology): “ter sharing to jointly learn different sequence-tagging tasks (NER, POS tagging, Chunking) across languages. They also use word and character embeddings and share character embeddings in their model. [19] explore a similar set-up, but sharing is limited to the initial layer. In all three papers, the”
Exploring and Predicting Transferability across NLP Tasks (2020, methodology): “Identifying beneficial task relationships: To predict transferable tasks, some methods (Martı́nez Alonso and Plank, 2017; Bingel and Søgaard, 2017) rely on features derived from dataset characteristics and learning curves.”
Sluice networks: Learning what to share between loosely related tasks (2017, methodology): “We provide an analysis of the ability of sluice networks to ﬁt noise, as well as what properties are predictive of gains with sluice networks, seeing that the effect size correlates highly with label entropy, conﬁrming previous ﬁndings for hard parameter sharing [19, 4].”
Scheduled Multi-Task Learning: From Syntax to Translation (2018, background): “n to English translation task. Many approaches have been employing multitask learning in order to inject linguistic knowledge with great success (Luong et al., 2015b; Niehues and Cho, 2017; Mart´ınez Alonso and Plank, 2017, inter-alia). The ﬁnal representation is then adapted to solve multiple tasks”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, background): “(Alonso and Plank, 2016) trains 1440 task combinations, each with a main task and one or two auxiliary tasks, and finds that performance on the main task improves the most with auxiliary tasks whose label distributions have high entropy and low kurtosis.”
When does deep multi-task learning work for loosely related document classification tasks? (2018, background): “Recent studies have tried to study empirically when multi-task learning leads to improvements (Alonso and Plank, 2017; Bingel and Søgaard, 2017).”
Multitask Learning for Fine-Grained Twitter Sentiment Analysis (2017, methodology): “Indeed, previous work on multitask learning showed the effectiveness of biLSTMs in a variety of problems: (Alonso & Plank, 2016) tackled sequence prediction, while (Plank, 2016) and (Kiperwasser & Goldberg, 2016) used biLSTMs for Named Entity Recognition and dependency parsing respectively.”
Neural Sequence Learning Models for Word Sense Disambiguation (2017, methodology): “Furthermore, introducing LEX (cf. Section 4) as auxiliary task was generally helpful; on the other hand, POS did not seem to help, corroborating previous ﬁndings (Alonso and Plank, 2017; Bingel and Søgaard, 2017).”
Multi-Task Learning of Keyphrase Boundary Classification (2017, methodology): “Hard parameter sharing has been studied for several tasks, including CCG super tagging (Søgaard and Goldberg, 2016), text normalisation (Bollman and Søgaard, 2016), neural machine translation (Dong et al., 2015; Luong et al., 2016), and super-sense tagging (Mart´ınez Alonso and Plank, 2017).”
Multi-task learning for natural language processing in the 2020s: where are we going? (2020, background): “Alonso and Plank (2017) pose an additional question: most auxiliary tasks are classiﬁcation tasks, how do regression tasks fare as auxiliary tasks?”
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020, result): “But multi-task NLP models may produce inferior results compared with their single-task counterparts (Martínez Alonso and Plank, 2017; Bingel and Sø-gaard, 2017).”