Deep multi-task learning with low level tasks supervised at lower layers

By Anders Søgaard, Yoav Goldberg. 2016

474 citations recorded by the source corpus.

In all previous work on deep multi-task learning we are aware of, all task supervisions are on the same (outermost) layer. We present a multi-task learning architecture with deep bi-directional RNNs, where different tasks supervision can happen at different layers. We present experiments in syntactic chunking and CCG supertagging, coupled with the additional task of POS-tagging. We show that it is consistently better to have POS supervision at the innermost rather than the outermost layer. We argue that this is because “lowlevel” tasks are better kept at the lower layers, enabling the higher-level tasks to make use of the shared representation of the lower-level tasks. Finally, we also show how this architecture can be used for domain adaptation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “For NLP, most work has focused on a small number of closely related tasks (Lu-ong et al., 2016; Zhang and Weiss, 2016; Søgaard and Goldberg, 2016; Peng et al., 2017).”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “More recently, there has been work using labels from core NLP tasks to supervise training of lower levels of deep neural networks (Søgaard and Goldberg, 2016; Hashimoto et al., 2016) and automatically learning cross-task sharing mechanisms for multi-task learning (Ruder et al.”
Deep Contextualized Word Representations (2018, background): “For example, introducing multi-task syntactic supervision (e.g., part-of-speech tags) at the lower levels of a deep LSTM can improve overall performance of higher level tasks such as dependency parsing (Hashimoto et al., 2017) or CCG super tagging (Søgaard and Goldberg, 2016).”
Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “Previous work has explored meth-ods for jointly learning the bidirectional RNN with supplemental labeled data from other tasks (e.g., Søgaard and Goldberg, 2016; Yang et al., 2017).”
An Overview of Multi-Task Learning in Deep Neural Networks (2017, background): “In contrast, in natural language processing (NLP), recent work focused on ﬁnding better task hierarchies for multi-task learning: [Søgaard and Goldberg, 2016] show that low-level tasks, i.e. NLP tasks typically used for preprocessing such as part-of-speech tagging and named entity recognition,…”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “…recent work for NLP has centered on neural architecture design: e.g., ensuring only beneﬁcial information is shared across tasks (Liu et al., 2017; Ruder et al., 2019) or arranging tasks in linguistically-motivated hierarchies (Søgaard and Goldberg, 2016; Hashimoto et al., 2017; Sanh et al., 2019).”
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks (2016, methodology): “Søgaard and Goldberg (2016) proposed to jointly learn POS tagging and chunking in different layers, but they only showed improvement for chunking.”
Exploring Shared Structures and Hierarchies for Multiple NLP Tasks (2018, neutral): “Hierarchical-sharing When the tasks are related but belong to different stages in a pipeline or general data ﬂow graph, the sharing scheme should have a hierarchical structure, where lower level tasks branching off earlier to feed to their private heads (Søgaard and Goldberg 2016).”
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017, methodology): “Long-Short-Term-Memory (LSTM) Networks (Hochreiter and Schmidhuber, 1997) are widely used in NLP and achieve state-of-the-art performance for many sequence-tagging-tasks including Part-of-Speech tagging (Ma and Hovy, 2016), Named Entity Recognition (Ma and Hovy, 2016) and Chunking (Søgaard and Goldb”
Latent Multi-Task Architecture Learning (2017, methodology): “Our meta-architecture is very ﬂexible and can be seen as a generalization over several existing algorithms for transfer and multi-task learning, including (Caruana 1998; Daumé III 2007; Søgaard and Goldberg 2016; Misra et al. 2016).”
Identifying beneficial task relations for multi-task learning in deep neural networks (2017, methodology): “, 2016), results are also often mixed (Collobert and Weston, 2008; Søgaard and Goldberg, 2016; Martı́nez Alonso and Plank, 2017), and theoretical guarantees no longer apply.”
GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (2017, background): “…multitask learning has applications in multiple other ﬁelds, from natural language processing (Hashimoto et al. (2016); Collobert & We-ston (2008); Søgaard & Goldberg (2016)) to speech synthesis (Wu et al. (2015); Seltzer & Droppo (2013)), from very domain-speciﬁc applications like trafﬁc…”
Sluice networks: Learning what to share between loosely related tasks (2017, methodology): “While there is theory providing guarantees for effectivity of some types of multi-task learning [2, 3], none of these hold for the loosely related tasks to which multi-task learning is most often applied in natural language processing (NLP) or computer vision [8, 25].”
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (2019, background): “More recently, alternative approaches have been suggested that only share parameters across lower layers (Søgaard and Goldberg, 2016).”
When is multitask learning effective? Semantic sequence prediction under varying data conditions (2016, methodology): “…learning (MTL) has been applied with success to a variety of sequence-prediction tasks including chunking and tagging (Collobert et al., 2011; Søgaard and Goldberg, 2016; Bjerva et al., 2016; Plank, 2016), name error detection (Cheng et al., 2015) and machine translation (Luong et al., 2016).”
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks (2018, methodology): “Following (Hashimoto et al. 2017; Søgaard and Goldberg 2016), we therefore introduce a hierarchy between the tasks so that low level tasks are supervised at lower levels of the architecture while keeping more com-”