A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks

By Victor Sanh, Thomas Wolf, Sebastian Ruder. 2018

248 citations recorded by the source corpus.

Much effort has been devoted to evaluate whether multi-task learning can be leveraged to learn rich representations that can be used in various Natural Language Processing (NLP) down-stream applications. However, there is still a lack of understanding of the settings in which multi-task learning has a significant effect. In this work, we introduce a hierarchical model trained in a multi-task learning setup on a set of carefully selected semantic tasks. The model is trained in a hierarchical fashion to introduce an inductive bias by supervising a set of low level tasks at the bottom layers of the model and more complex tasks at the top layers of the model. This model achieves state-of-the-art results on a number of tasks, namely Named Entity Recognition, Entity Mention Detection and Relation Extraction without hand-engineered features or external NLP tools like syntactic parsers. The hierarchical training supervision induces a set of shared semantic representations at lower layers of the model. We show that as we move from the bottom to the top layers of the model, the hidden states of the layers tend to represent more complex semantic information.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “…recent work for NLP has centered on neural architecture design: e.g., ensuring only beneﬁcial information is shared across tasks (Liu et al., 2017; Ruder et al., 2019) or arranging tasks in linguistically-motivated hierarchies (Søgaard and Goldberg, 2016; Hashimoto et al., 2017; Sanh et al., 2019).”
Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, methodology): “We use proportional sampling (Sanh et al., 2018) to select batches from each task for training, with batch size 128.”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, methodology): “For reference, we also train a model on each individual task family using proportional example mixing (Sanh et al., 2019).”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, background): “…with the aim of learning a shared representation that will enable the model to generalize better on each task (Caruana, 1997; Collobert and Weston, 2008; Nam et al., 2014; Liu et al., 2016, 2017; Zhang and Yang, 2017; Ruder, 2017; Ruder et al., 2019; Sanh et al., 2019; Pfeiffer et al., 2020a).”
Speech Model Pre-training for End-to-End Spoken Language Understanding (2019, methodology): “By using phonemes as intermediate pre-training targets [20, 35, 36], we are able to pre-train on speech segments with no word label.”
Transformer is All You Need: Multimodal Multitask Learning with a Unified Transformer (2021, unknown): “[52, 16, 32, 49, 10]), or multimodal areas ( e.g .”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, methodology): “Following these two works, (Sanh et al., 2019) introduces a similarly inspired model for a different set of tasks, achieving SOTA results for Named Entity Recognition, Entity Mention Detection and Relation Extraction.”
HyperGrid Transformers: Towards A Single Model for Multiple Tasks (2021, background): “Similarly, (Sanh et al., 2019) proposed a hierarchical multi-task model based on the intuition of low-level and high-level tasks.”
On the Contributions of Visual and Textual Supervision in Low-resource Semantic Speech Retrieval (2019, unknown): “For MTL, each mini-batch is sampled from either set B or set C with probability proportional to the number of data points in each set, as in [23].”
HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections (2020, background): “Similarly, [Sanh et al., 2019] proposed a hierarchical multi-task model based on the intuition of low-level and high-level tasks.”
Multi-Task Learning for Dense Prediction Tasks: A Survey (2020, methodology): “[27] trained the network by randomly sampling a single task for the weight update during every iteration.”
A general framework for information extraction using dynamic span graphs (2019, methodology): “While the performance of DyGIE beneﬁts from ELMo, it advances over some systems (Luan et al., 2018a; Sanh et al., 2019) that also incorporate ELMo.”
A Comprehensive Survey on Graph Anomaly Detection With Deep Learning (2021, background): “Opportunities : Multi-task learning provides effective machinery with which incorporate associated tasks [198], [199].”
UniT: Multimodal Multitask Learning with a Unified Transformer (2021, unknown): “[52, 17, 33, 50, 10]), or multimodal areas ( e.g .”