A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

By Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, Richard Socher. 2016

589 citations recorded by the source corpus.

Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks. Ideally, the linguistic levels of morphology, syntax and semantics would benefit each other by being trained in a single model. We introduce a joint many-task model together with a strategy for successively growing its depth to solve increasingly complex tasks. Higher layers include shortcut connections to lower-level task predictions to reflect linguistic hierarchies. We use a simple regularization term to allow for optimizing all model weights to improve one task’s loss without exhibiting catastrophic interference of the other tasks. Our single end-to-end model obtains state-of-the-art or competitive results on five different tasks from tagging, parsing, relatedness, and entailment tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, background): “…(Strubell et al., 2017) 90.65 ± 0.15 86.84 ± 0.19 Tri-Trained LSTM (Lewis et al., 2016) 94.7 Shortcut LSTM (Wu et al., 2017) 95.08 97.53 JMT* (Hashimoto et al., 2017) 95.77 97.55 94.67 92.90 LM-LSTM-CNN-CRF (Liu et al., 2017) 95.96 ± 0.08 91.71 ± 0.10 97.53 ± 0.03 TagLM † (Peters et al.,…”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “More recent work has explored using labels from core NLP tasks to supervise training of lower levels of deep neural networks (Søgaard and Goldberg, 2016; Hashimoto et al., 2016) and automatically learning cross-task sharing mechanisms for multi-task learning (Ruder et al., 2017).”
Deep Contextualized Word Representations (2018, background): “, part-of-speech tags) at the lower levels of a deep LSTM can improve overall performance of higher level tasks such as dependency parsing (Hashimoto et al., 2017) or CCG super tagging (Søgaard and Goldberg, 2016).”
Learned in Translation: Contextualized Word Vectors (2017, methodology): “For VQA, Fukui et al. [2016] propose to combine text representations with visual representations extracted by a pretrained residual network [He et al., 2016].”
Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “75 improvement in F1 when including supervised labels from the PTB POS tags or CoNLL 2003 entities (Yang et al., 2017; Søgaard and Goldberg, 2016; Hashimoto et al., 2016).”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “can contribute to models performing higher-level tasks (Hashimoto et al., 2016).”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “…foundation to extend proposals of uniﬁed, multi-task systems for all of NLP (Collobert & Weston, 2008; Collobert et al., 2011), parsing and tagging (Hashimoto et al., 2016), multiple languages (Wu et al., 2016; Johnson et al., 2017), and multiple modalities (Luong et al., 2015; Kaiser et al.,…”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, background): “Hashimoto et al. (2016) similarly present a multi-task framework for textual entailment with task supervision at different levels of learning.”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “…recent work for NLP has centered on neural architecture design: e.g., ensuring only beneﬁcial information is shared across tasks (Liu et al., 2017; Ruder et al., 2019) or arranging tasks in linguistically-motivated hierarchies (Søgaard and Goldberg, 2016; Hashimoto et al., 2017; Sanh et al., 2019).”
Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning (2017, methodology): “We use ﬁxed GloVe word embeddings (Pennington et al., 2014) and character n-gram embeddings (Hashimoto et al., 2016).”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “Traditionally, this literature focuses on setups that involve evaluation on tasks that are observed during training (Collobert and Weston, 2008; Hashimoto et al., 2017).”
Deep Biaffine Attention for Neural Dependency Parsing (2016, methodology): “We make a few modiﬁcations to the graph-based architectures of Kiperwasser & Goldberg (2016), Hashimoto et al. (2016), and Cheng et al. (2016), shown in Figure 2: we use biafﬁne attention instead of bilinear or traditional MLP-based attention; we use a biafﬁne dependency label classiﬁer; and we…”
Exploring Shared Structures and Hierarchies for Multiple NLP Tasks (2018, positive): “As proposed by (Hashimoto et al. 2017), tasks of POS Tag-ging, Chunking, NER and sentence-level problems form a hierarchical linguistic structure, to which multi-task models should conform.”
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017, background): “Current state-of-the-art performance for this setup is at about 95% F1-measure (Hashimoto et al., 2016).”
AdapterHub: A Framework for Adapting Transformers (2020, background): “, 2019), catastrophic interference where the performance of a set of tasks deteriorates when adding new tasks (Hashimoto et al., 2017), and intricate task weighting for tasks with different distributions (Sanh et al.”
Efficient and Robust Question Answering from Minimal Context over Documents (2018, background): “…the question by concate-nating 300 -dimensional Glove embeddings pre-trained on the 840B Common Crawl corpus (Pen-nington et al., 2014), 100 -dimensional character n-gram embeddings by Hashimoto et al. (2017), and 300 -dimensional contextualized embeddings pre-trained on WMT (McCann et al., 2017).”
Multi-task Deep Reinforcement Learning with PopArt (2018, background): “…adoption is indicated by the wider use of this kind of learning in supervised settings (Johnson et al. 2017; Lu et al. 2016; Misra et al. 2016; Hashimoto et al. 2016), where loss functions are naturally well scaled (e.g. cross entropy), or can be easily scaled thanks to the stationarity of…”