Multi-Task Deep Neural Networks for Natural Language Understanding

By Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao. 2019

1,357 citations recorded by the source corpus.

In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks. MT-DNN not only leverages large amounts of cross-task data, but also benefits from a regularization effect that leads to more general representations to help adapt to new tasks and domains. MT-DNN extends the model proposed in Liu et al. (2015) by incorporating a pre-trained bidirectional transformer language model, known as BERT (Devlin et al., 2018). MT-DNN obtains new state-of-the-art results on ten NLU tasks, including SNLI, SciTail, and eight out of nine GLUE tasks, pushing the GLUE benchmark to 82.7% (2.2% absolute improvement) as of February 25, 2019 on the latest GLUE test set. We also demonstrate using the SNLI and SciTail datasets that the representations learned by MT-DNN allow domain adaptation with substantially fewer in-domain labels than the pre-trained BERT representations. Our code and pre-trained models will be made publicly available.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Language Models are Few-Shot Learners (2020, background): “For Ro-En, few shot GPT-3 is very close to the overall SOTA which is achieved with unsupervised pretraining, ﬁnetuning on 608K labeled examples, and backtranslation [LHCG19b].”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “This has led to the development of a wide variety of pre-training objectives [Dai and Le, 2015; Ramachandran et al., 2016; Radford et al., 2018; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019b; Wang et al., 2019a; Song et al., 2019; Dong et al., 2019; Joshi et al., 2019].”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “QNLI: Recent submissions on the GLUE leaderboard adopt a pairwise ranking formulation for the QNLI task, in which candidate answers are mined from the training set and compared to one another, and a single (question, candidate) pair is classiﬁed as positive (Liu et al., 2019b,a; Yang et al., 2019).”
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “Multi-task ensembles on test (from leaderboard as of Oct 28, 2019) MT-DNN⇤ [20] 87.”
Adversarial Training for Large Neural Language Models (2020, methodology): “…on unlabeled text has proven very effective to improve generalization performance for a variety of downstream tasks, as exempliﬁed by BERT (Devlin et al., 2018) and other transformer-based models (Liu et al., 2019c; Radford et al., 2018; Clark et al., 2020; Dong et al., 2019; Bao et al., 2020).”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, background): “We use Transformer-based models (Vaswani et al., 2017) to learn the probability p ( x i | x 1: i − 1 ) , which has been demonstrated to scale to large self-supervised models with outstanding performance in generation quality and representation learning, including BERT (Devlin et al., 2018), GPT-2 (R”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “To rule out the latter, we trained only on ANLI ( ∼ 163k training examples): RoBERTa matches BERT when trained on the much larger, fully in-domain SNLI+MNLI combined dataset (943k training examples) on MNLI, with both getting ∼ 86 (the third row in Table 6).”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “Recent parallel work (Ramachandran et al., 2016; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Liu et al., 2019b; Dai et al., 2019; Yang et al., 2019; Liu et al., 2019a; Lan et al., 2019) further builds upon these ideas by not just transferring the language model to extract…”
Story Ending Prediction by Transferable BERT (2019, methodology): “MT-DNN [ Liu et al. , 2019 ] extended multi-task learning by incorporating a pre-trained BERT model, which is very close to the work of this paper.”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “Much recent work has made similar observations about the limitations of existing pretrained models (Jia and Liang, 2017; Naik et al., 2018; McCoy and Linzen, 2019; McCoy et al., 2019; Liu et al., 2019a,b).”
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding (2019, methodology): “MT-DNN This is the model described in Section 2 and Liu et al. (2019).”
A Hybrid Neural Network Model for Commonsense Reasoning (2019, methodology): “Following (Liu et al., 2019b; Kocijan et al., 2019), the training procedure of HNN consists of two steps: (1) pretraining the text encoder on raw text 2 , and (2) multi-task learning of HNN on WSCR which is the most popular WSC dataset, as suggested by Kocijan et al. (2019).”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “While recent work Liu et al. (2019a) has applied several task-speciﬁc strategies to increase performance on the individual GLUE tasks, we follow BERT’s single-task setting and only add a linear classiﬁer on top of the [CLS] token for these classiﬁcation tasks.”
A Surprisingly Robust Trick for the Winograd Schema Challenge (2019, background): “Building on top of a LM has proven to be very successful, producing state-of-the-art (SOTA) results (Liu et al., 2019; Trinh and Le, 2018) on benchmark datasets like G LUE (Wang et al., 2019) or W SC 273 (Levesque et al., 2011).”
Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, methodology): “Our model also outperforms the best extractive model (Liu, 2019) by 1 point in ROUGE-L.”
Unifying Question Answering and Text Classification via Span Extraction (2019, methodology): “Unlike similar approaches using task-speciﬁc heads (Liu et al., 2019), SEBert allows for a single model across a broader set of tasks.”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “Follow-up work has explored parameterefficient fine-tuning (Stickland and Murray, 2019; Houlsby et al., 2019) and better target task adaptation via multitask fine-tuning (Phang et al., 2018; Liu et al., 2019), but work in this area is nascent.”
ERNIE: Enhanced Representation through Knowledge Integration (2019, methodology): “MT-DNN (Liu et al., 2019) combine pre-training learning and multi-task learning to improve the performances over several different tasks in GLUE (Wang et al., 2018).”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, methodology): “A crucial difference distinguishing our work from the STILTs, Snorkel MeTaL, and MT-DNN KD methods in Table 2 is that we do not single-task ﬁne-tune our model.”