Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks

By Jason Phang, Thibault Févry, Samuel R. Bowman. 2018

487 citations recorded by the source corpus.

Pretraining sentence encoders with language modeling and related unsupervised tasks has recently been shown to be very effective for language understanding tasks. By supplementing language model-style pretraining with further training on data-rich supervised tasks, such as natural language inference, we obtain additional performance improvements on the GLUE benchmark. Applying supplementary training on BERT (Devlin et al., 2018), we attain a GLUE score of 81.8---the state of the art (as of 02/24/2019) and a 1.4 point improvement over BERT. We also observe reduced variance across random restarts in this setting. Our approach yields similar improvements when applied to ELMo (Peters et al., 2018a) and Radford et al. (2018)'s model. In addition, the benefits of supplementary training are particularly pronounced in data-constrained regimes, as we show in experiments with artificially limited training data.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “A better notion of the relatedness of tasks could also help choose supervised pre-training tasks, which has been shown to be helpful for the GLUE benchmark [Phang et al., 2018].”
Story Ending Prediction by Transferable BERT (2019, methodology): “STILTs [ Phang et al. , 2018 ] ﬁne-tuned a GPT model on some intermediate tasks to get better performance on the GLUE [ Wang et al. , 2018 ] benchmark.”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “Phang et al. (2018) show that BERT can be improved using two-stage pretraining, i.e., ﬁne-tuning the pretrained model on an intermediate data-rich supervised task before ﬁne-tuning it again on a data-poor target task.”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, background): “Contemporaneously with our work, Phang et al. (2018) showed that pre-training on supervised tasks could be beneﬁcial even when using pre-trained language models, especially for a textual entailment task.”
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding (2019, methodology): “We compare MT-DNN KD with existing state-of-the-art models including BERT (Devlin et al., 2018), STILT (Phang et al., 2018), Snorkel MeTal (Hancock et al., 2019), and MT-DNN (Liu et al., 2019).”
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding (2019, background): “This follows the two-stage transfer learning STILTs introduced in [24].”
Unifying Question Answering and Text Classification via Span Extraction (2019, methodology): “For training the models, we closely follow the original BERT setup(Devlin et al., 2018) and (Phang et al., 2018).”
Cloze-driven Pretraining of Self-attention Networks (2019, neutral): “STILTs adds another ﬁne-tuning step on another downstream task which is similar to the ﬁ-nal task.”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “MNLI and QQP have previously been shown to be effective for pretraining in other settings (Conneau et al., 2017; Sub-ramanian et al., 2018; Phang et al., 2018).”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, methodology): “Concurrently with our work, several other recent works also explore ﬁne-tuning BERT us-ing multiple tasks (Phang et al., 2018; Liu et al., 2019b; Keskar et al., 2019).”
Making Pre-trained Language Models Better Few-shot Learners (2021, background): “termediate training (Phang et al., 2018; Yin et al., 2020), where a related, intermediate task is given.”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, methodology): “Phang et al. (2019) use an intermediate fine-tuning stage using four tasks with large datasets for Natural Language Understanding.”
Transformers as Soft Reasoners over Language (2020, background): “This additional fine-tuning step has been previously shown to help with sensitivity to hyperparameters [Phang et al., 2018] and improve question-answering [Sun et al.”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, methodology): “Efforts on the data side have focused on intermediate ﬁne-tuning either by using unlabeled target domain data (Karouzos et al., 2021; Gururangan et al., 2020) or via labeled data from other tasks (Phang et al., 2018; Agha-janyan et al., 2021; Vu et al., 2020).”
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, background): “NLI has broad applicability in NLP: it has proven useful for pretraining (Clark et al., 2019; Phang et al., 2018), and can be applied to verify candidate answers in question-answering (Chen et al., 2021) or factuality of generated summaries (Maynez et al., 2020).”
True Few-Shot Learning with Language Models (2021, methodology): “This result may be due to the fact that we retrain the model using the CV-chosen hyperparameters, but ﬁnetuning pretrained LMs often has high variance in performance [68, 81].”