Self-training Improves Pre-training for Natural Language Understanding

By Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Çelebi, Michael Auli, Ves Stoyanov, Alexis Conneau. 2020

173 citations recorded by the source corpus.

Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semi-supervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with improvements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “Work on retrieval-based data augmentation shows that self-training captures signal complementary to pre-training but has focused on classiﬁcation tasks (Du et al., 2021).”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, positive): “Experiments on the full Librispeech corpus as well as the low-resource labeled data setups of Libri-light show that self-training and unsupervised pre-training are indeed complementary, a finding that is inline with recent work in natural language understanding [25].”
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, positive): “To fully exploit unsupervised data, we take a self-training framework similar to (Lee et al., 2013; Du et al., 2021).”
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021, background): “Self-training can yield complementary accuracy gains with pretraining (Du et al., 2020) and can improve robustness to domain shift (Kumar et al.”
Self-Supervised Representation Learning: Introduction, advances, and challenges (2021, unknown): “However, preliminary results in text [100] suggest that SSL and SSRL can be synergistic when used together.”
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning (2021, methodology): “Du et al. (2021) show that self-training improves upon language model pre-training.”
Large-Scale Self- and Semi-Supervised Learning for Speech Translation (2021, background): “As previously shown for computer vision [48], natural language understanding [37] and speech recognition [9], supervised or unsupervised pretraining can be complementary to self-training.”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, background): “…papers have adapted various resources for pretraining models to enhance their performances on fewshot learning, such as pretraining on hypertext (Aghajanyan et al., 2021b), question-infused pretraining (Jia et al., 2021), and self-training (Du et al., 2021; Vu et al., 2021; Wang et al., 2021b).”
Few-Shot Named Entity Recognition: A Comprehensive Study (2020, methodology): “…semi-supervised methods, and has recently achieved improved performance for tasks such as ImageNet classiﬁcation (Xie et al., 2020), visual object detection (Zoph et al., 2020), neural machine translation (He et al., 2019), sentence classiﬁcation (Mukherjee and Awadallah, 2020; Du et al., 2020).”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, methodology): “[33] introduce SentAugment, a data augmentation method that computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web.”
Leveraging QA Datasets to Improve Generative Data Augmentation (2022, background): “Du et al. (2021) compute task-specific query embeddings to retrieve sentences from unlabeled documents from the Internet.”