STraTA: Self-Training with Task Augmentation for Better Few-shot Learning

By Tu Vu, Minh-Thang Luong, Quoc V. Le, Grady Simon, Mohit Iyyer. 2021

66 citations recorded by the source corpus.

Despite their recent successes in tackling many NLP tasks, large-scale pre-trained language models do not perform as well in few-shot settings where only a handful of training examples are available. To address this shortcoming, we propose STraTA, which stands for Self-Training with Task Augmentation, an approach that builds on two key ideas for effective leverage of unlabeled data. First, STraTA uses task augmentation, a novel technique that synthesizes a large amount of data for auxiliary-task fine-tuning from target-task unlabeled texts. Second, STraTA performs self-training by further fine-tuning the strong base model created by task augmentation on a broad distribution of pseudo-labeled data. Our experiments demonstrate that STraTA can substantially improve sample efficiency across 12 few-shot benchmarks. Remarkably, on the SST-2 sentiment dataset, STraTA, with only 8 training examples per class, achieves comparable results to standard fine-tuning with 67K training examples. Our analyses reveal that task augmentation and self-training are both complementary and independently effective.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “Task augmentation (Vu et al., 2021) generates data in the target domain by us-ing a model trained on the auxiliary task of natural language inference.”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, background): “Task transferability We also build on existing work on task transferability (Wang et al., 2019a; Liu et al., 2019a; Talmor and Berant, 2019; Pruksachatkun et al., 2020; Vu et al., 2020, 2021).”
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022, methodology): “…used this mixing (Xue et al., 2021; Shakeri et al., 2021) Previous work has also explored intermediate adaptation of pre-trained models, which has been shown to be effective for M ODEL T UNING (Howard and Ruder, 2018; Phang et al., 2019; Vu et al., 2020, 2021) and P ROMPT T UNING (Vu et al., 2022).”
Leveraging QA Datasets to Improve Generative Data Augmentation (2022, methodology): “This is in line with previous research findings (Du et al., 2021; Vu et al., 2021; Pryzant et al., 2022).”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, background): “…papers have adapted various resources for pretraining models to enhance their performances on fewshot learning, such as pretraining on hypertext (Aghajanyan et al., 2021b), question-infused pretraining (Jia et al., 2021), and self-training (Du et al., 2021; Vu et al., 2021; Wang et al., 2021b).”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, methodology): “Supervised data generation methods are also studied for building auxiliary tasks (Vu et al., 2021) and dataset creation based on human and machine collaboration (Liu et al., 2022).”