Towards Zero-Label Language Learning

By Zirui Wang, Adams Wei Yu, Orhan Firat, Yuan Cao. 2021

115 citations recorded by the source corpus.

This paper explores zero-label learning in Natural Language Processing (NLP), whereby no human-annotated data is used anywhere during training and models are trained purely on synthetic data. At the core of our framework is a novel approach for better leveraging the powerful pretrained language models. Specifically, inspired by the recent success of few-shot inference on GPT-3, we present a training data creation procedure named Unsupervised Data Generation (UDG), which leverages few-shot prompts to synthesize high-quality training data without real human annotations. Our method enables zero-label learning as we train task-specific models solely on the synthetic data, yet we achieve better or comparable results from strong baseline models trained on human-labeled data. Furthermore, when mixed with labeled data, our approach serves as a highly effective data augmentation procedure, achieving new state-of-the-art results on the SuperGLUE benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “For BoolQ, T5 + UDG [69] is currently the best supervised model.”
Ethical and social risks of harm from Language Models (2021, background): “, 2019a) for language understanding (He et al., 2021; Sun et al., 2021; Wang et al., 2021).”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “, 2020), as well as various NLP applications (Liu et al., 2019a; Corazza et al., 2020; Wang et al., 2021).”
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “A variety of works have proposed using LMs for data generation (Schick and Schütze, 2021; Wang et al., 2021; Liu et al., 2022; Meng et al., 2023) or augmentation (Feng et al.”
Learning by Distilling Context (2022, background): “Wang et al. (2021) emphasizes the aspect of creating a dataset without any human annotations and uses a language model to generate task inputs and their labels.”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “There is large room for future work, including but not limited to: Extension to few-shot learning settings, exploring larger generator models [25, 68], better fine-tuning techniques to leverage generated data and better strategies for selecting quality training data.”
Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning (2022, methodology): “…(Anaby-Tavor et al., 2020; Yang et al., 2020) with the standard language modeling objective on the training set or using label-specific prompts (Gao et al., 2023; Meng et al., 2022a; Schick & Sch ¨ utze, 2021c; Wang et al., 2021; Ye et al., 2022) to steer text generation towards the desired label.”
A Holistic Approach to Undesired Content Detection in the Real World (2022, unknown): “Contrary to previous studies (Wang et al. 2021b; Schick and Schütze 2021), we found mixing noisy synthetic data into training hurt model performance.”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, methodology): “Without any model training, Wang et al. (2021) propose to directly use unlabeled in-domain examples as prompts to synthesize high-quality training data.”
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, unknown): “LLMs have also been extensively used for data generation (Wang et al., 2021b; Meng et al., 2023), augmentation (Feng et al., 2021) and in self-training setups (Wang et al., 2022b; Madaan et al., 2023).”
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, methodology): “LLMs have also been extensively used for data generation (Wang et al., 2021b; Meng et al., 2023), augmentation (Feng et al., 2021) and in self-training setups (Wang et al., 2022b; Madaan et al., 2023).”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “In the context of NLU tasks, certain studies (Ye et al., 2022; Gao et al., 2023a; Wang et al., 2021a) explore the use of labels as meta-information to synthesize corresponding samples for data augmentation.”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, unknown): “, 2020c), Generation from Few-shot Models (GF) (Ye et al., 2022; Wang et al., 2021); and (iii) Hidden-space augmentation: Adversarial”