Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

By Yu Meng, Jiaxin Huang, Yu Zhang, Jiawei Han. 2022

302 citations recorded by the source corpus.

Pretrained language models (PLMs) have demonstrated remarkable performance in various natural language processing tasks: Unidirectional PLMs (e.g., GPT) are well known for their superior text generation capabilities; bidirectional PLMs (e.g., BERT) have been the prominent choice for natural language understanding (NLU) tasks. While both types of models have achieved promising few-shot learning performance, their potential for zero-shot learning has been underexplored. In this paper, we present a simple approach that uses both types of PLMs for fully zero-shot learning of NLU tasks without requiring any task-specific data: A unidirectional PLM generates class-conditioned texts guided by prompts, which are used as the training data for fine-tuning a bidirectional PLM. With quality training data selected based on the generation probability and regularization techniques (label smoothing and temporal ensembling) applied to the fine-tuning stage for better generalization and stability, our approach demonstrates strong performance across seven classification tasks of the GLUE benchmark (e.g., 72.3/73.8 on MNLI-m/mm and 92.8 on SST-2), significantly outperforming zero-shot prompting methods and achieving even comparable results to strong few-shot approaches using 32 training samples per class.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Jiawei Han
3. Yu Meng
4. Bernard Ghanem
5. Chao Zhang
6. Heng Ji
7. Lidong Bing
8. Guohao Li

In Large Language Models

1. Jiawei Han
2. Alexander J. Ratner
3. Dacheng Tao
4. Heng Ji
5. Meng Jiang
6. Yu Meng
7. Chao Zhang
8. Hasan Hammoud

Citations in the corpus

• Large Language Models Can Self-Improve (2022, positive): “In the second format, we use examples of questions and their direct answers as standard prompting, and the language model output is supposed to also only contain the direct answer.”
• A Survey on Knowledge Distillation of Large Language Models (2024, background): “Several studies have also focused on multiple NLU tasks (Ding et al., 2023a; He et al., 2023a; Wang et al., 2021a; He et al., 2022; Ye et al., 2022; Meng et al., 2022).”
• Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning (2022, methodology): “…generalization and stability of training: First fine-tune C ϕ on D train with standard supervised training, and then continue fine-tuning it on D gen by applying label smoothing (Szegedy et al., 2016) and temporal ensembling (Laine & Aila, 2017) as regularization, following (Meng et al., 2022a).”
• Is GPT-3 a Good Data Annotator? (2022, methodology): “One approach to address the need for labeled data is through data augmentation (Feng et al., 2021; Meng et al., 2022; Chen et al., 2023), which involves generating new data by modifying existing data points using transformations based on prior knowledge about the problem’s structure (Yang et al.,…”
• On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, unknown): “For methods based on heuristic metrics, the key step is to design appropriate criteria based on the learning dynamics, such as confidence score (Seedat et al., 2023), influence function (Ye et al., 2022b), and generation ability (Meng et al., 2022).”
• ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, methodology): “One concurrent work (Meng et al., 2022) studies dataset generation for text classification and natural language inference tasks.”
• Large Language Models on Graphs: A Comprehensive Survey (2023, methodology): “The philosophy of using LLM to generate pseudo data is widely explored in NLP [83], [89].”
• CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, methodology): “There is also a large chunk of work that has proposed methods for automatic dataset creation [67, 57, 19, 75, 20, 98, 59, 96, 129, 62, 130, 86, 8].”
• Language Models Can Teach Themselves to Program Better (2022, methodology): “Temperature sweeps for the GPT-Neo models solving the Codex puzzles was performed as shown in Figure 8.”
• PERSONACHATGEN: Generating Personalized Dialogues using GPT-3 (2022, methodology): “Meng et al. (2022) used a unidirectional PLM to generate a dataset that corresponds to given label information for the zero-shot learning of NLU tasks.”
• A Survey on Large Language Models for Code Generation (2024, unknown): “between smaller and larger LLMs. 7B/13B models outperform the 70B model.”
• Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias (2023, methodology): “For the generated synthetic dataset, we stick to the strict zero-shot learning setting [30], train all the models for 6 epochs and use the model from the last epoch without using the validation set for evaluation.”