Generating Datasets with Pretrained Language Models

By Timo Schick, Hinrich Schütze. 2021

276 citations recorded by the source corpus.

To obtain high-quality sentence embeddings from pretrained language models (PLMs), they must either be augmented with additional pretraining objectives or finetuned on a large set of labeled text pairs. While the latter approach typically outperforms the former, it requires great human effort to generate suitable datasets of sufficient size. In this paper, we show how PLMs can be leveraged to obtain high-quality sentence embeddings without the need for labeled data, finetuning or modifications to the pretraining objective: We utilize the generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch, which we then use for finetuning much smaller and more efficient models. Our fully unsupervised approach outperforms strong baselines on several semantic textual similarity datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “…(Yarowsky, 1995), relation extraction (Brin, 1999; Agichtein and Gravano, 2000), parsing (McClosky et al., 2006), sequence generation (He et al., 2020), few-shot text classification (Schick and Schütze, 2021a) and retrieval (Izacard and Grave, 2021) to reasoning (Zelikman et al., 2022).”
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “A variety of works have proposed using LMs for data generation (Schick and Schütze, 2021; Wang et al., 2021; Liu et al., 2022; Meng et al., 2023) or augmentation (Feng et al.”
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, methodology): “Inspired by recent work on utilizing language models for data generation (Schick and Schütze, 2021b; Lee et al., 2021; Liu et al., 2022a), we collect data in a fully automatic manner by prompting a pretrained language model with three examples from the Super-Natural Instructions2 dataset (Mishra et…”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, methodology): “In this way, by selecting the appropriate prompts we can manipulate the model behavior so that the pre-trained LM itself can be used to predict the desired output, sometimes even without any additional task-specific training (Table 1(d); e.g., Brown et al. [9], Petroni et al. [100], Radford et al. [”
Discovering Language Model Behaviors with Model-Written Evaluations (2022, methodology): “Our work pushes the above direction further, by examining whether generated data serves as high-quality evaluation data; useful generated training data can contain noisy or mislabeled examples (Schick and Schütze, 2021), while useful evaluation data must contain relatively few errors.”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “Other work has shown that good few-shot performance can also be obtained by combining gradient based optimization with textual prompts (Schick and Sch¨utze, 2021b; Gao et al., 2021; Tam et al., 2021; Wang et al., 2021).”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “There have been studies with similar goals, such as generating similar/dissimilar sentences for training sentence embeddings [58] and using labeled samples as demonstrations to prompt large PLMs [81] for creating novel training data.”
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, background): “Our approach contrasts with previous instructionbased generation of dataset examples (Schick and Schütze, 2021; West et al., 2021), which requires the model to understand the task from context, fun-”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, neutral): “Thus, our dataset features a diverse collection of instructions (including human-written, template-based and LLM-generated), each paired with a human-written output.”
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021, background): “Similar concurrent works are conducted from Schick and Schütze (2021).”
CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society (2023, background): “LLMs have been explored for data generation in [59, 38, 40, 66], and Self-Instruct [70] proposes a semi-automated process for instruction instance generation.”
A Holistic Approach to Undesired Content Detection in the Real World (2022, unknown): “Contrary to previous studies (Wang et al. 2021b; Schick and Schütze 2021), we found that mixing noisy synthetic data into training hurt model performance.”
PERSONACHATGEN: Generating Personalized Dialogues using GPT-3 (2022, unknown): “Schick and Schütze (2021) first released a textual similarity dataset generated using a pre-trained language model (PLM) with instructions.”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, methodology): “To further explore the potential of prompting, we investigate the two-stage conditional prompt inspired by (Schick and Schütze, 2021).”
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, background): “LLMs have been explored for data generation in [101, 63, 68, 114], and Self-Instruct [119] proposes a semi-automated process for instruction instance generation.”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “4 Schick and Schütze [166] use a generative PLM (GPT2-XL) to generate the continuation, replacing the token.”
Text Data Augmentation for Deep Learning (2021, background): “For example, DINO [26] generates natural language inference data by either seeding the generation with “mean the same thing” or “are on completely different topics”.”