Few-Shot Text Generation with Natural Language Instructions

By Timo Schick, Hinrich Schütze. 2020

160 citations recorded by the source corpus.

Providing pretrained language models with simple task descriptions in natural language enables them to solve some tasks in a fully unsupervised fashion. Moreover, when combined with regular learning from examples, this idea yields impressive few-shot results for a wide range of text classification tasks. It is also a promising direction to improve data efficiency in generative settings, but there are several challenges to using a combination of task descriptions and example-based learning for text generation. In particular, it is crucial to find task descriptions that are easy to understand for the pretrained model and to ensure that it actually makes good use of them; furthermore, effective measures against overfitting have to be implemented. In this paper, we show how these challenges can be tackled: We introduce GenPET, a method for text generation that is based on pattern-exploiting training, a recent approach for combining textual instructions with supervised learning that only works for classification tasks. On several summarization and headline generation datasets, GenPET gives consistent improvements over strong baselines in few-shot settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, background): “…can also be accomplished by combining textual templates (“prompts”) and various forms of model ﬁnetuning, either fully updating a model’s parameters, e.g. for classiﬁcation (Schick & Schütze, 2021a; Schick & Schutze, 2021; Gao et al., 2021; Tam et al., 2021) or generation (Schick & Schütze, 2021b).”
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022, methodology): “Inspired by recent work on utilizing language models for data generation (Schick and Schütze, 2021b; Lee et al., 2021; Liu et al., 2022a), we collect data in a fully automatic manner by prompting a pretrained language model with three examples from the Super-Natural Instructions2 dataset (Mishra et…”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, background): “Constructing effective discrete prompts for language models to perform NLP tasks is an active area of research (Schick and Schütze, 2021; Reynolds and McDonell, 2021; Liu et al., 2021).”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, methodology): “Schick and Schütze [117], Schick and Schütze [118, 120] use an ensemble model to annotate a large dataset to distill the knowledge from multiple prompts.”
PEER: A Collaborative Language Model (2022, methodology): “This enables us to use self-training techniques (e.g., Yarowsky, 1995; Sennrich et al., 2016; He et al., 2020a; Schick and Schütze, 2021a) for training PEER with synthetic plans, edits, explanations and documents.”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “The prompts can be either in natural language [57] or as learnable parameters [31].”
Reframing Instructional Prompts to GPTk’s Language (2021, background): “Discrete Prompts Constructing effective discrete prompts for language models to perform NLP tasks is an active area of research (Schick and Schütze, 2021; Le Scao and Rush, 2021; Tam et al., 2021; Logan IV et al., 2021; Reynolds and McDonell, 2021).”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, background): “…al., 2018; Devlin et al., 2019, inter alia ), where recent work has focused on designing templates for few-shot ﬁne-tuning (Reynolds and McDonell, 2021; Schick and Schütze, 2021a,c,b; Le Scao and Rush, 2021; Tam et al., 2021; Gao et al., 2021; Sorensen et al., 2022), and optimizing soft prompts…”
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022, methodology): “prompts to perform NLP tasks is an active area of research (Schick and Schütze, 2021; Liu et al., 2021a).”
Exploring Prompt-based Few-shot Learning for Grounded Dialog Generation (2021, methodology): “We compared several representative pre-trained models to be used as the backbone models, which are also popularly adopted in previous works of dialog generation (Mi et al., 2021; Shuster et al., 2021).”