Zero-shot Text Classification With Generative Language Models

By Raul Puri, Bryan Catanzaro. 2019

121 citations recorded by the source corpus.

This work investigates the use of natural language to enable zero-shot model adaptation to new tasks. We use text and metadata from social commenting platforms as a source for a simple pretraining task. We then provide the language model with natural language descriptions of classification tasks as input and train it to generate the correct answer in natural language via a language modeling objective. This allows the model to generalize to new classification tasks without the need for multiple multitask classification heads. We show the zero-shot performance of these generative language models, trained with weak supervision, on six benchmark text classification datasets from the torchtext library. Despite no access to training data, we achieve up to a 45% absolute improvement in classification accuracy over random or majority class baselines. These results show that natural language can serve as simple and powerful descriptors for task adaptation. We believe this points the way to new metalearning strategies for text problems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, background): “Recently, Puri and Catanzaro (2019) applied the same idea to text classification, providing actual task descriptions.”
How many data points is a prompt worth? (2021, methodology): “While this approach can be used for zero-shot classification (Puri and Catanzaro, 2019) or priming (Brown et al.”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, methodology): “Enabling LMs to perform zero-shot learning by providing task descriptions was proposed by Radford et al. (2019) and has been applied to text classiﬁ-cation (Puri and Catanzaro, 2019), commonsense knowledge mining (Davison et al., 2019) and ar-gumentative relation classiﬁcation (Opitz, 2019).”
Generating Datasets with Pretrained Language Models (2021, methodology): “Providing PLMs with task descriptions for zero- or fewshot learning has been studied extensively (e.g., Radford et al., 2019; Puri and Catanzaro, 2019; Brown et al., 2020; Schick and Schütze, 2020, 2021b,a; Weller et al., 2020; Gao et al., 2021; Tam et al., 2021).”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “The current mainstream of zero-shot learning is based on transfer learning: By converting a set of tasks with abundant annotations into instruction templates [42, 54, 70, 74], entailment pairs [79, 80] or question-answer formats [50, 86] and fine-tuning PLMs on them, the PLMs acquire the cross-task ”
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021, methodology): “Our approach for performing self-diagnosis builds heavily on recent work that explores zero-shot learning using task descriptions (Radford et al., 2019; Puri and Catanzaro, 2019; Schick and Schütze, 2021a).”
Document Ranking with a Pretrained Sequence-to-Sequence Model (2020, background): “As Puri and Catanzaro [13] have shown, the choice of target words impacts effectiveness.”
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification (2020, background): “Reformulating problems as language modeling tasks has been explored in fully unsupervised settings (Radford et al., 2019; Puri and Catanzaro, 2019; Davison et al., 2019), in few-shot scenarios with limited amounts of training data (Opitz, 2019; Shwartz et al., 2020; Brown et al., 2020), and even in…”
Entailment as Few-Shot Learner (2021, background): “…GPT works (Radford et al., 2019; Brown et al., 2020) proposed to add a task description (prompt) and annotated examples as demonstration to enable few-shot learning, which has been commonly applied to classification (Puri & Catanzaro, 2019), QA, commonsense knowledge mining (Davison et al., 2019).”
On Robust Prefix-Tuning for Text Classification (2022, background): “Recently, increasing interest has been focused on prompt-based tuning approaches for pretrained language models (Wallace et al., 2019; Puri & Catanzaro, 2019; Shin et al., 2020; Jiang et al., 2020b; Zhong et al., 2021; Gao et al., 2021; Hu et al., 2021; Liu et al., 2021).”
Training Question Answering Models from Synthetic Data (2020, methodology): “This method of multi-input controlled text generation draws on inspiration from prior work (Puri & Catanzaro, 2019; Raffel et al., 2019; Dong et al., 2019).”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, methodology): “This idea has since been applied to solve a wide range of different NLP tasks without any task-speciﬁc training data (Puri and Catanzaro, 2019; Opitz, 2019; Davison et al., 2019; Schick et al., 2021; Schick and Schütze, 2021; Wei et al., 2021; Sanh et al., 2021).”
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks (2020, result): “The idea of phrasing classiﬁcation tasks as sentence completion problems is motivated by recent works [Radford et al., 2019, Puri and Catanzaro, 2019, Schick and Schütze, 2020] that show that many downstream tasks can be solved by next word prediction for an appropriately conditioned language model.”
Generated Knowledge Prompting for Commonsense Reasoning (2021, methodology): “Consequently, these models can directly perform inference on tasks like commonsense reasoning (Trinh and Le, 2018; Yang et al., 2020), text classiﬁcation (Shin et al., 2020; Puri and Catanzaro, 2019), and natural language inference (Shin et al., 2020; Schick and Schütze, 2021).”