Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification

By Timo Schick, Helmut Schmid, Hinrich Schütze. 2020

228 citations recorded by the source corpus.

A recent approach for few-shot text classification is to convert textual inputs to cloze questions that contain some form of task description, process them with a pretrained language model and map the predicted words to labels. Manually defining this mapping between words and labels requires both domain expertise and an understanding of the language model’s abilities. To mitigate this issue, we devise an approach that automatically finds such a mapping given small amounts of training data. For a number of tasks, the mapping found by our approach performs almost as well as hand-crafted label-to-word mappings.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Meta & Continual Learning

Citations in the corpus

Challenges in Detoxifying Language Models (2021, background): “This could potentially negatively impact self-debiasing strategies (Schick et al., 2020).”
How many data points is a prompt worth? (2021, background): “Finally, recent papers have experimented with discovering prompts through automated processes tailored to the language model (Jiang et al., 2020; Schick et al., 2020).”
Making Pre-trained Language Models Better Few-shot Learners (2021, methodology): “This approach is similar to the automatic verbalizer search methods in Schick and Schütze (2021a); Schick et al. (2020), except that we use a much simpler search process (brute-force) and also apply re-ranking— which we find to be quite helpful.”
PTR: Prompt Tuning with Rules for Text Classification (2021, background): “To alleviate this issue, prompt tuning has been proposed to bridge the gap of objective forms in pre-training and fine-tuning (Brown et al., 2020; Schick et al., 2020).”
Do Prompts Solve NLP Tasks Using Natural Language? (2022, background): “Though other work has also investigated automatic label word generation (Schick et al., 2020), we do not include them all because the label word searching methodology is not our focus.”
Black-Box Tuning for Language-Model-as-a-Service (2022, unknown): “More advanced techniques such as prompt engineering (Gao et al., 2021), label words engineering (Schick et al., 2020; Shin et al., 2020; Hu et al., 2021b), prompt pre-training (Gu et al., 2021), and prompt ensembling (Lester et al., 2021) are orthogonal to this work and therefore can further…”
Template-Based Named Entity Recognition Using BART (2021, background): “Schick et al. (2020) and Gao et al. (2020) extend Schick and Schütze (2020) by automatically generating label words and templates, respectively.”
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification (2021, methodology): “Some other studies (Gao et al., 2021; Shin et al., 2020; Liu et al., 2021; Schick et al., 2020) matic verbalizer searching methods for better ver-balizer choices, however, their methods require adequate training set and validation set for optimization.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, unknown): “This study adopts a progressive reasoning strategy tailored to address the complex linguistic phenomena involved in TC.”
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning (2022, background): “LLMs are especially sensitive to prompting perturbations (Gao et al., 2021; Schick et al., 2020; Liang et al., 2022).”
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, methodology): “Manual design requires domain expertise and understanding of the base LMs. Previous research devised various methods for automatic verbalizer search (Schick et al., 2020; Shin et al., 2020; Gao et al., 2021).”
Prompt-free and Efficient Few-shot Learning with Language Models (2022, background): “c) Automatically finding verbalizers in a discrete way (Schick et al., 2020; Schick and Schütze, 2021a) is computationally expensive and does not perform as well as manually designed ones.”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, background): “Accordingly, some works attempt to obtain either prompts (Shin et al., 2020; Gao et al., 2021; Li and Liang, 2021; Lester et al., 2021) names for output classes (Schick et al., 2020; Gao et al., 2021) without any human involvement.”
Few-shot Learning with Multilingual Language Models (2021, methodology): “Jiang et al. (2019) and Shin et al. (2020) used data mining techniques and gradient based search respectively to find better prompts than manually chosen ones and Schick et al. (2020) found better tokens to represent labels of classification tasks.”