Noisy Channel Language Model Prompting for Few-Shot Text Classification

By Sewon Min, Michael Lewis, Hannaneh Hajishirzi, Luke S. Zettlemoyer. 2021

238 citations recorded by the source corpus.

We introduce a noisy channel approach for language model prompting in few-shot text classification. Instead of computing the likelihood of the label given the input (referred as direct models), channel models compute the conditional probability of the input given the label, and are thereby required to explain every word in the input. We use channel models for recently proposed few-shot learning methods with no or very limited updates to the language model parameters, via either in-context demonstration or prompt tuning. Our experiments show that, for both methods, channel models significantly outperform their direct counterparts, which we attribute to their stability, i.e., lower variance and higher worst-case accuracy. We also present extensive ablations that provide recommendations for when to use channel prompt tuning instead of other competitive models (e.g., direct head tuning): channel prompt tuning is preferred when the number of training examples is small, labels in the training data are imbalanced, or generalization to unseen labels is required.

Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Emergent Abilities of Large Language Models (2022, positive): “, 2021) or using a noisy channel (Min et al., 2021) have improved performance on a range of tasks.”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, methodology): “Prior work proposes better ways of formulating the problem (Zhao et al., 2021; Holtzman et al., 2021; Min et al., 2021a), better ways of choosing labeled examples for the demonstrations (Liu et al., 2021; Lu et al., 2021; Rubin et al., 2021), meta-training with an explicit in-context learning…”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “[21] proposed ensemble ICL, where instead of using the output probability from concatenating the k training examples, the output probabilities of the model on each training example (i.”
MetaICL: Learning to Learn In Context (2021, methodology): “It has been further improved by later work (Zhao et al., 2021; Holtzman et al., 2021; Min et al., 2021), showing promising results on a variety of tasks.”
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks (2021, background): “Prompting’s advantage and effectiveness in a wide range of NLP applications has been verified in recent literature, including text classification (Hu et al., 2021; Min et al., 2021; Sun et al., 2021; Li et al., 2021; Zhang et al., 2021b), entity typing (Ding et al.”
Learning To Retrieve Prompts for In-Context Learning (2021, methodology): “, 2019; Schick and Schütze, 2021), decoding methods (Min et al., 2021; Zhao et al., 2021; Holtzman et al., 2021), and methods for extracting either hard (Shin et al.”
A Survey for In-context Learning (2023, methodology): “Min et al. (2022c) investigated that the inﬂuence of demonstration samples come from four aspects: the input-label pairing format, the label space, the input distribution, and the input-label mapping.”
Selective Annotation Makes Language Models Better Few-Shot Learners (2022, methodology): “…methods to improve in-context learning in many aspects: e.g., meta-training (Chen et al., 2022; Min et al., 2022b), task instructions (Efrat & Levy, 2020; Mishra et al., 2022; Wei et al., 2021; Sanh et al., 2022), or task formulation (Holtzman et al., 2021; Zhao et al., 2021; Min et al., 2022a).”
Gradient-based Constrained Sampling from Language Models (2022, background): “·| Intu-itively, they are required to explain every word in the input, potentially amplifying the class signal and improving robustness (Min et al., 2021).”
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations (2022, methodology): “The left subfigure in Figure 2 comes from the template and normalization strategy from the prior work, Channel inference method (Min et al., 2021), (macro-)F1 metric, and five selected classification datasets in Figure 4 of the prior work.”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, methodology): “…where the model’s parameters are ﬁxed and examples are provided as additional context (Brown et al., 2020; Lu et al., 2021; Kumar and Talukdar, 2021; Min et al., 2021), ﬁnetuning the entire model (Schick and Schütze, 2021a,c; Gao et al., 2021; Tam et al., 2021), and prompt tuning , where only the…”
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, unknown): “Few-Shot Setting Following previous works (Gao et al., 2021; Min et al., 2021; Sun et al., 2022),”
Prompt-free and Efficient Few-shot Learning with Language Models (2022, methodology): “For the baselines, we used the carefully manually designed patterns and verbalizers in Gao et al. (2021), Min et al. (2021), and Schick and Schütze (2021b) (usually 5 different options per datasets; see Appendix B).”
AdaPrompt: Adaptive Model Training for Prompt-based NLP (2022, methodology): “Prompt-based methods have been shown particularly useful in zero-shot and few-shot settings (Petroni et al., 2019; Yin et al., 2019; Min et al., 2022), where with limited direct task data, prompt-based inference ben-eﬁts more from large-scale pretraining than task-oriented ﬁne-tuning.”
Natural Language to Code Translation with Execution (2022, background): “These results lead to prompt-based approaches for few-shot or zero-shot text classiﬁcation (Shin et al., 2020; Gao et al., 2021; Min et al., 2021, inter alia ), question answering (Khashabi et al., 2020), machine translation (Radford et al., 2019), and evaluation of generated text (Yuan et al.,…”
Offline RL for Natural Language Generation with Implicit Language Q Learning (2022, background): “However, such models are not designed to optimize any user-specified utility, instead requiring considerable trial-anderror to design prompts that “coerce” the models into producing desirable outcomes [1, 2, 3].”
Automatic Chain of Thought Prompting in Large Language Models (2022, background): “…fine-grained information, such as incorporating task instruction [Mishra et al., 2022, Wei et al., 2022b, Sanh et al., 2022]; (iii) manipulating output probabilities of LLMs instead of directly computing the likelihood of target labels [Holtzman et al., 2021, Zhao et al., 2021, Min et al., 2022a].”