Do Prompt-Based Models Really Understand the Meaning of Their Prompts?

By Albert Webson, Ellie Pavlick. 2021

482 citations recorded by the source corpus.

Recently, a boom of papers has shown extraordinary progress in zero-shot and few-shot learning with various prompt-based models. It is commonly argued that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language. In this study, we experiment with over 30 prompts manually written for natural language inference (NLI). We find that models can learn just as fast with many prompts that are intentionally irrelevant or even pathologically misleading as they do with instructively “good” prompts. Further, such patterns hold even for models as large as 175 billion parameters (Brown et al., 2020) as well as the recently proposed instruction-tuned models which are trained on hundreds of prompts (Sanh et al., 2021). That is, instruction-tuned models often produce good predictions with irrelevant and misleading prompts even at zero shots. In sum, notwithstanding prompt-based models’ impressive improvement, we find evidence of serious limitations that question the degree to which such improvement is derived from models understanding task instructions in ways analogous to humans’ use of task instructions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

OPT: Open Pre-trained Transformer Language Models (2022, background): “Results vary signiﬁcantly and unpredictably with the selection of the prompt (Lu et al., 2021), and models do not seem to understand the prompts as fully as we expect (Webson and Pavlick, 2021).”
Large Language Models Are Human-Level Prompt Engineers (2022, neutral): “However, LLMs require careful prompt engineering, either manually (Reynolds & McDonell, 2021) or automatically (Gao et al., 2021; Shin et al., 2020), as models do not seem to understand the prompts in the same way a human would (Webson & Pavlick, 2021; Lu et al., 2021).”
Can language models learn from explanations in context? (2022, background): “Although it is not always clear what is being learned or inferred from these prompts (e.g. Min et al., 2022; Webson and Pavlick, 2021), prompting is a growing subﬁeld (Liu et al., 2021).”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, background): “This has been partially veriﬁed by Webson and Pavlick (2022) who showed that the model performance does not degrade much with irrelevant or misleading instructions.”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “Finally, the exact formatting of the prompt (including the wording [11] and ordering of examples [12]) can have significant and unpredictable impact on the model’s performance, far beyond inter-run variation of fine-tuning.”
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022, background): “The process of prompt engineering is critical for successful deployment as choices in prompting can affect downstream predictions signiﬁcantly, particularly in the zero-shot setting (Perez et al., 2021; Zhao et al., 2021; Webson and Pavlick, 2021).”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, background): “For example, recent preliminary studies have begun investigating the robustness of prompt-based methods in large-scale language models [109, 121, 140].”
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2022, background): “Webson and Pavlick (2022) finds that for instruction models, the performance on natural language inference tasks has small degradations under irrelevant or misleading instructions.”
A Comprehensive Overview of Large Language Models (2023, unknown): “This paper will serve as a valuable resource for researchers, offering insights into the recent advancements in LLMs and providing fundamental concepts and details to develop improved LLMs.”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, methodology): “Slightly different patterns can have very different performance (Jiang et al., 2020; Schick and Schütze, 2020; Schick et al., 2021; Webson and Pavlick, 2021; Sanh et al., 2022, inter alia) and popular model selection criteria cannot reliably identify the best-performing patterns in few-shot…”
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022, background): “Furthermore, many works focused on investigating whether LMs understands meaning of natural language and prompts (Webson and Pavlick, 2021; Zhao et al., 2021).”
The Unreliability of Explanations in Few-Shot In-Context Learning (2022, background): “However, this learning process is still poorly understood: models are biased by the order of examples they are shown (Zhao et al., 2021) and may not leverage the instructions or even the labels of the few-shot examples in the ways we expect (Min et al., 2022; Webson and Pavlick, 2022).”
Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers (background): “This is an area of increasing interest (e.g. Min et al., 2022; Razeghi et al., 2022; Webson and Pavlick, 2021; Xie et al., 2021).”
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? (2023, background): “Large language models (LLMs) have been shown to be able to solve a variety of natural language processing (NLP) tasks zero shot—i.e., without relying on any training data for a given downstream task—by conditioning the model on appropriate prompts (Brown et al., 2020; Chowdhery et al., 2022a).”