Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting

By Melanie Sclar, Yejin Choi, Yulia Tsvetkov, Alane Suhr. International Conference on Learning Representations, 2023

839 citations recorded by the source corpus.

As large language models (LLMs) are adopted as a fundamental component of language technologies, it is crucial to accurately characterize their performance. Because choices in prompt design can strongly influence model behavior, this design process is critical in effectively using any modern pre-trained generative language model. In this work, we focus on LLM sensitivity to a quintessential class of meaning-preserving design choices: prompt formatting. We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt formatting in few-shot settings, with performance differences of up to 76 accuracy points when evaluated using LLaMA-2-13B. Sensitivity remains even when increasing model size, the number of few-shot examples, or performing instruction tuning. Our analysis suggests that work evaluating LLMs with prompting-based methods would benefit from reporting a range of performance across plausible prompt formats, instead of the currently-standard practice of reporting performance on a single format. We also show that format performance only weakly correlates between models, which puts into question the methodological validity of comparing models with an arbitrarily chosen, fixed prompt format. To facilitate systematic analysis we propose FormatSpread, an algorithm that rapidly evaluates a sampled set of plausible prompt formats for a given task, and reports the interval of expected performance without accessing model weights. Furthermore, we present a suite of analyses that characterize the nature of this sensitivity, including exploring the influence of particular atomic perturbations and the internal representation of particular formats.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/17a6116e5bbd8b87082cbb2e795885567300c483

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Dan Hendrycks
  2. Sergey Levine
  3. Percy Liang
  4. Quoc V. Le
  5. Luke S. Zettlemoyer
  6. Jianfeng Gao
  7. Mohit Bansal
  8. Pieter Abbeel

In Electrical and Electronic Engineering

  1. Jimmy Ba
  2. Timothy P. Lillicrap
  3. Dan Hendrycks
  4. Sergey Levine
  5. Pieter Abbeel
  6. Shansi Zhang
  7. Yuxi Li
  8. Shimon Whiteson

Citations in the corpus