Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts

By Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sameer Singh, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal and 1 more. 2021

92 citations recorded by the source corpus.

Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning. Motivated by these promising results, we investigate the feasibility of extracting a discrete (textual) interpretation of continuous prompts that is faithful to the problem they solve. In practice, we observe a “wayward” behavior between the task solved by continuous prompts and their nearest neighbor discrete projections: We can find continuous prompts that solve a task while being projected to an arbitrary text (e.g., definition of a different or even a contradictory task), while being within a very small (2%) margin of the best continuous prompt of the same size for the task. We provide intuitions behind this odd and surprising behavior, as well as extensive empirical analyses quantifying the effect of various parameters. For instance, for larger model sizes we observe higher waywardness, i.e, we can find prompts that more closely map to any arbitrary text with a smaller drop in accuracy. These findings have important implications relating to the difficulty of faithfully interpreting continuous prompts and their generalization across models and tasks, providing guidance for future progress in prompting language models.

Subfields: Artificial Intelligence, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

Citations in the corpus

Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, methodology): “[58] and its similarity to other PEFT methods He et al.”
Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2021, background): “Finally, a growing body of concurrent work also questions the degree to which models need meaningful instructions (Khashabi et al., 2021; Prasad et al., 2022). One particularly noteworthy finding is that Min et al. (2022) show that models learn just as well with incorrect labels as opposed to correc”
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, background): “By their continuous nature, however, soft prompts are difficult to understand (Lester et al., 2021; Hambardzumyan et al., 2021; Khashabi et al., 2021), require expensive gradient information (Sun et al., 2022; Diao et al., 2022) and are incompatible for reuse across models due to mismatched latent…”
Steering Language Models With Activation Engineering (2023, background): “…engineering (Shin et al. 2020; Zhou et al. 2022); • Intervening on token embeddings , as with ‘soft prompting’ (Li & Liang 2021; Lester et al. 2021; Khashabi et al. 2022); • Intervening on activations , for instance by freezing the weights of the LLM and searching for a ‘steering vector’ of…”
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (2023, background): “Other work has focused on providing textual interpretations of continuous prompt representations (Khashabi et al., 2022).”
Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery (2023, background): “However, subsequent work by Khashabi et al. (2022) showed that the sequences of embeddings produced by this technique could map to token sequences with limited semantic scrutability.”
Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (2024, unknown): “2) Soft prompts often fall short of interpretability (Khashabi et al., 2021; Lester et al., 2021b; Hambardzumyan et al., 2021; Mokady et al., 2021).”
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022, result): “Last, our interpretation of FFN outputs as updates to the output distribution relates to recent works that interpreted groups of LM parameters in the discrete vocabulary space (Geva et al., 2021; Khashabi et al., 2021), or viewed the representation”