The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

By Eric Wallace, Kai Xiao, R. Leike, Lilian Weng, Johannes Heidecke, Alex Beutel. arXiv.org, 2024

416 citations recorded by the source corpus.

Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts. In this work, we argue that one of the primary vulnerabilities underlying these attacks is that LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. To address this, we propose an instruction hierarchy that explicitly defines how models should behave when instructions of different priorities conflict. We then propose a data generation method to demonstrate this hierarchical instruction following behavior, which teaches LLMs to selectively ignore lower-privileged instructions. We apply this method to GPT-3.5, showing that it drastically increases robustness -- even for attack types not seen during training -- while imposing minimal degradations on standard capabilities.

Subfields: Law, Political Science and International Relations

View Paper: https://www.semanticscholar.org/paper/f18e5a844c37e5342f8f3d409c74c1a9c91d1f8f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. David Wagner
2. Sizhe Chen
3. Chawin Sitawarin
4. Julien Piet
5. Qiusi Zhan

Citations in the corpus

• StruQ: Defending Against Prompt Injection with Structured Queries (2024, positive): “After the release of our paper, Wallace et al. [43] introduced the instruction hierarchy, which can be viewed as a generalization of StruQ.”
• gpt-oss-120b&gpt-oss-20b Model Card (2025, neutral): “The gpt-oss models are autoregressive Mixture-of-Experts (MoE) transformers [1, 2, 3, 4] that build upon the GPT-2 and GPT-3 architectures.”
• Workerville: Towards an Organizational Behavior Account of Agent Safety (2026, unknown): “Safety evaluations typically center on individual mechanisms, examining instruction hierarchies and user authority (Wallace et al., 2024), peer and multi-agent influence (Zhang et al., 2024), or memory poisoning and persistent state in isolation (Lin et al., 2026).”