StruQ: Defending Against Prompt Injection with Structured Queries

By Sizhe Chen, Julien Piet, Chawin Sitawarin, David Wagner. USENIX Security Symposium, 2024

336 citations recorded by the source corpus.

Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications, which perform text-based tasks by utilizing their advanced language understanding capabilities. However, as LLMs have improved, so have the attacks against them. Prompt injection attacks are an important threat: they trick the model into deviating from the original application's instructions and instead follow user directives. These attacks rely on the LLM's ability to follow instructions and inability to separate prompts and user data. We introduce structured queries, a general approach to tackle this problem. Structured queries separate prompts and data into two channels. We implement a system that supports structured queries. This system is made of (1) a secure front-end that formats a prompt and user data into a special format, and (2) a specially trained LLM that can produce high-quality outputs from these inputs. The LLM is trained using a novel fine-tuning strategy: we convert a base (non-instruction-tuned) LLM to a structured instruction-tuned model that will only follow instructions in the prompt portion of a query. To do so, we augment standard instruction tuning datasets with examples that also include instructions in the data portion of the query, and fine-tune the model to ignore these. Our system significantly improves resistance to prompt injection attacks, with little or no impact on utility. Our code is released at https://github.com/Sizhe-Chen/StruQ.

View Paper: https://www.semanticscholar.org/paper/f5e7e22036c3fe7d6660eee90642f716c3b303f5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024, methodology): “…et al., 2023; Piet et al., 2023), replacing command words with encoded versions and instructing the LLM to accept only these encoded commands (Suo, 2024), and separating prompts and data into two channels using a secure front-end for formatting alongside a specially trained LLM (Chen et al., 2024).”
Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, background): “Other defenses against prompt injection attacks: We note that several recent studies [17,38,55] proposed other defenses against prompt injection attacks.”
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024, background): “Defenses for Prompt Injection For prompt injection on closed-domain tasks (Section 3.2), recent work has advocated for teaching a model to treat third-party user inputs as data, not as instructions (Chen et al., 2024; Willison, 2023; Zverev et al., 2024; Yi et al., 2023; Liu et al., 2023).”