Formalizing and Benchmarking Prompt Injection Attacks and Defenses
By Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, Neil Zhenqiang Gong. USENIX Security Symposium, 2023
403 citations recorded by the source corpus.
A prompt injection attack aims to inject malicious instruction/data into the input of an LLM-Integrated Application such that it produces results as an attacker desires. Existing works are limited to case studies. As a result, the literature lacks a systematic understanding of prompt injection attacks and their defenses. We aim to bridge the gap in this work. In particular, we propose a framework to formalize prompt injection attacks. Existing attacks are special cases in our framework. Moreover, based on our framework, we design a new attack by combining existing ones. Using our framework, we conduct a systematic evaluation on 5 prompt injection attacks and 10 defenses with 10 LLMs and 7 tasks. Our work provides a common benchmark for quantitatively evaluating future prompt injection attacks and defenses. To facilitate research on this topic, we make our platform public at https://github.com/liu00222/Open-Prompt-Injection.
Subfields: Artificial Intelligence, Signal Processing
View Paper: https://www.semanticscholar.org/paper/9adddfe5a596d20b534a29d9937efcd3cb85d3cf
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Signal Processing
Citations in the corpus
- StruQ: Defending Against Prompt Injection with Structured Queries (2024, positive): “This introduces the risk of prompt injection attacks [5, 6, 7], where a malicious user can supply data with injected prompts and subvert the operation of the LLM-integrated application.”
- Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, positive): “Formally, the adversarial goal is to maximize (Alon & Kamfonas, 2023; Jain et al., 2023; Liu et al., 2024) Identify compromised memory by measuring its text perplexity.”