Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

By Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, C. Endres, Thorsten Holz, Mario Fritz. AISec@CCS, 2023

1,594 citations recorded by the source corpus.

Large Language Models (LLMs) are increasingly being integrated into applications, with versatile functionalities that can be easily modulated via natural language prompts. So far, it was assumed that the user is directly prompting the LLM. But, what if it is not the user prompting? We show that LLM-Integrated Applications blur the line between data and instructions and reveal several new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (i.e., without a direct interface) exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved at inference time. We derive a comprehensive taxonomy from a computer security perspective to broadly investigate impacts and vulnerabilities, including data theft, worming, information ecosystem contamination, and other novel security risks. We then demonstrate the practical viability of our attacks against both real-world systems, such as Bing Chat and code-completion engines, and GPT-4 synthetic applications. We show how processing retrieved prompts can act as arbitrary code execution, manipulate the application's functionality, and control how and if other APIs are called. Despite the increasing reliance on LLMs, effective mitigations of these emerging threats are lacking. By raising awareness of these vulnerabilities, we aim to promote the safe and responsible deployment of these powerful models and the development of robust defenses that protect users from potential attacks.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/705e49afd92130f2bc1e0d4d0b1f6cb14e88803f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2024, neutral): “More specifically, AgentDojo (Debenedetti et al., 2024) focuses on prompt injections (Greshake et al., 2023) and harmless requests.”
LLM Censorship: A Machine Learning Challenge or a Computer Security Problem? (2023, methodology): “A comprehensive taxonomy of attacks and vulnerabilities of LLMs, particularly in settings when integrated with external tools, was provided by Greshake et al. [26].”
LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked (2023, background): “As LLMs have grown in complexity and capability, so has their attack surface [9].”
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024, background): “Consequently, if an application has access to confidential data or can take actions in the world, prompt injections can cause catastrophic harms such as exfiltrating user data (Greshake et al., 2023) or hijacking the LLM’s actions (Toyer et al., 2024).”
Effective Long-Context Scaling of Foundation Models (2023, background): “Long-context language models can process extended inputs in their context window, but at the same time, they also face a higher risk of jailbreak, especially through means such as prompt injection (Greshake et al., 2023).”
Baseline Defenses for Adversarial Attacks Against Aligned Language Models (2023, background): “As the role of LLMs expands in its scope and complexity, so does their attack surface (Hendrycks et al., 2022; Greshake et al., 2023).”
GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2023, background): “Furthermore, their widespread use has made them targets for adversarial attacks, including backdoor attacks [7, 33, 42], prompt injection [24, 37, 50], and data poisoning [35, 44, 73].”
A StrongREJECT for Empty Jailbreaks (2024, methodology): “The model is instructed to respond in base64.”
Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, background): “However, existing works–including both research papers [22,36] and blog posts [23,41,51,52]–are mostly about case studies and they suffer from the following limitations: 1) they lack frameworks to formalize prompt injection attacks and defenses, and 2) they lack a comprehensive evaluation of prompt…”
Many-shot Jailbreaking (2024, neutral): “Although the harms from jailbroken models are significant (Bommasani et al., 2022; Bender et al., 2021; Weidinger et al., 2021), we believe the benefits of disclosing and characterizing new jail-breaks currently outweigh the risk that new jailbreak meth-ods are widely adopted.”
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025, neutral): “While LLMs primarily function as reactive prompt followers [61], AI Agents operate automatically within explicitly defined scopes, engaging dynamically with inputs and producing actionable outputs in real-time environments [62].”
StruQ: Defending Against Prompt Injection with Structured Queries (2024, neutral): “Use of LLMs in applications opens up the risk of prompt injection attacks [26, 27, 9, 5, 6, 7, 28, 29].”
FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts (2023, methodology): “By putting text on visual input, the typographic attack can be used to cause misclassifications or manipulate the output of VLMs [18], which falls under the category of prompt injection that has a similar threat model but a different goal compared to jailbreak attack.”
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024, background): “Notably, real-world instances of IPI attacks on Ope-nAI plugins have been documented, leading to various consequences, such as phishing link insertion, conversational history exfiltration, GitHub code theft, and more (Rehberger, 2023a,b,c; Greshake, 2023; Samoilenko, 2023; Piltch, 2023).”
Prompt Injection attack against LLM-integrated Applications (2023, methodology): “Prompt injection [6,20,44] overrides an LLM’s original prompt and directs it to follow malicious instructions.”
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, neutral): “…injection can target the user prompt directly (Perez & Ribeiro, 2022; Selvi, 2022; Toyer et al., 2023; Yu et al., 2023a; Kang et al., 2024) or indirectly influence the agent’s behavior by manipulating its accessible external environment (Liu et al., 2023; Greshake et al., 2023; Yi et al., 2023).”
Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models (2023, neutral): “While the same positive correlation is also observed in code attacks, the corresponding Pearson correlation coefficient is lower than that of text attacks.”
ER-MIA: Black-Box Adversarial Memory Injection Attacks on Long-Term Memory-Augmented Large Language Models (2026, unknown): “…memory-augmented LLMs critically depend on retrieved memories to guide inference, and when misleading memories are stored and retrieved, they can persistently corrupt reasoning behavior across future interactions (Chang et al., 2026; Greshake et al., 2023; Liu et al., 2023; Zhan et al., 2025).”