Ignore Previous Prompt: Attack Techniques For Language Models

By Fábio Perez, I. Ribeiro. 2022

954 citations recorded by the source corpus.

Transformer-based large language models (LLMs) provide a powerful foundation for natural language tasks in large-scale customer-facing applications. However, studies that explore their vulnerabilities emerging from malicious user interaction are scarce. By proposing PromptInject, a prosaic alignment framework for mask-based iterative adversarial prompt composition, we examine how GPT-3, the most widely deployed language model in production, can be easily misaligned by simple handcrafted inputs. In particular, we investigate two types of attacks -- goal hijacking and prompt leaking -- and demonstrate that even low-aptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3's stochastic nature, creating long-tail risks. The code for PromptInject is available at https://github.com/agencyenterprise/PromptInject.

Subfields: Artificial Intelligence, Health Informatics, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Safety, Robustness & Interpretability

Citations in the corpus

Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, methodology): “…the main meth-ods employed in LMs safety evaluation, including model preference-based safety testing (Nadeem et al., 2021; Xu et al., 2023a), adversarial safety attacks (Perez et al., 2022; Ganguli et al., 2022), safety risk detection (Dinan et al., 2019; Sun et al., 2022), and other advancements.”
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, background): “…to adversarial inputs that can elicit undesired behavior (Gehman et al., 2020; Bommasani et al., 2021; walkerspider, 2022; Perez et al., 2022; Perez & Ribeiro, 2022; Kang et al., 2023; Li et al., 2023; Ganguli et al., 2022; Schulhoff et al., 2023; OpenAI, 2023b; Jones et al., 2023; Zou et…”
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024, methodology): “Moreover, we explore an enhanced setting where the attacker instructions are reinforced with a “hacking prompt,” a tactic frequently employed in prompt injection attacks (Perez and Ribeiro, 2022; Stubbs, 2023), to examine its impact on the outcomes of the attacks.”
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023, background): “In contrast, we show that adversaries can now remotely affect other users’ systems by strategically injecting the prompts into data likely to be retrieved at inference time.”
StruQ: Defending Against Prompt Injection with Structured Queries (2024, positive): “Recent research has uncovered a variety of ways that attackers can use to make prompt injection attacks more effective, such as misleading sentences [9], unique characters [6], and other methods [30].”
Prompt Injection attack against LLM-integrated Applications (2023, methodology): “Prompt injection [6,20,44] overrides an LLM’s original prompt and directs it to follow malicious instructions.”
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks (2023, methodology): “Each measure was done on a ﬁve-point Likert scale.”
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (2024, background): “These attacks are straightforward to execute (Perez & Ribeiro, 2022; Zhang & Ippolito, 2023; Schulhoff et al., 2023)—even when models are told to avoid repeating their prompt—by using attacks such as: System Message: You are an email assistant.”
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (2023, neutral): “…our work from existing efforts in safety research of LLMs. Specifically, both SafetyPrompts (Sun et al., 2023) and prompt injection attacks (Perez & Ribeiro, 2022; Greshake et al., 2023; Liu et al., 2023b) are engineered to spotlight potentially harmful instructions that could steer LLMs…”
Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, methodology): “Context Ignoring: This attack [36] uses a task-ignoring text (e.g., “Ignore my previous instructions.”) to explicitly tell the LLM that the target task should be ignored.”
Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (2023, neutral): “These adversarial textual inputs and prompt injections (Liu et al., 2023b; Perez & Ribeiro, 2022) exhibit high transferability, enabling them to bypass the safety guards of different LLMs (Wei et al., 2023).”
Tool Learning with Foundation Models (2023, neutral): “But the attempt of crafting misleading or even harmful queries will undoubtfully persist as well (Perez & Ribeiro, 2022).”
"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models (2023, positive): “Perez and Ribeiro [68] study prompt injection attacks against LLMs and find that LLMs can be easily misaligned by simple handcrafted inputs.”
Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models (2023, neutral): “As LLMs continue to develop, their security has become increasingly important [4], [6], [14], [34].”
The rise and potential of large language model based agents: a survey (2025, positive): “There are also some relevant attack methods such as dataset poisoning [520], backdoor attacks [521,522], and prompt-speciﬁc attacks [523,524], with the potential to induce LLMs to generate toxic content [525–527].”
Generative artificial intelligence (2023, positive): “However, applications can still be tricked into bypassing filters and safeguards of GAI models, for instance, through prompt injections that insert malicious prompts to achieve misaligned outputs of generative AI applications (Perez & Ribeiro, 2022).”